Getting Started - Mediation

Introduction

The mediation process allows jBilling to read in purchases and orders from an external source. In most cases jBilling isn't the primary system for tracking customer usage and interaction. For example, a telecommunication company would have special systems dedicated to routing calls, tracking minutes and customer usage. You wouldn't want jBilling to handle the arduous task of routing phone calls, but by feeding in events from this system into jBilling, we can handle pricing, customer balances, invoicing and other complex billing tasks.

The mediation process is similar to a synchronization process but there are some notable differences. In synchronization all aspects of the data being moved from one environment to another are preserved. Item A should equal synchronized item A. In a mediation process the data being moved is transformed into an agreed upon format and jBilling does it's best to resolve this data into orders and items, some data may be kept and some discarded. Exactly what data is moved into jBilling is handled by a user defined format definition and mediation rule base.

Simply put, the goal of the mediation process is to allow jBilling to closely integrate with an existing system - by turning external "events" into billable orders.

This guide is an overview of how to implement a mediation process with jBilling. For detailed documentation, please read the Telecom Guide.

For the purposes of this guide, we're going to assume that you've already worked through the Getting Started and Getting Started - BRMS guides.

Process Overview

The mediation process can be broken into three distinct components, a reader, mediation format, and a rule set.

A mediation reader is a plug-in who's purpose is to read in data from an external source. Flavors include a SeparatorFileReader for field delimited text files, FixedFileReader for fixed length records, a JDBCReader for reading from a database and a MySQLReader that provides a simpler configuration for MySQL databases. In this guide we'll follow a simple example using a SeparatorFileReader to process a CSV file of events.

The mediation format defines the incoming data (called events) so that it can be parsed and converted to an internal format. The configured mediation reader plug-in takes this mediation format as a plug-in parameter. As we'll see later in this guide, the mediation format definition controls what information we'll have available to us when processing rules.

The job of adding the mediated events into jBilling as billable elements belongs to the Mediation rule set and other rules based plug-ins. You can think of the readers and mediation format definition as an input or data source, and the rule set a parser. Keep in mind that the Mediation process doesn't just fire a specific mediation-only rules package, it also triggers item management and pricing rules as well. This allows complex business logic (buy 10, get one free!) and pricing rules to be applied to mediated orders the same as those created through the main jBilling UI.

It's important to note that the mediation process doesn't call out to other item management and pricing plug-ins. It actually accepts multiple rule packages and executes them all simultaneously. While this seems like an asynchronous approach, the reality is that rules still end up being executed in a predictable sequence. This is because item rules depend on an order line being present, so they don't fire until mediation adds an order line. Pricing rules won't execute without a pricing request being present, so they don't fire until either mediation or item management rules request a price; And so on. Because of the way the rules are structured, we end up with a "flow" that looks like this:

mediation rules flow

Configuration

Main Subscriptions

Ideally, all mediated events for a period should be added to a single order. When determining destination order for mediated events, jBilling attempts to work forward from a customers "Main Subscription" order. Generally speaking, a main subscription order is a reoccurring order that contains the customers subscribed service plan - and although any reoccurring order can be marked as the "Main Subscription", a customer should only ever have one. Without a main subscription order, the mediation process will be unable to group mediation events onto the same order, resulting in a new order being created for every incoming event (messy to say the least).

To enable the main subscription flag, set the integer value of Preference 41 to '1' (see appendix A of the User Guide).

insert into preference (id, type_id, table_id, foreign_id, int_value) 
values ((select max(id)+1 from preference), 41, 5, 1, 1);

Once the preference has been set, the "Main Subscription" flag can be set when creating or editing an order.

We've already taken care of enabling the preference for the example Trend company to save you the pain of setting up a SQL client for the in-memory database.

Lets create a new order for Penny Bright, a customer within the example Trend company that's bundled with jBilling. Log into Trend billing with the user 'admin', password '123qwe', and company id 1. Click on Orders and then Create, and select the user Penny Bright. Set the order period to "Monthly" and check off the "Main subscription" check-box to flag this order. Click continue, and add the "Start-Up Plan - 1000 monthly clicks" item to Penny's order. Follow through the remaining order screens and save your changes.

order with main subscription

Lets do the same for Terry Wilson. The difference with Terry is that he already has an order containing a subscription to the "Start-Up Plan - 1000 monthly clicks" item, but it was created before the "main subscription" preference was added so we'll need to edit the order to set the flag. Click on Orders, then click on the radio button beside order #12 (for twilson). Click the "Edit This Order" link on the left hand side, and check off "Main Subscription" flag. Click continue and follow through the order screens to save your changes.

Mediation Format Definition

The mediation reader is a plug-in who's sole purpose is to read data in from an external source and transform it into something that jBilling can work with. All reader plug-ins accept a mediation format definition as an XML file that describes the incoming data, allowing jBilling to make sense of it. The mediation XML format must conform to the provided mediation.dtd (jbilling/resources/mediation/), but otherwise it's composition is entirely up to you.

The mediation format is described by a <format> tag containing one or many <field> sections. Each <field> defines a named portion of data that will be parsed from the incoming event data. <field> elements can contain the following child elements:

<name>#CDATA</name>
<type>(string|integer|float|date)</type>
<startPosition>#CDATA</startPosition>
<length>#CDATA</length>
<durationFormat>#CDATA</durationFormat>
<isKey/>

As an example, the below XML defines a mediation format that will read the event id, event date, user name, item and quantity:

mediation-format.xml:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE format SYSTEM "mediation.dtd" >
<format>
    <field>
        <name>event_id</name>

        <type>string</type>
        <isKey/>
    </field>
    <field>
        <name>event_date</name>

        <type>date</type>
    </field>
    <field>
        <name>username</name>
        <type>string</type>

    </field>
    <field>
        <name>item_number</name>
        <type>string</type>
    </field>

    <field>
        <name>quantity</name>
        <type>integer</type>
    </field>
</format>

Which would work with this set of incoming event data (in CSV format):

mediation-sample.csv:
01,20100111-121559,pbright,B-03,120
02,20100111-121559,pbright,B-04,23
03,20100111-121559,twilson,B-03,10
04,20100111-121559,twilson,B-04,225

Note that the mediation format must define a set of Key fields that represent a unique key for a given row of data. This unique key is used by the mediation process to determine if a record has already been processed or not, allowing jBilling to filter out duplicate data. Key fields are marked using the <isKey/> element, if multiple fields are marked jBilling will group all of the key fields into a composite key. In the example above, we've created a event id for each row to use as our key. If your reading from a database, you may want to consider using an auto-incrementing column or sequence to generate a unique id for each row.

The above files (mediation-format.xml and mediation-sample.csv) should be placed in the jbilling/resources/mediation/ directory. As we'll see in a moment, it is possible to configure jBilling to look elsewhere for these files, but we'll stick with the default to make things easy.

Reader Plug-in

Reader plug-ins are slightly different from other jBilling plug-ins. For one, they don't care what processing order they're placed in. Secondly, jBilling doesn't attempt to look up a configured reader plug-in like it would with the other pluggable types; we have to tell the mediation configuration what reader plug-in to use when processing. This allows us to have multiple mediation configurations with separate readers, which can be extremely useful if you happen to have several sources of events.

Click on System then Plug-ins and add a new SeparatorFileReader for use with the mediation-format.xml and mediation-sample.csv files we created above.

reader plug-in

Add a format_file parameter with the value 'mediation-format.xml', by default the reader looks in jbilling/resources/mediation/ for the defined file.
Add a suffix parameter with the value 'mediation-sample.csv'. This parameter restricts the reader to files ending with the given suffix (again, in the jbilling/resources/mediation/ folder). Note that the suffix parameter does not accept RegEx or other matching patterns. If you wanted to read all CSV files in the directory, you could use a suffix of ".csv" - in this case we only want read in our crafted mediation-sample.csv file.

Only the format_file parameter is required. If omitted, the suffix parameter will default to "ALL" causing the reader to read in all files in the directory. It's always a good idea to provide a suffix parameter to ensure that the reader is getting the correct type of file, especially if you use editors like Emacs or Vim which can leave temp files behind.

Other optional file reader parameters include:

format_directory - The directory where the format file is located. Defaults to jbilling/resources/mediation/.
directory - The directory where event files are located, Defaults to jbilling/resources/mediation/.
rename - Boolean parameter (values 'true' or 'false'), causes the reader to rename event files after they have been processed by appending '.done' to the name of the file.
dateFormat - Defines the format the reader uses when converting 'date' type fields into Java Date values. Accepts formatting patterns used with the Java SimpleDataFormat class. defaults to 'yyyyMMdd-HHmmss'.
separator - Separator character used to separate each field in the event file. Defaults to comma (',').

Mediation Configuration

Take note of the ID of the configured reader plug-in, and click on System then Mediation. Click the "Add a new configuration" button and create a new mediation configuration using the reader plug-in ID.

mediation configuration

Once you've added a Mediation Configuration, the process becomes live. Mediation runs on the same schedule as the rest of the batch processes (billing processes, ageing, credit card expiry etc.). By default, you can expect to see the mediation process run 12 hours after the jBilling startup time. For tighter control of the batch process schedule, take a look at the User Guide and the jbilling.properties file.

Last but not least, you'll need to add a Mediation Process plug-in. The reader plug-ins job is simply to read in data, where as the Mediation Process plug-ins job is to make sense of it and add orders to jBilling. Without a configured Mediation Process plug-in, the reader is essentially talking to itself. It will read data in but none of it is going to make it into jBilling.

Luckily, the example Trend company already has the RulesMediationTask configured. Click on System then Plug-ins, find the RulesMediationTask in the list of plug-ins and note its configuration. Like all other rules tasks (from the Getting Started - BRMS guide) it accepts either a list of files or URLs as a parameter. We have the RulesMediationTask wired up to the DROOLS Guvnor to make management easier. As mentioned above, the mediation plug in takes all three rule packages (separated by spaces).

http://localhost:8080/drools-guvnor/org.drools.guvnor.Guvnor/package/trend_mediation/LATEST
http://localhost:8080/drools-guvnor/org.drools.guvnor.Guvnor/package/trend_item_management/LATEST
http://localhost:8080/drools-guvnor/org.drools.guvnor.Guvnor/package/trend_pricing/LATEST

Mediation Rules

All mediation rules operate on a MediationResult object that represents the processing of one row of data. The MediationResult object is designed to hold all the relevant information to the current row being processed, including the user, event date and the users current order (to which order items will be added).

We also have a PricingField object which represent an individual field from the row of data being processed. All PricingFields share a result id that link's them back to a MediationResult. These objects are the output of the configured reader plug-in, and are the data that you will be "parsing" with rules to resolve into items and order lines.

Now we can see how everything fits together.

The reader plug-in reads the mediation-sample.csv file
The mediation format definition is applied, each row is turned into a set of PricingField objects, each one representing a field as defined by the mediation format
A set of PricingField objects is passed off to the RulesMediationTask, where the set is grouped together with a MediationResult using a shared id.

For Example, A given row of data: 01,20100111-121559,pbright,B-03,120

Becomes:

MediationResult{
   id = 123,
   userId = ?,
   currentOrder = ?,
   eventDate = ?,
}

and

PricingField{ resultId = 123, name = "event_id", value = 123 }
PricingField{ resultId = 123, name = "event_date", value = 20100111-121559 }
PricingField{ resultId = 123, name = "username", value = "pbright" }
PricingField{ resultId = 123, name = "item_number",  value = "B-03" }
PricingField{ resultId = 123, name = "quantity",  value = 120 }

The MediationResult is pretty sparse when it is first created. User id and event date are both null to begin with, and thus there is no current order to operate on. This is because we rely on the PricingField objects to tell us what user we're operating on and what date this event is for. The first step to processing this row is to fill in the MediationResult.

Log into the DROOLS Guvnor (http://localhost:8080/drools-guvnor), Click on Knowledge Bases, open the 'trend_mediation' package, then click on Technical rule assets. There should be several rules grouped together in a "configuration" category. These rules are used instantiate the MediationResult object and are required by all mediation setups, although some alteration may be required for these rules to work with your mediation format. The example Trend company has the basics wired in for you - user setter, date setter and current order setter rules.

The user and date setter rule looks something like this:

rule 'date setter'
when
    $result : MediationResult(eventDate == null)
    $field : PricingField( name == "event_date", resultId == $result.id)
then
    modify( $result ) { setEventDate( $field.getDateValue() ); }
end

rule 'user setter'
when
    $result : MediationResult(userId == null)
    $field : PricingField( name == "username", resultId == $result.id)
    $company : CompanyDTO( ) # needed to determine a user by its user name
then
    Integer userId = getUserIdFromUsername($field.getStrValue(), $company.getId());
    modify( $result ) { setUserId( userId ); }       
end

We'll skip over the rest of the setter rules since they're all pretty much the same. If your developing a mediation process from scratch, it's worthwhile to look at the defaults here and use it as a starting point. In most situations these boilerplate "configuration" rules need little to no modification and can be left as-is.

Next, we'll need to add some rules to resolve our PricingFields into order lines. You'll need one of these rules for each item you intend to add through mediation.

rule 'resolve banner click and request price'
when
    $result : MediationResult( currentOrder != null, done == false )
    $quantity : PricingField( name == "quantity", resultId == $result.id)

    # Click on front page banner
    PricingField( name == "item_number", value == "B-03", resultId == $result.id) 

    not ( OrderLineDTO( itemId == 14 ) from $result.lines )
then
    $result.getLines().add(newLine(14, new BigDecimal($quantity.getStrValue())));
    update( $result );

    PricingResult request = priceRequest(14, $result);
    insert( request );
end

rule 'resolve generic banner click and request price'
when
    $result : MediationResult( currentOrder != null, done == false )
    $quantity : PricingField( name == "quantity", resultId == $result.id)

    # Click on front page banner - generic
    PricingField( name == "item_number", value == "B-04", resultId == $result.id) 

    not ( OrderLineDTO( itemId == 16 ) from $result.lines )
then
    $result.getLines().add(newLine(16, new BigDecimal($quantity.getStrValue())));
    update( $result );

    PricingResult request = priceRequest(16, $result);
    insert( request );
end

Remember that we configured the RulesMediationTask with all three rules packages (mediation, item management, pricing). For subscribers of the "Start-Up Plan - 1000 monthly clicks" item, generic clicks added through the mediation process will be converted to "included" items.

Running the Mediation Example Included with jBilling 2.1.1

The following are the instructions to run the sample Mediation example included with jBilling 2.1.1:

1. Unzip the distribution.
2. Open the file /jbilling/conf/jbilling.properties and change base_dir to /jbilling/resources/
(Note the ending '/' character. If you do not end the directory with a '/', this will not work. )
3. In the same file as above change process.frequency to 3. This step can be skipped if you are using a production deployment of jBilling.
4. Start Tomcat by running startup.sh on *nix or startup.bat on Windows. Login to the jBilling administration site at http://localhost:8080/billing/
5. Navigate to the Orders page where you'll see some old orders. After 3 minutes passes, refresh the browser and you'll see new orders generated by the mediation process.
6. Note that only one order was created for user twilson even though there are two records in the sample file ( /jbilling/resources/mediation/mediation-sample.csv). This is because twilson has an order marked as "Main Subscription".

Sequence Diagram of Mediation Steps

Going Further

Theres no reason to restrict the mediation process to orders and order lines. With some planing and consideration, you could add mediation configurations designed to process customer records, items, or anything within the jBilling domain. All of jBillings business logic (BL) classes can easily be made available to the rules engine through import statements. You can even use your own custom classes if need be.

Documentation

Latest News Items