Implementing SLM(Service Level Monitoring) in Datapower

SLM has the ability to monitor traffic and take action if traffic patterns meet certain criteria. For example, consider the case where you have a backend that can only process 300 transactions per second (TPS). SLM can be used to take action when more than 300 TPS inbound transactions are received.

DataPower can be used to ensure this capacity is not exceeded or to send notifications to ensure that any overage is appropriately billed.

DataPower SLM can take action at a transaction threshold in three ways:

Notify: DataPower can send a notification to its logging system via any protocol supported by the device including e-mail, WebSphere MQ, and HTTP. For example, in the case where the REST customer has met its capacity, DataPower could send a notification to a business process that notifies the customer.

Shape: DataPower can delay traffic in an attempt to keep the transaction rate at the given threshold. Take the example where the backend can only process 300 TPS. Here, shaping can be used to queue any transactions that exceed that rate.

Throttle: DataPower can reject transactions that exceed the given threshold.

SLM is supported on multiple appliances that share load with SLM peer groups. The appliances in a peer group continuously communicate with each other to ensure SLM thresholds are shared across the entire group.

A peer group is a collection of DataPower appliances that exchange service level data to enforce SLM policies in the group. A peer group establishes a data-sharing protocol. The traffic that each appliance in the group processes is passed to the other peers to calculate whether a threshold is reached.

Check that all the criteria below is met for creating SLM Peer Group:

1.     All the DataPower Appliances must be clock-synchronized. You can use the NTP service.

2.  The XML Management Interface has to be enabled on both devices.

3. Ensure that there is no firewall in between the appliances blocking the port for the XML Management Interface.

4. Review the XML Management Interface configuration and check the advanced tab, to ensure, the default user agent is not used, or if used, that it has been set up with an SSL Proxy Profile.

5. SLM Peer Groups communicate over XML Management Interface (XMI), and XMI uses SOAP over HTTPS to exchange data. If specifying the “Custom User Agent” property of the XMI and you have an “SLM Peer Group” configured, you must make sure to have an SSL Proxy Profile defined in the “Custom User Agent”.

6. If this was not the case, the outbound request with the “Custom User Agent” property set is not processed by the other appliance in the “SLM Peer Group”.

7. To exchange SLM data at periodic interval, peer members must have identical SLM configurations, including the members list.

8. Ensure that the URLs in the peer group are correct

9. When using hostnames in the peer group URLs, make sure you have a working DNS configuration

10. The SLM policy needs to be identical on both appliances. If they do not match, the peering will not work.

11. SLM Interval length agreement:

An SLM Policy allows administrators to set a threshold level interval set on the SLM Policy Statements page.

When an SLM Policy being enforced by more than one device, administrators must also set a SLM update Interval on the SLM tab of the XML Management Interface configuration page found only in the default domain.

The SLM update Interval must be equal to or less than the threshold interval set in any SLM Policy statement in use on the device. If not, the updates deliver outdated information to the individual policies and are ignored, defeating the purpose of peering.

12. Use the same packet transmission method: unicast or multicast.

13. SLM unicast peering

SLM unicast peering uses unicast packet transmission to update peers in a connected manner.

Consider SLM unicast peering when:

1.     DataPower appliances in the peer group are not physically close or not connected within the same sub network.

2.     The rate of incoming data is low or moderate.

3.     The data sharing interval equals to or is more than one second.

14. SLM multicast peering

SLM multicast peering uses IP multicast packet transmission to update peers in a connectionless manner. Multicast peering supports small threshold intervals and allows data sharing intervals of less than one second.

Consider SLM multicast peering when:

1.     DataPower appliances in the peer group are physically close and connected within the same sub network.

2.     The rate of incoming data is high.

3.     The data sharing interval is less than one second, and you need higher SLM peering accuracy.

4.     The peering traffic and the data traffic use separate networks.

To define a peer group using SLM Unicast:

1.     Click Objects → Network → Peer Groups.

2.     Click Add.

3. In the Name field, enter the name for the configuration.

4. Set Administrative State to identify the administrative state of the configuration.

5. Optional: In the Comments field, enter a descriptive summary.

6. From the Type list, select the type of SLM peering for packet transmission.

7. For SLM unicast peering, specify the IP unicast configuration and update interval to exchange data among peers.

In the URL field, enter the URL of each peer, including the local appliance, and click Add.

8. Click Apply to save the changes to the running configuration.

9. Optional: Click Save Config to save the changes to the startup configuration.

Enable probe in DataPower

Login to DP console  >> > control panel

Click on required services -à Click on Services name

Click on Show Probe button à click on Enable probe.

SOAP Web Service Mockup tutorial on DataPower

It is often the case that a DataPower developer will have work to do, but does  not yet have a backend Service Provider to point the DataPower service at.  In general, anything beyond trivial examples will need to point at something that returns a valid response.  In order to satisfy this requirement, an XML Firewall can be setup in loopback mode that uses a custom stylesheet  to generate a valid response.  This tutorial will demonstrate how to create this XML Firewall.

For our example, we are going to return a valid SOAP response for thisWSDL.

1. Log into the DataPower appliance and navigate to the Control Panel screen if not already there.

1. Click on the XML Firewall icon.

1. Click the “Add Wizard” button.

1. Chose “Pass Thru (Testing Only) radio button.
2. Click the Next button.

1. Type a name for this XML Firewall in the text field.  For this tutorial, we will use MathWS.
2. Click the Next button.

1. Chose “loopback-proxy” from the drop-down menu.
2. Click the Next  button.

1. In the Device Address field, either a valid IP address or Host Alias must be added.  For this example, a Host Alias called PublicInterface, which maps to the public IP address used by one of the ethernet interfaces on this appliance.
2. The Device Port field should have a valid TCP port.  For this example, we are using 10002.
3. For now, SSL will not be used in this example.  So, keep the “off” radio button checked for the last parameter.
4. Click the Next button.

1. This screen lets us review the settings we have chosen in this wizard.
2. Click the Commit button.

1. Click the Done button.

1. Click the save link in the upper, right-hand corner.

1. Click on the XML Firewall icon again.

1. Click on the MathWS link.

1. In the right-hand column, at the bottom, choose “XML” from the Request Type dropdown menu.
2. Click the Apply button.
3. Click the Advanced tab link.

1. In the left-hand column, click the off radio button under “Disallow GET (and HEAD)”. 
2. In the right-hand column, click the on radio button under “Process Messages Whose Body IS Empty”.
3. Click Apply.
4. Click the save config link.

1. Next, we need to create an XSLT stylesheet that will return a static response to SOAP queries.  As a starting point, copy the contents of the stylesheet given in this tutorial into your favorite XML editor.
2. So, you should now have the following in your XML editor:

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:dp="http://www.datapower.com/extensions"
    xmlns:dpfunc="http://www.datapower.com/extensions/functions"
    xmlns:dpconfig="http://www.datapower.com/param/config"
    xmlns:func="http://exslt.org/functions"
    extension-element-prefixes="dp func"
    exclude-result-prefixes="dp dpfunc dpconfig func"> 
   
    <xsl:template match="/">  
       
    </xsl:template>
</xsl:stylesheet>

Next, we need to generate a valid SOAP response.   This can be done with SOAPUI.

Download the WSDL to your local file system.

Download SOAPUI or a similar tool.

Launch SOAPUI.

 

Click File->New SOAPUI Project.

Enter “MathWS” in the Project Name field.

Click the browse button.

Choose the MathWS.wsdl file that was selected earlier.

Click Ok.

After some processing, a project will be created.

Expand the MathWS->MathWSSoapBinding->sum->Request 1 tree.

Double-click on the Request 1 node.

You will see a request message with ‘?’s in place of values.

Click  the SOAP icon (third icon from the left) on the Request 1 window.

The following pop up box will appear.

Click the Yes button.

Enter a name for the Mock Service.

Click the Ok button.

When prompted for “Open MockResponse editor?”, click the Yes button.

We now have a valid SOAP response that can be added to our new stylesheet.

Copy the text in the Response1 window into the <template> of the stylesheet.

Replace the ‘?’ with ‘6’ or your favorite number.

The stylesheet should now look like:

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:dp="http://www.datapower.com/extensions"
    xmlns:dpfunc="http://www.datapower.com/extensions/functions"
    xmlns:dpconfig="http://www.datapower.com/param/config"
    xmlns:func="http://exslt.org/functions"
    extension-element-prefixes="dp func"
    exclude-result-prefixes="dp dpfunc dpconfig func"> 
   
    <xsl:template match="/"> 
        <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"xmlns:ws="http://ws.rcbj.com/">
            <soapenv:Header/>
            <soapenv:Body>
                <ws:sumResponse>
                    <return>6</return>
                </ws:sumResponse>
            </soapenv:Body>
        </soapenv:Envelope>
    </xsl:template>
</xsl:stylesheet>

Save this file as MathWS-Response.xslt on your local file system.

Navigate back to the DataPower Control Panel in your browser.

Click on the File Management icon.

Click the Actions link next to local:///.

Choose the “Create Subdirectory” option.

Enter a meaningful directory name.  This tutorial will use “example”.

Click the “Confirm Create” button.

Expand the local: directory. 

You should see a subdirectory called example.

Click the “Actions.” link next to the example subdirectory.

Chose “Upload Files” link.

Click the “Choose File” button.

Select the MathWS-Response.xslt file.

The “Save As:” field should be automatically filled in with “MathWS-Response.xslt”.

Select the “Overwrite Existing Files” checkbox.

Click Upload button.

Click the Continue button.

Navigate back to the MathWS XML Firewall configuration screen.

Click on the “.” button below “XML Firewall Policy” on the right-hand side.

Expand this window to give yourself more room to work.

Find the round icon in the large window towards the bottom marked “Transform”.

Drag this Transform Action icon onto the Request Rule between the diamond shaped Match Action and the square shaped Response Action.

Double-click the the yellow highlighted Transform Action.

From the Transform drop-down menu that currently shows “local:///”, chose “local:///example”.

From the file drop down directly below it, chose the MathWS-Response.xslt file.

All other fields can be left as their default values.

Click the Done button.

Click the “Apply Policy” button.

Click the Close Window link.

Click the Apply button.

Click the Save Config link.

Now, return to the SOAPUI project that was created earlier.

Go to the “Request 1″ screen.

We need to replace the two ‘?’ with integers.  I am going to use ‘2’ and ‘4’.  So, the request now looks like:

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"xmlns:ws="http://ws.rcbj.com/">
   <soapenv:Header/>
   <soapenv:Body>
      <ws:sum>
         <arg0>2</arg0>
         <arg1>4</arg1>
      </ws:sum>
   </soapenv:Body>
</soapenv:Envelope>

Modify the service URL to behttp://dp2.rcbjconsulting.com:10002/service/MathWS (choose edit current from the drop down menu).  Replace the URL with whatever is a valid URL for your appliance.

Click the Execute button.

A response containing the following should come back:

<soapenv:Envelope xmlns:ws="http://ws.rcbj.com/"xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
   <soapenv:Header/>
   <soapenv:Body>
      <ws:sumResponse>
         <return>6</return>
      </ws:sumResponse>
   </soapenv:Body>
</soapenv:Envelope>

You have successfully constructed a “dummy service” that can be used in the absence of a legitimate Service Provider implementation.

XI50 to XI50 Data Power Migration

Here are steps: 
1 Check and upgrade the firmware version and level of DataPower XI50 to that of the firm-ware version and level of the DataPower XI52. 
2 Secure backup XI50 
4 Secure backup XI52; this might be useful in case if you need to rollback to original con-figurations. Also export configurations from XI52; this would be used later for referring for resetting various configurations. 
5 Validate and then restore DataPower XI52 from the secure backup files from the Data-Power XI50 
6 Access DataPower XI52 via remote console and update 
a. IP interface to have its own IPs. The Ethernet Interface names are different on XI52 appliances; need to reconfigure the eth settings 
b. host alias 
c. system info 
d. ip routing 

7 Test restored Datapower XI52 
a. That the DataPower is accessible using its own IPs. Test login through WebGUI and CLI. 
b. Compare objects on the DataPower XI52 against the objects on the DataPower XI50 that they are up and running. 

WebSphere DataPower SOA Appliance performance tuning

Introduction

IBM WebSphere DataPower SOA Appliance is a purpose-built device which usually acts as a Gateway and/or Enterprise Service Bus (ESB) to help secure, accelerate, transform, enrich and route messages. This article provides the how-to knowledge of tuning WebSphere DataPower to reach the desired performance results.

The article is will go through the journey of performance tuning starting from profiling to tuning and finally performance testing prerequisites, approaches and some tips.

All information, steps and diagrams in this article are based on the WebSphere DataPower Integration Appliance XI50 device, firmware 3.8.0.1.

Back to top

Profiling

Before you start tuning WebSphere DataPower you need to gather some runtime statistics and data for your deployed solution.

Each part of your deployed code (XSLTs), configuration objects (ex:MQ Object), inbound/outbound connections and processing rules should be isolated and profiled to decide which part is most affecting the performance (that is using most of the device resources).

XSLTs

XSLTs is probably the most thing you will find affecting the performance since it is the place where you write your business (sometimes complex) transformation logic, specially if you are using url-open extension function.

WebSphere DataPower provides XSLTs profiling feature, below are the steps needed to enable profiling for all XSLT files executed in any call.

1. Go to Objects > XML Processing > XML Manager > default (unless you are using another one)
2. In the Main tab, choose a Compile Option Policy (or create another one)
3. Edit the selected Compile Option Policy
4. Choose the default (or create a new) Profile rule

After saving this new configuration and re-running your client application you can find the profiling results in the following location:

Open Status > XML Processing > Stylesheet Profiles

Figure 1. Profiled Stylesheets

As you can see, it is quite clear from both Count and Time columns which XSLT file is taking more time to execute, also note that the Time value aggregates the time consumed in all execution attempts (shown in Count).

Services

Service Objects such as Web Service Proxy or XML Firewall can be profiled as follows:

Enable Statistics
This step enables statistics collection, which will be very useful in profiling different objects.

1. Open Administration > Device > Statistics Settings
2. Enable it – if it is not enabled - and Click Apply

Viewing Profiling Information

Open Status > Connection > Transaction Times

Figure 2. Transaction Times

Now you can see the average transaction time for each of your services.

Message Flows

The message flows are categorized in 4 types:

1. Message: The time taken to process the request message received from client, plus processing time by the server, plus time taken to process the server response by the device (The full transaction cycle).
2. Request: The time taken by DataPower to process the request message before sending it to the server.
3. Server: The time taken by the server to process the request message sent to it by WebSphere DataPower.
4. Response: The time taken by WebSphere DataPower to process the message received from the server before sending it back to the client.

Follow the following steps to profile the flow of the message in the device:

Create Message Duration Monitor

1. Open Objects > Monitoring > Message Duration Monitor.
2. Click Add.
3. Choose the measure, which is one of the four types described above.
4. Complete the form and Apply.

View Statistics

1. Open Control Panel > Status.
2. Click on Messages (duration).
3. You will find the duration monitor that you have just created.

Figure 3. Message Duration Monitor

Connections

You can get a snapshot of inbound, outbound and internal TCP connections, you can also watch services listening on specific ports as shown in Figure 4.

Open Status > IP-Network > TCP Port Status or TCP Port Summary

Figure 4. TCP Port Status

Please note that more connections means more memory and also note that opening/closing a connection consumes CPU time.

Backend

In addition to running the performance test on WebSphere DataPower you can re-run the test directly on the backend and compare the latency in the two tests.

This will show if the latency is because of the device or the Backend, this can also be achieved by using a message duration monitor of type "server" as discussed in Message Flows section above.

Device Utilization

It is so beneficial to watch the device metrics (CPU, Memory and System Usage) under different situations. This is usually done via periodic SNMP polling in most environments. Monitoring System Load/Usage provides a better metric of appliance load than CPU usage.

CPU and Memory

Open Control Panel > View Status > System Information (memory and CPU usage)

Figure 5. Memory Usage

System Usage

The System Usage gives you an overview of the load on the Device including CPU, memory, pending messages in processing queue (Work List), file/connection handles and a total load indicator.

Open Status > System > System Usage

Figure 6. System Usage

Note: System Usage is more important and more accurate when compared with CPU usage, since CPU usage can show momentarily spikes while System Usage takes duration into account.

Back to top

Tuning

Once you have profiling information, you can create your tuning plan, the next section will discuss and describe the most important tuning steps that will directly affect the performance.

Caching

Caching is the most important option for performance tuning, think of the time used for network communication, retrieval of WSDL documents or compiling XSLT files.

WebSphere DataPower provides many options for caching, you can cache XSLT files, documents (specifically remote documents), WSRR WSDL retrieval, LDAP/TAM responses and others, below I will describe the steps to cache the objects mentioned earlier.

XSLTs and Documents

XSLs are cached by default. You can set the number of XSLs to be cached by specifying this option in the XML Manager:

Open Objects > XML Processing > XML Manager > Your XML Manager (or default)

Set XSL cache size to the desired number.

To cache documents, open the Document Cache tab in the XML Manager and set the Document Cache Count to an appropriate number andDocument Cache Size to an appropriate positive number.

To cache remote documents do the following:

1. Open the Document Cache Policy tab
2. Click Add
3. Enter the URL Match expression which is any URL either internal or external to the device (ex: WSRR REST Calls)
4. Complete the rest of the form
5. Apply and Save

Figure 7. Document Cache Policy

WSRR WSDL Retrieval

If you are using WSRR WSDL subscription, the WSDL Retrieval is cached by default, make sure you have selected the appropriate refresh interval.

1. Open Control Panel > Web Service Proxy > Your Proxy
2. Open WSDL tab
3. Expand you WSDL subscription
4. Enter an appropriate value for Refresh Interval (note: the default value is ok if you do not have any other preferences)

Figure 8. WSDL Refresh Interval

AAA Cache

In the AAA object, the authentication and authorization Cache is set to 3 seconds by default, unless you have a requirement or a logical reason for leaving it low (increasing cache time can lead to weaker security) it will be beneficial - from performance perspective - to increase this value.

Open Objects > XML Processing > AAA Policy > Your Policy

In the Authenticate tab increase the value of Cache Lifetime, in Authorize tab increase the same value.

Figure 9. TAM Authorization Cache Lifetime

HTTP Persistent Connections

HTTP Persistent connections means reusing opened connection for sending multiple messages instead of opening/closing a connection for each message, this option decreases CPU cycles, network traffic and latency. For more information check HTTP Persistent Connections in the Resources section.

WebSphere DataPower supports Persistent connections for inbound/outbound calls as part of HTTP/1.1 specs, it is enabled by default in HTTP related services such as HTTP front side handler, Web Service Proxy and XML Firewall, below is an example of how to control this option in the Web Service Proxy.

1. From the Control Panel Open the Web Service Proxy > Your Proxy
2. Open the Advanced Proxy Tab
3. You can set Persistent Connections "On" or "Off" and specify persistence timeout

Figure 10. Persistent connection settings

Processing Rules

Processing Rules are the flows which contains the processing nodes (Action Nodes) that will be applied on incoming messages.

PIPE and NULL Contexts

Whenever applicable use PIPE as Input and Output between two contiguous action nodes, this will prevent extra context processing and will keep memory usage down.

Use NULL as Input or Output of any action that doesn't need to produce or use predecessor actions' results (ex: using a Transform Action to set context variables only), this is beneficial because you will save the time of creating and parsing contexts and the action becomes more readable and self describing, plus the solution will consume less memory.

Asynchronous Rules
Limit the usage of asynchronous rules because as you can imagine parallel processing will increase CPU Utilization (Context Switching) and will use more memory, also note that this applies to asynchronous actions too.

Reusable Rules
Reusable rules are often called many times from parent rules, this may introduce redundancy in action node execution, when using reusable rules, make sure you include only actions that needs to be called many times, other actions (such as Transform Action to set a variable) may be moved to the parent flow to avoid redundancy.

XSLT Code
Code for performance from the beginning, below are some XSLT Performance tips.

1. Avoid using "//"; specify the node you are targeting using its absolute tree path
2. Do not retrieve the same information twice, if you are using dp:variable function or selecting a value from a document <xsl:value-of select="document(/path)"/> more than one time in the same file, it will be better to save it in a variable once and use the variable for later references

WebSphere MQ Queue Manager

When connecting to MQ, use "dpmq://" (which uses a configurable MQ Queue Manager Object) instead of direct MQ connection "mq://", using direct MQ connection will create a queue manager object at runtime and will open a new connection with MQ for each call, instead if you used theMQ Queue Manager Object you will have connection pooling and many other good options.

You can create the MQ Manager Object by doing the following:

1. Open Objects > Network Settings > MQ Queue Manager
2. Click Add, complete the form and Apply
3. You can then use the name of this object in "dpmq://" urls

Figure 11. MQ Queue Manager

Streaming

WebSphere DataPower supports many types of streaming such as execution streaming (using SAX when executing XSLTs), attachment streaming, message streaming and context streaming (as described in Processing Rules section when using PIPE).

These options provides many advantages, one of which is decreasing memory usage.

For more information about streaming in WebSphere DataPower, you can access the "Optimizing through Streaming" documentation in the Resources section.

Back to top

Performance testing

This section is a guide for conducting performance tests on WebSphere DataPower devices.

Prerequisites

1. Set the logging level to error, do not use debug
2. Disable any Probe
3. Make sure there are no other activities occurring in the device (may be in other domains)
4. Before running the test do some warm up calls, this is beneficial in-case you are using caching and also to be sure that the call is successful
5. Make sure the environment you are using is similar to that of your customer (specially Network speed and Device type)
6. If you are using XML Threat protection, make sure it does not block your test (because of suspecting XML-DOS Attack)

Testing approach

1. Use incremental concurrency approach that is starting with small number of threads say 10 and then increasing (30, 50, 80, etc), at a certain level the performance will start to degrade, this way you know the point where you get best performance results (the capacity of your solution).
2. Use appropriate thread delay(wait time) in-case you are testing with high concurrency, since by not doing so you are stressing the device and it will affect the performance
3. Use different scenarios and datasets in your test, it would be also better if they match real production scenarios
4. Try to re-run the test for a long period of time (Endurance Testing), since some problems like memory leaks may only appear in such test
5. When generating huge load, make sure you select the appropriate Testing tools and Operating Systems (ex: Linux or Windows Server) since some testing tools and OSs may not be capable of generating load to saturate the device

Testing tools

The below list is a sampling of testing tools available:

1. IBM Rational Performance Tester for SOA Quality
2. JMeter
3. SOAP UI
4. Apache Bench

Back to top

Tips and tricks

Firmware Updates/Tech-notes

Always check the latest firmware and tech-notes through IBM support site for WebSphere DataPower (found in Resources section), they may include fixes and workarounds for performance issues such as memory leaks.

Device Processing Queue

If the client application is sending a large number of messages to the device, and these messages are processed and sent to the server asynchronously and the server wasn't tuned enough to consume these messages with the rate that WebSphere DataPower process and sends them; this may lead to a situation which is similar to Memory Leak as the device queue the messages in memory until the server is able to consume them.

The solution for this situation is to tune the server to accept more messages at any given point in time.

Back to top

Conclusion

Throughout this article you have learned how to Profile, Tune and Test WebSphere DataPower for Performance, in-case you are using features and objects other than those described in the article; you can still use the provided techniques and examples to reach your tuning objectives.