WebSphere DataPower SOA Appliance performance tuning

Introduction

IBM WebSphere DataPower SOA Appliance is a purpose-built device which usually acts as a Gateway and/or Enterprise Service Bus (ESB) to help secure, accelerate, transform, enrich and route messages. This article provides the how-to knowledge of tuning WebSphere DataPower to reach the desired performance results.

The article is will go through the journey of performance tuning starting from profiling to tuning and finally performance testing prerequisites, approaches and some tips.

All information, steps and diagrams in this article are based on the WebSphere DataPower Integration Appliance XI50 device, firmware 3.8.0.1.

Back to top

Profiling

Before you start tuning WebSphere DataPower you need to gather some runtime statistics and data for your deployed solution.

Each part of your deployed code (XSLTs), configuration objects (ex:MQ Object), inbound/outbound connections and processing rules should be isolated and profiled to decide which part is most affecting the performance (that is using most of the device resources).

XSLTs

XSLTs is probably the most thing you will find affecting the performance since it is the place where you write your business (sometimes complex) transformation logic, specially if you are using url-open extension function.

WebSphere DataPower provides XSLTs profiling feature, below are the steps needed to enable profiling for all XSLT files executed in any call.

1. Go to Objects > XML Processing > XML Manager > default (unless you are using another one)
2. In the Main tab, choose a Compile Option Policy (or create another one)
3. Edit the selected Compile Option Policy
4. Choose the default (or create a new) Profile rule

After saving this new configuration and re-running your client application you can find the profiling results in the following location:

Open Status > XML Processing > Stylesheet Profiles

Figure 1. Profiled Stylesheets

As you can see, it is quite clear from both Count and Time columns which XSLT file is taking more time to execute, also note that the Time value aggregates the time consumed in all execution attempts (shown in Count).

Services

Service Objects such as Web Service Proxy or XML Firewall can be profiled as follows:

Enable Statistics
This step enables statistics collection, which will be very useful in profiling different objects.

1. Open Administration > Device > Statistics Settings
2. Enable it – if it is not enabled - and Click Apply

Viewing Profiling Information

Open Status > Connection > Transaction Times

Figure 2. Transaction Times

Now you can see the average transaction time for each of your services.

Message Flows

The message flows are categorized in 4 types:

1. Message: The time taken to process the request message received from client, plus processing time by the server, plus time taken to process the server response by the device (The full transaction cycle).
2. Request: The time taken by DataPower to process the request message before sending it to the server.
3. Server: The time taken by the server to process the request message sent to it by WebSphere DataPower.
4. Response: The time taken by WebSphere DataPower to process the message received from the server before sending it back to the client.

Follow the following steps to profile the flow of the message in the device:

Create Message Duration Monitor

1. Open Objects > Monitoring > Message Duration Monitor.
2. Click Add.
3. Choose the measure, which is one of the four types described above.
4. Complete the form and Apply.

View Statistics

1. Open Control Panel > Status.
2. Click on Messages (duration).
3. You will find the duration monitor that you have just created.

Figure 3. Message Duration Monitor

Connections

You can get a snapshot of inbound, outbound and internal TCP connections, you can also watch services listening on specific ports as shown in Figure 4.

Open Status > IP-Network > TCP Port Status or TCP Port Summary

Figure 4. TCP Port Status

Please note that more connections means more memory and also note that opening/closing a connection consumes CPU time.

Backend

In addition to running the performance test on WebSphere DataPower you can re-run the test directly on the backend and compare the latency in the two tests.

This will show if the latency is because of the device or the Backend, this can also be achieved by using a message duration monitor of type "server" as discussed in Message Flows section above.

Device Utilization

It is so beneficial to watch the device metrics (CPU, Memory and System Usage) under different situations. This is usually done via periodic SNMP polling in most environments. Monitoring System Load/Usage provides a better metric of appliance load than CPU usage.

CPU and Memory

Open Control Panel > View Status > System Information (memory and CPU usage)

Figure 5. Memory Usage

System Usage

The System Usage gives you an overview of the load on the Device including CPU, memory, pending messages in processing queue (Work List), file/connection handles and a total load indicator.

Open Status > System > System Usage

Figure 6. System Usage

Note: System Usage is more important and more accurate when compared with CPU usage, since CPU usage can show momentarily spikes while System Usage takes duration into account.

Back to top

Tuning

Once you have profiling information, you can create your tuning plan, the next section will discuss and describe the most important tuning steps that will directly affect the performance.

Caching

Caching is the most important option for performance tuning, think of the time used for network communication, retrieval of WSDL documents or compiling XSLT files.

WebSphere DataPower provides many options for caching, you can cache XSLT files, documents (specifically remote documents), WSRR WSDL retrieval, LDAP/TAM responses and others, below I will describe the steps to cache the objects mentioned earlier.

XSLTs and Documents

XSLs are cached by default. You can set the number of XSLs to be cached by specifying this option in the XML Manager:

Open Objects > XML Processing > XML Manager > Your XML Manager (or default)

Set XSL cache size to the desired number.

To cache documents, open the Document Cache tab in the XML Manager and set the Document Cache Count to an appropriate number andDocument Cache Size to an appropriate positive number.

To cache remote documents do the following:

1. Open the Document Cache Policy tab
2. Click Add
3. Enter the URL Match expression which is any URL either internal or external to the device (ex: WSRR REST Calls)
4. Complete the rest of the form
5. Apply and Save

Figure 7. Document Cache Policy

WSRR WSDL Retrieval

If you are using WSRR WSDL subscription, the WSDL Retrieval is cached by default, make sure you have selected the appropriate refresh interval.

1. Open Control Panel > Web Service Proxy > Your Proxy
2. Open WSDL tab
3. Expand you WSDL subscription
4. Enter an appropriate value for Refresh Interval (note: the default value is ok if you do not have any other preferences)

Figure 8. WSDL Refresh Interval

AAA Cache

In the AAA object, the authentication and authorization Cache is set to 3 seconds by default, unless you have a requirement or a logical reason for leaving it low (increasing cache time can lead to weaker security) it will be beneficial - from performance perspective - to increase this value.

Open Objects > XML Processing > AAA Policy > Your Policy

In the Authenticate tab increase the value of Cache Lifetime, in Authorize tab increase the same value.

Figure 9. TAM Authorization Cache Lifetime

HTTP Persistent Connections

HTTP Persistent connections means reusing opened connection for sending multiple messages instead of opening/closing a connection for each message, this option decreases CPU cycles, network traffic and latency. For more information check HTTP Persistent Connections in the Resources section.

WebSphere DataPower supports Persistent connections for inbound/outbound calls as part of HTTP/1.1 specs, it is enabled by default in HTTP related services such as HTTP front side handler, Web Service Proxy and XML Firewall, below is an example of how to control this option in the Web Service Proxy.

1. From the Control Panel Open the Web Service Proxy > Your Proxy
2. Open the Advanced Proxy Tab
3. You can set Persistent Connections "On" or "Off" and specify persistence timeout

Figure 10. Persistent connection settings

Processing Rules

Processing Rules are the flows which contains the processing nodes (Action Nodes) that will be applied on incoming messages.

PIPE and NULL Contexts

Whenever applicable use PIPE as Input and Output between two contiguous action nodes, this will prevent extra context processing and will keep memory usage down.

Use NULL as Input or Output of any action that doesn't need to produce or use predecessor actions' results (ex: using a Transform Action to set context variables only), this is beneficial because you will save the time of creating and parsing contexts and the action becomes more readable and self describing, plus the solution will consume less memory.

Asynchronous Rules
Limit the usage of asynchronous rules because as you can imagine parallel processing will increase CPU Utilization (Context Switching) and will use more memory, also note that this applies to asynchronous actions too.

Reusable Rules
Reusable rules are often called many times from parent rules, this may introduce redundancy in action node execution, when using reusable rules, make sure you include only actions that needs to be called many times, other actions (such as Transform Action to set a variable) may be moved to the parent flow to avoid redundancy.

XSLT Code
Code for performance from the beginning, below are some XSLT Performance tips.

1. Avoid using "//"; specify the node you are targeting using its absolute tree path
2. Do not retrieve the same information twice, if you are using dp:variable function or selecting a value from a document <xsl:value-of select="document(/path)"/> more than one time in the same file, it will be better to save it in a variable once and use the variable for later references

WebSphere MQ Queue Manager

When connecting to MQ, use "dpmq://" (which uses a configurable MQ Queue Manager Object) instead of direct MQ connection "mq://", using direct MQ connection will create a queue manager object at runtime and will open a new connection with MQ for each call, instead if you used theMQ Queue Manager Object you will have connection pooling and many other good options.

You can create the MQ Manager Object by doing the following:

1. Open Objects > Network Settings > MQ Queue Manager
2. Click Add, complete the form and Apply
3. You can then use the name of this object in "dpmq://" urls

Figure 11. MQ Queue Manager

Streaming

WebSphere DataPower supports many types of streaming such as execution streaming (using SAX when executing XSLTs), attachment streaming, message streaming and context streaming (as described in Processing Rules section when using PIPE).

These options provides many advantages, one of which is decreasing memory usage.

For more information about streaming in WebSphere DataPower, you can access the "Optimizing through Streaming" documentation in the Resources section.

Back to top

Performance testing

This section is a guide for conducting performance tests on WebSphere DataPower devices.

Prerequisites

1. Set the logging level to error, do not use debug
2. Disable any Probe
3. Make sure there are no other activities occurring in the device (may be in other domains)
4. Before running the test do some warm up calls, this is beneficial in-case you are using caching and also to be sure that the call is successful
5. Make sure the environment you are using is similar to that of your customer (specially Network speed and Device type)
6. If you are using XML Threat protection, make sure it does not block your test (because of suspecting XML-DOS Attack)

Testing approach

1. Use incremental concurrency approach that is starting with small number of threads say 10 and then increasing (30, 50, 80, etc), at a certain level the performance will start to degrade, this way you know the point where you get best performance results (the capacity of your solution).
2. Use appropriate thread delay(wait time) in-case you are testing with high concurrency, since by not doing so you are stressing the device and it will affect the performance
3. Use different scenarios and datasets in your test, it would be also better if they match real production scenarios
4. Try to re-run the test for a long period of time (Endurance Testing), since some problems like memory leaks may only appear in such test
5. When generating huge load, make sure you select the appropriate Testing tools and Operating Systems (ex: Linux or Windows Server) since some testing tools and OSs may not be capable of generating load to saturate the device

Testing tools

The below list is a sampling of testing tools available:

1. IBM Rational Performance Tester for SOA Quality
2. JMeter
3. SOAP UI
4. Apache Bench

Back to top

Tips and tricks

Firmware Updates/Tech-notes

Always check the latest firmware and tech-notes through IBM support site for WebSphere DataPower (found in Resources section), they may include fixes and workarounds for performance issues such as memory leaks.

Device Processing Queue

If the client application is sending a large number of messages to the device, and these messages are processed and sent to the server asynchronously and the server wasn't tuned enough to consume these messages with the rate that WebSphere DataPower process and sends them; this may lead to a situation which is similar to Memory Leak as the device queue the messages in memory until the server is able to consume them.

The solution for this situation is to tune the server to accept more messages at any given point in time.

Back to top

Conclusion

Throughout this article you have learned how to Profile, Tune and Test WebSphere DataPower for Performance, in-case you are using features and objects other than those described in the article; you can still use the provided techniques and examples to reach your tuning objectives.