WebSphere DataPower SOA Appliance performance tuning

Introduction

IBM WebSphere DataPower SOA Appliance is a purpose-built device which usually acts as a Gateway and/or Enterprise Service Bus (ESB) to help secure, accelerate, transform, enrich and route messages. This article provides the how-to knowledge of tuning WebSphere DataPower to reach the desired performance results.

The article is will go through the journey of performance tuning starting from profiling to tuning and finally performance testing prerequisites, approaches and some tips.

All information, steps and diagrams in this article are based on the WebSphere DataPower Integration Appliance XI50 device, firmware 3.8.0.1.

Back to top

Profiling

Before you start tuning WebSphere DataPower you need to gather some runtime statistics and data for your deployed solution.

Each part of your deployed code (XSLTs), configuration objects (ex:MQ Object), inbound/outbound connections and processing rules should be isolated and profiled to decide which part is most affecting the performance (that is using most of the device resources).

XSLTs

XSLTs is probably the most thing you will find affecting the performance since it is the place where you write your business (sometimes complex) transformation logic, specially if you are using url-open extension function.

WebSphere DataPower provides XSLTs profiling feature, below are the steps needed to enable profiling for all XSLT files executed in any call.

1. Go to Objects > XML Processing > XML Manager > default (unless you are using another one)
2. In the Main tab, choose a Compile Option Policy (or create another one)
3. Edit the selected Compile Option Policy
4. Choose the default (or create a new) Profile rule

After saving this new configuration and re-running your client application you can find the profiling results in the following location:

Open Status > XML Processing > Stylesheet Profiles

Figure 1. Profiled Stylesheets

As you can see, it is quite clear from both Count and Time columns which XSLT file is taking more time to execute, also note that the Time value aggregates the time consumed in all execution attempts (shown in Count).

Services

Service Objects such as Web Service Proxy or XML Firewall can be profiled as follows:

Enable Statistics
This step enables statistics collection, which will be very useful in profiling different objects.

1. Open Administration > Device > Statistics Settings
2. Enable it – if it is not enabled - and Click Apply

Viewing Profiling Information

Open Status > Connection > Transaction Times

Figure 2. Transaction Times

Now you can see the average transaction time for each of your services.

Message Flows

The message flows are categorized in 4 types:

1. Message: The time taken to process the request message received from client, plus processing time by the server, plus time taken to process the server response by the device (The full transaction cycle).
2. Request: The time taken by DataPower to process the request message before sending it to the server.
3. Server: The time taken by the server to process the request message sent to it by WebSphere DataPower.
4. Response: The time taken by WebSphere DataPower to process the message received from the server before sending it back to the client.

Follow the following steps to profile the flow of the message in the device:

Create Message Duration Monitor

1. Open Objects > Monitoring > Message Duration Monitor.
2. Click Add.
3. Choose the measure, which is one of the four types described above.
4. Complete the form and Apply.

View Statistics

1. Open Control Panel > Status.
2. Click on Messages (duration).
3. You will find the duration monitor that you have just created.

Figure 3. Message Duration Monitor

Connections

You can get a snapshot of inbound, outbound and internal TCP connections, you can also watch services listening on specific ports as shown in Figure 4.

Open Status > IP-Network > TCP Port Status or TCP Port Summary

Figure 4. TCP Port Status

Please note that more connections means more memory and also note that opening/closing a connection consumes CPU time.

Backend

In addition to running the performance test on WebSphere DataPower you can re-run the test directly on the backend and compare the latency in the two tests.

This will show if the latency is because of the device or the Backend, this can also be achieved by using a message duration monitor of type "server" as discussed in Message Flows section above.

Device Utilization

It is so beneficial to watch the device metrics (CPU, Memory and System Usage) under different situations. This is usually done via periodic SNMP polling in most environments. Monitoring System Load/Usage provides a better metric of appliance load than CPU usage.

CPU and Memory

Open Control Panel > View Status > System Information (memory and CPU usage)

Figure 5. Memory Usage

System Usage

The System Usage gives you an overview of the load on the Device including CPU, memory, pending messages in processing queue (Work List), file/connection handles and a total load indicator.

Open Status > System > System Usage

Figure 6. System Usage

Note: System Usage is more important and more accurate when compared with CPU usage, since CPU usage can show momentarily spikes while System Usage takes duration into account.

Back to top

Tuning

Once you have profiling information, you can create your tuning plan, the next section will discuss and describe the most important tuning steps that will directly affect the performance.

Caching

Caching is the most important option for performance tuning, think of the time used for network communication, retrieval of WSDL documents or compiling XSLT files.

WebSphere DataPower provides many options for caching, you can cache XSLT files, documents (specifically remote documents), WSRR WSDL retrieval, LDAP/TAM responses and others, below I will describe the steps to cache the objects mentioned earlier.

XSLTs and Documents

XSLs are cached by default. You can set the number of XSLs to be cached by specifying this option in the XML Manager:

Open Objects > XML Processing > XML Manager > Your XML Manager (or default)

Set XSL cache size to the desired number.

To cache documents, open the Document Cache tab in the XML Manager and set the Document Cache Count to an appropriate number andDocument Cache Size to an appropriate positive number.

To cache remote documents do the following:

1. Open the Document Cache Policy tab
2. Click Add
3. Enter the URL Match expression which is any URL either internal or external to the device (ex: WSRR REST Calls)
4. Complete the rest of the form
5. Apply and Save

Figure 7. Document Cache Policy

WSRR WSDL Retrieval

If you are using WSRR WSDL subscription, the WSDL Retrieval is cached by default, make sure you have selected the appropriate refresh interval.

1. Open Control Panel > Web Service Proxy > Your Proxy
2. Open WSDL tab
3. Expand you WSDL subscription
4. Enter an appropriate value for Refresh Interval (note: the default value is ok if you do not have any other preferences)

Figure 8. WSDL Refresh Interval

AAA Cache

In the AAA object, the authentication and authorization Cache is set to 3 seconds by default, unless you have a requirement or a logical reason for leaving it low (increasing cache time can lead to weaker security) it will be beneficial - from performance perspective - to increase this value.

Open Objects > XML Processing > AAA Policy > Your Policy

In the Authenticate tab increase the value of Cache Lifetime, in Authorize tab increase the same value.

Figure 9. TAM Authorization Cache Lifetime

HTTP Persistent Connections

HTTP Persistent connections means reusing opened connection for sending multiple messages instead of opening/closing a connection for each message, this option decreases CPU cycles, network traffic and latency. For more information check HTTP Persistent Connections in the Resources section.

WebSphere DataPower supports Persistent connections for inbound/outbound calls as part of HTTP/1.1 specs, it is enabled by default in HTTP related services such as HTTP front side handler, Web Service Proxy and XML Firewall, below is an example of how to control this option in the Web Service Proxy.

1. From the Control Panel Open the Web Service Proxy > Your Proxy
2. Open the Advanced Proxy Tab
3. You can set Persistent Connections "On" or "Off" and specify persistence timeout

Figure 10. Persistent connection settings

Processing Rules

Processing Rules are the flows which contains the processing nodes (Action Nodes) that will be applied on incoming messages.

PIPE and NULL Contexts

Whenever applicable use PIPE as Input and Output between two contiguous action nodes, this will prevent extra context processing and will keep memory usage down.

Use NULL as Input or Output of any action that doesn't need to produce or use predecessor actions' results (ex: using a Transform Action to set context variables only), this is beneficial because you will save the time of creating and parsing contexts and the action becomes more readable and self describing, plus the solution will consume less memory.

Asynchronous Rules
Limit the usage of asynchronous rules because as you can imagine parallel processing will increase CPU Utilization (Context Switching) and will use more memory, also note that this applies to asynchronous actions too.

Reusable Rules
Reusable rules are often called many times from parent rules, this may introduce redundancy in action node execution, when using reusable rules, make sure you include only actions that needs to be called many times, other actions (such as Transform Action to set a variable) may be moved to the parent flow to avoid redundancy.

XSLT Code
Code for performance from the beginning, below are some XSLT Performance tips.

1. Avoid using "//"; specify the node you are targeting using its absolute tree path
2. Do not retrieve the same information twice, if you are using dp:variable function or selecting a value from a document <xsl:value-of select="document(/path)"/> more than one time in the same file, it will be better to save it in a variable once and use the variable for later references

WebSphere MQ Queue Manager

When connecting to MQ, use "dpmq://" (which uses a configurable MQ Queue Manager Object) instead of direct MQ connection "mq://", using direct MQ connection will create a queue manager object at runtime and will open a new connection with MQ for each call, instead if you used theMQ Queue Manager Object you will have connection pooling and many other good options.

You can create the MQ Manager Object by doing the following:

1. Open Objects > Network Settings > MQ Queue Manager
2. Click Add, complete the form and Apply
3. You can then use the name of this object in "dpmq://" urls

Figure 11. MQ Queue Manager

Streaming

WebSphere DataPower supports many types of streaming such as execution streaming (using SAX when executing XSLTs), attachment streaming, message streaming and context streaming (as described in Processing Rules section when using PIPE).

These options provides many advantages, one of which is decreasing memory usage.

For more information about streaming in WebSphere DataPower, you can access the "Optimizing through Streaming" documentation in the Resources section.

Back to top

Performance testing

This section is a guide for conducting performance tests on WebSphere DataPower devices.

Prerequisites

1. Set the logging level to error, do not use debug
2. Disable any Probe
3. Make sure there are no other activities occurring in the device (may be in other domains)
4. Before running the test do some warm up calls, this is beneficial in-case you are using caching and also to be sure that the call is successful
5. Make sure the environment you are using is similar to that of your customer (specially Network speed and Device type)
6. If you are using XML Threat protection, make sure it does not block your test (because of suspecting XML-DOS Attack)

Testing approach

1. Use incremental concurrency approach that is starting with small number of threads say 10 and then increasing (30, 50, 80, etc), at a certain level the performance will start to degrade, this way you know the point where you get best performance results (the capacity of your solution).
2. Use appropriate thread delay(wait time) in-case you are testing with high concurrency, since by not doing so you are stressing the device and it will affect the performance
3. Use different scenarios and datasets in your test, it would be also better if they match real production scenarios
4. Try to re-run the test for a long period of time (Endurance Testing), since some problems like memory leaks may only appear in such test
5. When generating huge load, make sure you select the appropriate Testing tools and Operating Systems (ex: Linux or Windows Server) since some testing tools and OSs may not be capable of generating load to saturate the device

Testing tools

The below list is a sampling of testing tools available:

1. IBM Rational Performance Tester for SOA Quality
2. JMeter
3. SOAP UI
4. Apache Bench

Back to top

Tips and tricks

Firmware Updates/Tech-notes

Always check the latest firmware and tech-notes through IBM support site for WebSphere DataPower (found in Resources section), they may include fixes and workarounds for performance issues such as memory leaks.

Device Processing Queue

If the client application is sending a large number of messages to the device, and these messages are processed and sent to the server asynchronously and the server wasn't tuned enough to consume these messages with the rate that WebSphere DataPower process and sends them; this may lead to a situation which is similar to Memory Leak as the device queue the messages in memory until the server is able to consume them.

The solution for this situation is to tune the server to accept more messages at any given point in time.

Back to top

Conclusion

Throughout this article you have learned how to Profile, Tune and Test WebSphere DataPower for Performance, in-case you are using features and objects other than those described in the article; you can still use the provided techniques and examples to reach your tuning objectives.

Integrating Websphere Datapower, Message Broker, MQ and Transformation Extende

How to integrate Websphere datapower XI50 with Websphere Message Broker and MQ in one flow.
Firstly, create a new project in Message Broker.
Add 3 Nodes:
1. An MQInput Node
2. An MQOutput Node
3. And a HTTP Request Node
Connect the 3 nodes to look like the following flow below:

For the MQInput and Output Node specify an existing MQ queue from which a message or file will be sent and received. If files want to be sent over MQ use MQ FTE (File Transfer Edition)
For the Message Broker HTTP Request Node, specify the Datapower Multi Protocol Frontside Handler. 
Datapower setup:



HTTP Frontside Handler:


The HTTP Request URL will look like follow: http://IPAddressOfDP:8080
Change ‘IPAddressofDP’ to the address of your DP IP address.
When executing the Message Broker flow send a message to the Input Queue (MQInput) the message is sent to Datapower through the HTTP Request which does message transformation from Cobol to XML.
The Datapower Transformation Looks like follow:



The transformation Map was created and uploaded to Datapower using Websphere Transformation Extender Design Studio:



The reason for offloading the transformation to Datapower XI50 is for the speed of transformation and increase of flow speed.

XML Firewall,WSP and MPGW in datapower

XML Firewall:

The XML Firewall is designed to process generic XML requests and responses transmitted over HTTP or HTTPS. The XML Firewall uses a single protocol and contains one processing policy with a set of request, response, two-way, and error rules. Its configuration defines the listening IP address-port pair as well as general threat protection.Although the design of the XML Firewall is to process XML documents of all types, including SOAP-formatted messages, it can accept unprocessed (text/binary) documents. Through the processing policy, the XML Firewall can apply all of the various processing actions to the request and response message, regardless of format. Processing can include AAA, transformations, schema validation, logging, and cryptographic operations.Like all other DataPower services, the XML Firewall can be configured to proxy remote services. However, one of the commonly used features of the XML Firewall is to define the configuration as a loopback service. As a loopback service, the remote server is undefined. In this case, the service itself generates a response to the client after processing the request. This functionality is useful when developing, testing, and debugging a service when the remote server is unavailable.

Multi-Protocol Gateway:

The Multi-Protocol Gateway is a powerful and versatile service. In additional to threat protection and document processing capabilities, the Multi-Protocol Gateway can process requests between various protocols. The supported protocols are HTTP,HTTPS, WebSphere MQ, WebSphere JMS, IMS™, FTP, NFS, SFTP, and TIBCO EMS.The Multi-Protocol Gateway receives incoming requests, processes them with a processing policy, and forwards the request to the remote server. The Multi-Protocol Gateway processes the response similarly, applying the applicable response rule, if configured.The Multi-Protocol Gateway uses front-side handlers to manage client connections.A single Multi-Protocol Gateway can have multiple front-side handlers that listen or poll for requests. The ability of configuring multiple front-side handlers allows a Multi-Protocol Gateway to receive requests from different protocols. For example, a

Multi-Protocol Gateway can have one front-side handler listening for HTTP requests and another handler polling a WebSphere MQ queue for messages. Both front-side handlers forward the incoming message to the Multi-Protocol Gateway for processing and forwarding to the remote server.All of the available protocols on which the Multi-Protocol Gateway can receive incoming requests can also be used on the server-side to forward the request to its destination. The client-side protocol does not need to match the server-side

protocol.A Multi-Protocol Gateway service offers many of the same services and capabilities

as a Web Service Proxy service. Unlike a Web Service Proxy service, a Multi-Protocol Gateway service cannot use a WSDL to determine a configuration.

Web Service Proxy:

The Web Service Proxy provides security and abstraction for remote Web services.By loading a WSDL file and adding a front-side handler, a Web Service Proxy is ready to start receiving requests. Although this configuration is simplistic, it is a fully functioning, feature-rich service that provides endpoint/URI abstraction,

parser-based XML threat protection, XML well-formedness checking, SOAP schema validation, payload schema validation, hooks for monitoring, and a platform for building operation-level rules.The WSDL file provides critical information about a Web service, including endpoint locations, instruction on binding to these endpoints, and expected message schemas. With just the WSDL and a front-side handler, a Web Service

Proxy has the basic configuration. Additional configuration can be defined to meet your case requirements, such as AAA, document transformations, message encryption, and so forth.In addition to the Web Service Proxy being able to upload or fetch a WSDL file, the configuration of a Web Service Proxy can be through a subscription to a UDDI registry or WSRR server. Through subscription, the Web Service Proxy receives

automatic updates of the WSDL file or dynamically looks up the endpoints for the service.The Web Service Proxy has powerful monitoring and logging capabilities. Web services traffic that flows through a Web Service Proxy can be monitored and logged at the service level down to the WSDL operation level. Other features of a Web Service Proxy are service-level monitoring (SLM), WS-ReliableMessaging, WS-Policy, and WS-Addressing.

Generate Keys and Certificates in Datapower

You can generate a private cryptographic key and optionally a self-signed certificate from the Crypto Tools page. The Certificate Signing Request (CSR) needed by a certificate authority (CA) is created by default.

If the file is stored in the cert: directory, it cannot be edited. If a file is stored in the local: directory or in the temporary: directory, it can be edited.

To generate a key:

1. Click Administration → Miscellaneous → Crypto Tools.
2. Define the LDAP entry.
   1. Set LDAP (reverse) Order of RDNs to indicate whether to create the LDAP entry in reverse RDN order.onCreates the entry in reverse RDN order.off(Default) Creates the entry in forward RDN order.
   2. Optional: In the Country Name (C) field, enter a country name.
   3. Optional: In the State or Province (ST) field, enter a state name or a province name.
   4. Optional: In the Locality (L) field, enter a locality name.
   5. Optional: In the Organization (O) field, enter the name of an organization.
   6. Optional: In the Organizational Unit (OU) field, enter the name of an organizational unit.
   7. Optional: In the Organizational Unit 2 (OU), Organizational Unit 3 (OU), and Organizational Unit 4 (OU) fields, enter the names of additional organizational units.
   8. In the Common Name (CN) field, enter a common name.
3. From the RSA Key Length list, select the key length. This defaults to 1024.
4. In the File Name field, enter the name of the key file to generate. The value takes the directory:///name form. Leave blank to allow the action to create the name.
5. In the Validity Period field, enter the number of days that the key is valid.
6. In the Password field, enter a password to access the key file. The password must be at least six characters in length.
7. In the Password Alias field, enter a password alias to access the key file.
8. |On HSM-equipped appliances, set Private Key Exportable via hsmkwk to indicate |whether the key can be exported with the HSM key-wrapping-key. |The default value is off.|
   |Note:||On Type 7199 appliances, |you must select on or the operation |will fail. The ability to do a subsequent export of the key cannot |be disabled.|
   |
   |on|Indicates that the key can be exported.|
   |off|(Default) Indicates that the key cannot be exported.|
   |
9. Set Export Private Key to indicate whether the action writes the key file to the temporary: directory.onWrites the key file to the temporary: directory.off(Default) Does not write the key file to the temporary: directory.
10. Set Generate Self-Signed Certificate to indicate whether the action creates a self-signed certificate that matches the key.on(Default) Creates a self-signed certificate.offDoes not create a self-signed certificate.
11. Set Export Self-Signed Certificate to indicate whether the action writes the self-signed certificate to the temporary: directory.on(Default) Writes the self-signed certificate to the temporary: directory.offDoes not write the self-signed certificate to the temporary: directory.
12. Set Generate Key and Certificate Objects to indicate whether the action automatically creates the objects from the generated files.on(Default) Creates the objects from the generated files.offDoes not create the objects from the generated files.
13. In the Object Name field, enter the name to use for the Key object and for the Certificate object. Leave blank to allow the action to generate the names from the input information (based on the Common Name (CN) or File Name property).
14. On HSM-equipped appliances, set Generate Key on HSM to indicate whether to create the key on the HSM.|on|Creates the key on the HSM.|On Type 9235 appliances, |the file name (URL) for the key has the hsm://hsm1/name format.|On Type 7199 appliances, the file name (URL) for the |key has the hsm://hsm2/name format.offCreates the key on the appliance. The file name (URL) for the key has the cert:///name format.
15. In the Using Existing Key Object field, enter the name of an existing key. If supplied and valid, the action generates a new certificate and a new Certificate Signing Request (CSR) that is based on the key in the identified Key object. In this case, the appliance does not generate a new key.
16. Click Generate Key to generate a private key and, if requested, a self-signed certificate. A CSR is created automatically.
17. Follow the prompts.

The CSR can be submitted to a certificate authority (CA) to receive a certificate that is based on this private key. This action creates the following files and objects:

• Creates the private key file in the cert: directory; for example, cert:///sample-privkey.pem
• Creates the CSR in the temporary: directory; for example, temporary:///sample.csr
• If Generate Self-Signed Certificate is enabled, creates a self-signed certificate in the cert: directory; for example,cert:///sample-sscert.pem
• If Export Self-Signed Certificate is enabled, creates a copy of the self-signed certificate in the temporary: directory; for example, temporary:///sample-sscert.pem
• If Generate Key and Certificate Objects is enabled, creates a Key object and a Certificate object

If the action creates a self-signed certificate, you can use this certificate-key pair for the following purposes:

• Establish Identification Credentials
• Encrypt or decrypt XML documents