Introduction
Why do we need benchmarks for SOAP?
Who will benefit?
Publications
Benchmark Suite
Download Test Driver and Run Benchmark
Performance Results
Future Work
Contact

Introduction

The SOAP protocol has emerged as the standard mechanism for exchanging information between various Web services. Web services have been adopted as the underlying architecture for Grid computing. Consequently, many applications will transport data via SOAP. The advantages of SOAP include extensibility, robustness and interoperability, but not performance. SOAP implementations have been developed in many languages including C, C++, Java, Python and C#. These toolkits differ in their design and supported performance optimizations.

We have designed a benchmark suite to compare and quantify the performance of various toolkits. Based on our experience of developing three different, independent toolkits (gSOAP, XSOAP/XSUL and bSOAP), we propose a benchmark suite of SOAP benchmarks that have been designed to exercise the performance of each toolkit for a wide range of representative use patterns.

Why do we need SOAP benchmarks?

1. The requirements of Web services based applications is diverse. As a result, toolkits are designed to incorporate various features including end-to-end performance, serialization and deserialization efficiency, small memory footprint, specific security requirements and scalability. The benchmark suite will help decide which toolkit is best suited for a particular application.
2. SOAP presents some unique design and implementation challenges which require smart techniques. The naive approach can lead to unacceptable performance penalties. The benchmark is designed to highlight clever techniques.
3. Many SOAP toolkits are currently being developed, in addition to the large number of existing ones.

Who Will Benefit?

Our work is designed to benefit both SOAP library developers and application programmers who use SOAP based Web services. We provide many insights into the design and development of SOAP toolkits thereby guiding SOAP developers in building toolkits with enhanced performance. Web service users can compare and contrast the performance and features of various toolkits, based on our benchmark suite, and select the one that best suites their needs.

Publications

• Michael R. Head, Madhusudhan Govindaraju, Aleksander Slominski, Pu Liu, Nayef Abu-Ghazaleh, Robert van Engelen, Kenneth Chiu, Michael J. Lewis, "A Benchmark Suite for SOAP-based Communication in Grid Web Services," To appear in SC|05 (Supercomputing): International Conference for High Performance Computing, Networking, and Storage, Seattle WA, November 2005. [Download]

Benchmark Suite

We propose a benchmark suite consisting of WSDL files with operations to test the performance for various features, including:

• Serialization, Deserialization and End to End performance for the following data types:
  • Doubles: performance for various array sizes of floating point data.
  • Strings: performance for various sizes of String data.
  • Integers: performance for various sizes of integer data.
• Base64: Compares base64 encoding of to standard XML encoding. To quantify this advantage, for all types and sizes, we have included it in the benchmark suite for serialization, deserialization, and end-to-end performance measurement
• Candidate Features for Optimization
  • Streaming and Chunking : The steaming benchmark measures the performance of serialization, deserialization, and end-to-end performance when chunking and streaming is used with a persistent TCP/IP connection, compared to the case when it is turned off. For small messages, the cost of repeatedly establishing network connections can significantly impact performance. This benchmark quantifies the exact performance benefit of using streaming
• Latency of a toolkit: the overhead imposed by a toolkit to send and receive a SOAP call when no parameters have to be serialized or deserialized.
• Namespace benchmark : It is important to test the performacne of toolkits for SOAP payloads with varying level of nested data structures (linked lists). For this benchmark several of the tag names and attribute names in each level should be identical, forcing toolkits to correctly resolve them according to the namespace qualifications. The payloads should also have a varying number of namespace qualified attributes.
• Multi-ref: performance in the presence of a large number of co-referenced objects.

This WSDL file contains most of the benchmarks we used for the performance tests.

Test Driver

This source file is a sample implementation of the server bits of the benchmark for the Axis Java server.

Performance Results

All the SOAP Benchmark result graphs are posted here.

This Excel file contains all the results from the tests as run on the Windows platform.

Below are the individual result graphs from the tests selected for the submitted paper.

• echoBase64: This plot compares the end-to-end response time when echoing base64 encoded arrays of various sizes.
• echoDoubles: This plot compares the end-to-end response time when echoing variously sized arrays of doubles.
• echoInts: This plot compares the end-to-end response time when echoing variously sized arrays of ints.
• echoMeshInterfaceObjects: This plot compares the end-to-end response time when echoing variously sized arrays of MeshInterfaceObjects, which contain two ints and a double, representing a data structure commonly used in scientific computing.
• echoSimpleEvents-win32: We compare end-to-end (serialization and deserialization) for .NET, AxisJava, and XSUL on Windows. XSUL's performance drops for handling more than 10,000 events. For sizes greater than 12,000, AxisJava takes an average of 50\% more time than .NET to respond.
• receiveDoubles: Here we compare the deserialization performance of AxisJava, gSOAP and XSUL. Each toolkit is sent a SOAP payload for double arrays of various sizes, asked to deserialize it and return it's size to the driver. For arrays of 10,000 doubles, AxisJava takes 5.1 times as long to respond as XSUL, which in turn takes 7.8 times as long to respond as gSOAP.
• receiveDoubles-win32: This graph compares deserialization performance of Axis, XSUL and .NET for an array of doubles on Windows. XSUL and .NET are comparable, while AxisJava does not scale well for large array sizes. For an array of 10,000 doubles, AxisJava's deserialization time is 6 times greater than those of XSUL and .NET.
• receiveSimpleEvents: This graph compares the performance for the deserialization benchmark for events. Each event object contains an integer, a double and a string. XSUL's performance degrades considerably when it deserializes more than 15,000 elements, but for lesser number of events, it outperforms AxisJava. gSOAP is orders of magnitude faster in handling complex types compared to the Java toolkits.
• receiveSimpleEvents-streaming: We studied the effect of streaming by making AxisJava and gSOAP deserialize a large number of events with and without streaming enabled. AxisJava consistently performs better when streaming is used, taking 22% less time to complete for the 10,000 element array. gSOAP also trims its time with chunking on this test by 22%.
• receiveStrings: This graph compares the deserialization peformance for strings. Again, XSUL peforms significantly better than AxisJava for deserialization. For an array size of 25,000, it takes AxisJava 7.4 times longer to respond than XSUL, and XSUL takes 3.6 times as long to respond as gSOAP.
• sendBase64: This graphs shows the performance of AxisJava, gSOAP and XSUL for serializing data in Base64 format. AxisJava performs poorly compared to XSUL and gSOAP. XSUL takes 49% more time to complete than gSOAP for arrays of 25,000 elements.
• sendDoubles: Here we compare the serialization performance of four toolkits on arrays of doubles. As with the integer case in Figure \ref{fig:serInts}, the Java-based toolkits (AxisJava and XSUL) perform similarly. bSOAP and gSOAP send the largest size (25,000 doubles) arrays in 13% and 30%, respectively, of the time it takes for AxisJava to do the same.
• sendDoubles-diff-ser: Differential serialization is optimized for use-cases when subsequent sends are similar. In this benchmark. we measured the performance of bSOAP when 0% (bSOAP0), 25% (bSOAP25), 50% (bSOAP50), 75% (bSOAP75), and 100% (bSOAP100) of the values need to be reserialized, rather than copied, for each message from the previous message. As expected, bSOAP's optimizations reduce the serialization time as the percentage of values that need to be reserialized is reduced. For 100,000 doubles, the best case bSOAP0 is faster than the worst case bSOAP100 by a factor of 6.7.
• sendDoubles-streaming: We studied the effect of streaming by making AxisJava and gSOAP serialize arrays of doubles with and without streaming enabled. Unlike in the case of deserialization of events, streaming does not improve AxisJava's serialization performance. Streaming helps improve gSOAP's performance for all sizes up to 25,000 elements, the maximum we tested for this benchmark. By enabling streaming, gSOAP's average response time for 10,000 elements was reduced by 42%.
• sendInts: This graph shows the effect on serialization when the size of an integer array is scaled. The performance of AxisJava and XSUL is similar. bSOAP and gSOAP complete the benchmark in 4% and 17% respectively of the time it takes for AxisJava for the largest size (25,000 integers).

Future Work

We plan to add to the benchmark suite to test the following features:

• Memory footprint of toolkits and suitabidtty for hand-held devices
• Compliance and performance for emerging Web services based security features

Contact

For more information, please contact: