Benchmarking on the Cloud – Your Definitive Check List!

Posted 13 May 2009 @ 12:03 pm by Shay Hassidim

Introduction

Evaluating a new software product usually involves running benchmark tests many times, to assess the different capabilities of the product using some measurable elements. Generally, the end result of such tests is a matrix that includes the product latency, throughput or scalability values.

Such a matrix compares the product data feed, data retrieval or data processing speed, as a function of the payload size, number of concurrent users, or other factors.

Since such tests should be conducted in an isolated environment, it is necessary to construct an environment that contains all the relevant artifacts to accommodate everything that you need to achieve fair and correct benchmark results.

This blog post aims to provide you with a full and comprehensive check list to review before you start your benchmark project, and to suggest the Cloud environment as the ultimate environment to run accurate and fair benchmarks.

The Benchmark Check List

Dedicated Machines

This is the most expensive component, and the most important artifact required for your benchmark tests. If this is not already available, it is unlikely that you would get a budget for purchasing this. Most of the components you use are multi-threaded processes (database, IMDG, feeders) that utilize multiple cores, and are sensitive to the CPU response time. This means you must have some reasonable balance between the amount of clients and backend servers, and the amount of cores that are serving these processes. It does not make any sense to share the same machines with other processes that are not part of the benchmark - if you do that, there is a very good chance that you will get distorted results.

If you are planning a large scale benchmark to demonstrate throughput and scalability, you might need a relatively high number of machines. This means a large budget for machines that would be utilized for a very short period of time only, and that are not necessarily needed for the organization in any immediate production systems. In today's economy (not to mention the green era we are part of) - it simply does not make any sense to ask for such a budget, and purchase such a machine for a benchmark project.

On the other hand, a Cloud environment ensures that you get the exact machine profile that you need, with the exact CPU power and cores you have requested, and it ensures that you are the sole user of those machines. These are available to you around the clock, as long as you are running your benchmark. You can start with a very minimal number of machines, and add more as needed. With a single click you can spin hundreds or even thousands that number of machines without any down payment, or the need to purchase any hardware. You pay per use, exactly according to the amount of machine hours you utilize.

GigaSpaces provides special cloud images that have been fully configured to run applications that are deployed into the GigaSpaces scale-out application server, when using the GigaSpaces cloud computing framework (CCF4XAP). When running your benchmark, these machines can serve you and save you a substantial amount of time.

Dedicated Network

The benchmark tests do not run on a single machine. There needs to be several machines involved, forming a distributed system. This means the network robustness, latency and bandwidth have a greater impact on the benchmark results. Any network hiccup or noise could generate distorted results. If you have other systems utilizing the network, this means you might not be able to leverage the network to its maximum capacity to serve your system. In such a case, having a dedicated network is critical for having correct and good benchmark results.

By comparison, a Cloud environment runs the most advanced network devices, and can provide a very high quality of service in terms of the network connectivity, latency and bandwidth between the machines running within the cloud. Every cloud provider is committed to providing excellent networking facilities, allowing the machines to communicate with each other without disruption, abnormal disconnections or non-deterministic behavior. You get consistently steady connectivity, allowing you to run your benchmark without compromising on latency or network bandwidth.

Dedicated Database

You might choose to use a database to eventually store your application data. Reporting and any other offline systems access the database for their own purpose. When running throughput or latency benchmark tests, there is a good chance that you will not persist the data synchronously into the database, but go for asynchronous persistency (i.e. Mirror Service or Write-Through). With this architecture, the impact of the database performance is relatively minimal. However, it is still important to have a dedicated database server (serving the Mirror Service or the application itself for lazy load i.e. Read-Through) running on a dedicated machine. Moving the database activity out of the critical path of the application flow, means removing a major bottleneck you might face when running your benchmark. This allows the database server to utilize the machine CPU, Memory and Network in a very efficient manner.

A Cloud environment guarantees that no one is using your database server when your benchmark is running. You have a dedicated machine running only your database server, fully tuned and optimized to access the disk or the storage device maintained on the cloud in the most efficient manner. Every cloud provider runs the most advanced storage devices, allowing the database machines to perform IO operations at an incredible speed.

GigaSpaces provides the mySQL database image when using the CCF4XAP. A simple configuration as part of the cloud application deployment file starts a dedicated machine that runs the mySQL database server which you can use with your benchmark.

Dedicated Feeders

The feeders that push data into the system, greatly impact the throughout, scalability and latency benchmark results. Your feeders must run on dedicated machines, have multi-core CPU machines, and have fast CPUs, to be able to push the data into the backend system as fast as they can. Slow feeders or feeders that are shared with other systems without ability to scale linearly provide very distorted results. Many times once you analyze the benchmark results and bad results are revealed, people look for bottlenecks at the backend system (IMDG , Services , database), but the real bottleneck in the flow, is in fact the feeder component.

In a Cloud environment, you have dedicated feeders that push the data into the backend system without the need to share these with other systems. You can spin new machines as needed, to host additional feeders to push in unlimited amounts of data any rate you wish into the backend system, and generate a load that can simulate a large amount of concurrent users.

GigaSpaces provides the jakarta-jmeter tool as part of the images provided with the CCF4XAP. You can use these to load the system in a very simple manner.

Large Shared Memory Pool

One of the central components of your benchmark is the application Shared Memory - or the In-Memory-Data-Grid. This distributed service stores the application data objects in-memory. It is highly available, transactional, fault-tolerant, and resilient, with the ability to self heal if a failure occurs. This must use machines with large memory and fast CPUs, with fast network access. In some cases it might run collocated business logic that processes the incoming data.

A Cloud environment can provide your machines with plenty of RAM, running your IMDG across large numbers of machines. You can run the IMDG with a practically unlimited amount of memory, and test the system in the most extreme situations. Such a large amount of RAM can save you the need to access data residing on disk, which is by far slower than accessing data in memory. The IMDG and its collocated business logic provides faster data processing, due to the fact that data does not need to travel into the business logic with every call over the network. It is simply hosted within the same process memory address as the business logic.

GigaSpaces provides an IMDG (a clustered JavaSpaces implementation) when using the CCF4XAP. The IMDG supports different cluster topologies, persistency, query, remoting, map-reduce and many more options.

Coordinate Specific Date and Time

In many cases, the benchmark environment is not available on a regular basis within your organization. There is a need to schedule with the IT department a specific date and time to allow you to access the relevant lab machines, install relevant software components, configure the system, and run your benchmark. This very important fact requires very careful planning, and massive coordination between different teams within the organization. This can therefore be a very time consuming, irritating and annoying task.

A Cloud environment is available for you around the clock, so you don't need to schedule or coordinate your benchmark runs with anyone. You also don't need to do any planning with any other team to free the machines you need. You have practically unlimited time to evolve the benchmark as you want, and to provide the most accurate and impressive results your system can produce.

Install Relevant Software Components

Your benchmark environment involves many different software components. This includes the operating system itself, the software for the database and IMDG, the benchmark software, the feeders' software, the measurement software, the simulators, the management and provisioning software and any other third party software. Installing all these can take days, if not weeks. It does not make sense to go through this cycle with every benchmark project that you have. You should have these installed once only, to be reused for any future benchmark you or your team members run.

A Cloud environment allows you to create your own machine image(s) that have all the relevant components installed, fully configured, and tuned to accommodate your benchmark. You can place in the cloud repository, the different software components versions and download these into the started machines as part of their bootstrap phase. Such a setup produces a very dynamic and flexible environment that you can use to run your benchmarks without the hassle of installing the benchmark components on each machine separately, or re-creating the benchmark environment with each benchmark project. You will have your benchmark environment ready to roll forever and ever.

GigaSpaces provides several basic components of this kind, such as OS monitoring, management and provisioning tools, as part the CCF4XAP. You can collect statistics about the running machines and analyze the behavior of the system in real time or offline.

Tune the Machines OS

Tuning the OS involves a huge amount of experience. Sometimes you might need very special expertise with the OS network, memory and thread scheduling behavior. It is very time consuming to perform the same tuning task repeatedly on each machine running the benchmark (not to mention performing this for each benchmark project).

By comparison, a Cloud environment provides you with the choice to select any OS to run your benchmark. Any cloud provider can have an ‘out-of-the-cloud' set of pre-installed machines that you can use as a basis for your own image. These machines have their OS fully tuned by the best experts out there, avoiding the need for you to deal with such a sensitive and complex task yourself.

GigaSpaces provides such a tuned OS (Linux-based, and in the future also Windows and Solaris OS) when using the CCF4XAP. The image includes OS tuned network settings, and additional tuned parameters with which you can boost the overall performance of the machine.

Tune the Machines JVM

For many non-experienced Java users, tuning the JVM is almost voodoo. There are so many options and variations that it makes it an almost impossible task to get the correct combination to provide the best performance when running your benchmark. Having a set of JVM arguments to be used with the relevant benchmark scenario (large payload, low-latency, highly concurrent) is highly important, and saves time when running your benchmarks.

In a Cloud Environment, once you have your JVM configured, you can place these settings in the cloud repository, and reuse them whenever you run your benchmark. Then, there is no way that you can lose them, or somehow use the wrong JVM settings by mistake.

GigaSpaces provides such tuned JVM settings for the JVM running the GigaSpaces grid infrastructure (GSM, GSC) when using the CCF4XAP, and allows you to choose the JVM version.

Tune the Database

Database servers include a variety of options. These animals require special expertise and everyone knows how it is difficult to find a proper DBA that can make your database servers fly. Sometimes having the relevant indexing set or proper connection settings, can make a huge difference with your benchmark results. No one really wants to go through this experience again and again with every benchmark project.

In a Cloud Environment, any cloud provider offers a ready to use machine image, with a database fully tuned and configured. Just pick the one that fits your needs, and run it.

GigaSpaces provides such an image for mySQL, as part of the CCF4XAP. Application data stored within the IMDG can persist into the database in a synchronous or asynchronous mode.

Setup the Benchmark Configuration

Benchmark projects tend to have a set of scenarios they are aiming to test. For each scenario there might be different set of configurations you use. Applying the right configuration is essential for getting the correct benchmark results. Since you might run the tests several times it is important to make sure you run exactly the same configuration each time.

GigaSpaces provides an out-of-the-box benchmark framework you can use. This benchmark framework supports all common scenarios such as latency, throughput, and scalability tests, and can produce results that you can analyze and graph in a very simple manner. The benchmark source code is available and you can modify it to accommodate your exact needs. Benchmark frameworks are available for Java, .Net and C++ applications.

Linear Scalability Test

Linear Scalability is one of the most difficult benchmarks to execute. A scalability benchmark starts by running the system with a very minimal set of machines, establishing a baseline result that can act as the unit of scalability. This baseline result could be: using X amount of feeders, using Y amount of IMDG instances, providing Z latency or throughput. To measure the scalability of the system we increase the amount of feeders and IMDG in a linear manner, and expect the latency to stay constant and the throughput to increase in a linear manner.

To accomplish the above you need to start additional machines to host the additional feeders, IMDG instances and the services that are processing the incoming data, where each machine should be identical to the ones that were used with the baseline tests.

A Cloud environment provides you with the ability to run exactly the same machine profile as much as you need, and stretch the system to include a large number of machines with very minimal effort.
The CCF4XAP allows you to deploy a baseline setup of your benchmark, and increase the number of machines running the benchmark without shutting down the grid. Newly started machines join the grid running on the cloud, allowing you to increase the number of feeders, IMDG or services, and run another cycle of the benchmark to generate additional data points for your linear scalability measurements.

Iterative Process and Automation

Running benchmarks is not a once-off task. It is an iterative process that can take a long time, where each phase involves studying the benchmark results, changing a single parameter and re-running the tests. Automating this process saves a lot of time and makes sure you get correct results. You don't want to sit in front of the machines day and night, changing the scripts with some value, redeploying, and re-running the tests manually. You should have all of this done in a totally automatic manner.

Cloud environments provide a powerful API that you can use to start your machines, shut them down, and control the full life cycle of the machines in a very simple manner. GigaSpaces abstracts these APIs via a high level API and scripts you can use. You can form very sophisticated benchmark scenarios that start machines, deploy the benchmark framework, run it, collect the results and analyze these, all without intervention. These can run around the clock, in parallel manner, where each script can run a different scenario with different parameters, using a different set of machines on the cloud. With such an execution framework, you can complete your benchmark very quickly.

A Cloud environment is the perfect environment to run Benchmarks

Reviewing the check list above shows how difficult it is to run reasonable benchmarks within your in-house lab environment - especially when you want to run scalability tests. The number of machines involved, the complex configuration and the long list of variable factors enforce the use of a public hosting facility - i.e. the cloud.

This isolated independent environment has unlimited resources, such as identical machines, and a totally dedicated secured network you can use to run your benchmark. Using a cloud saves you the need to construct a new environment each time you run your benchmark project, and makes sure you get the most accurate results.

In some cases, such an environment can mimic your production environment, having the exact machine profiles you have in production, the same network settings and other elements that you use in your production environment. In other cases, the same cloud that has been used to run the benchmark can be used for the production environment. This provides your benchmark with a higher viability and justification, and predicts the real system behavior in a very accurate manner. This is obviously a key component for a successful launch and satisfied customers (and also a happy boss  ).

Simple Data Grid Benchmark on the Cloud

A simple Data Grid benchmark setup can be found here. Just follow the instructions and see how your Data-Grid runs. You may change the benchmark configuration , the Data Grid topology and re-run the benchmark very easily.

For more information about running benchmarks on the cloud, contact us at: cloud@gigaspaces.com

Shay Hassidim
GigaSpaces Deputy CTO