Are you Really using the Fastest and Most Scalable Cache with your Application?

We are hearing lately reports from several accounts "discovering" that Terracotta ehcache is not the fastest and most scalable cache out there. GigaSpaces local cache actually performs much better.

With the benchmark results below we see GigaSpaces Cache is about 8 times faster than echache when having multiple threads accessing the cache.


 
This benchmark test EHCache most popular use case against GigaSpaces cache. This benchmark is not about demonstrating GigaSpaces entire spectrum of functionality (elasticity , eventing , SQL query , Remoting , Map-Reduce, noSQL support, ESB support, CEP support, Cloud enabling , PaaS…) . The focus here is on a simple caching scenario many users test when evaluating caching products.

You can download the source code and run this benchmark yourself. With a small effort you can use any other cache implementation and compare it with GigaSpaces Cache. The benchmark is self-contained. It includes the source code, compiled code as well the ehCache 2.5 libraries.

What the Benchmark is Doing?

1. The benchmark will first write 100 objects into the cache.
2. Then, it will to read these repeatedly for 10 seconds.
3. After 10 seconds an additional thread will be started performing the read activity for 10 seconds until there will be 8 threads running. The total read benchmark time is 80 seconds.
4. Each thread calculating its average read time and a final result is displayed after each iteration with the total average read performance for all threads.
 

How can I run the Benchmark myself?

1. Download GigaSpaces XAP , extract the zip file and apply the license file provided as part of the email sent to you after the download.

2. Download the benchmark package and extract the attached into an empty folder.

3. Move into the localcacheVsehCacheBench folder.

4. Rename *.at files to *.bat if you are running on windows.

5. Edit the setExampleEnv.bat/setExampleEnv.sh to have the correct location of the JSHOMEDIR variable as GigaSpaces root folder.

6. To run the ehCache benchmark run: runBenchmarkehCache.bat/sh.

7. To run the GigaSpaces benchmark run: runBenchmarkGigaSpaces.bat/sh.
 

Expected Output

The expected output for ehCache benchmark should look like this:

Terracotta ehCache benchmark started!
1 Client threads started
——— Average TP for all 1 threads:5556916 read/sec———————
2 Client threads started
——— Average TP for all 2 threads:1523256 read/sec———————
3 Client threads started
——— Average TP for all 3 threads:1144619 read/sec———————
4 Client threads started
——— Average TP for all 4 threads:1036346 read/sec———————
5 Client threads started
——— Average TP for all 5 threads:998566 read/sec———————
6 Client threads started
——— Average TP for all 6 threads:907827 read/sec———————
7 Client threads started
——— Average TP for all 7 threads:893828 read/sec———————
8 Client threads started
——— Average TP for all 8 threads:820463 read/sec———————
——— Exit Demo ————–

The expected output for GigaSpaces benchmark should look like this:

GigaSpaces cache benchmark started!
1 Client threads started
——— Average TP for all 1 threads:8312982 read/sec———————
2 Client threads started
——— Average TP for all 2 threads:7971583 read/sec———————
3 Client threads started
——— Average TP for all 3 threads:8069370 read/sec———————
4 Client threads started
——— Average TP for all 4 threads:8783607 read/sec———————
5 Client threads started
——— Average TP for all 5 threads:8414489 read/sec———————
6 Client threads started
——— Average TP for all 6 threads:8356409 read/sec———————
7 Client threads started
——— Average TP for all 7 threads:8451054 read/sec———————
8 Client threads started
——— Average TP for all 8 threads:8225495 read/sec———————
——— Exit Demo ————–

 

Benchmark was running on Linux OS, using Sun JDK 1.6 , 16 Cores Intel CPU , Cisco UCS Machine.

Disclaimer

The benchmark in this post tests Echcache in a local cache mode only (standalone) vs GigaSpaces Local Cache. It compares the scalability of get operation which is a common use case for client side cache. The setup is based on the default out of the box configuration of both products. This benchmark wasn't designed to test and compare the full Terracotta and GigaSpaces product. I kept the results objective providing full source code and configuration for the benchmark. I encourage any of you to try out the benchmark yourself and see the result. Your Feedback is appreciated !

Shay Hassidim

Deputy CTO , GigaSpaces

Terracotta ehCache vs. GigaSpaces Cache benchmark
Shay Hassidim
Deputy CTO @ GigaSpaces
Shay joined GigaSpaces in 2003, and has more than 20 years of experience in the software industry. Shay is part of the product leadership team and oversees the technical activities of the US operations, serving as Deputy CTO Distinguished engineer, escorting the pre-sale and post-sale activities.
  • Nicolas

    My question is, do you think you benchmarked a realistic use of EhCache or GigaSpaces? Come on! To store 100 objects, I can use a HashMap, or even an array ! I suspect that theses home made solutions would provide even higher performing results. IF the objects themselves aren't big they may even fill entirery in the CPU cache!

    Somebody will really want to test performance of theses solutions in more realistics cases with gibabyte of cached data and see at least how the two product handle scalability with a few nodes… You'll also want to mix read/writes.

  •  
    Dear Nicolas,
    Thank you for this comment.
    You have a point. With your permission let me clarify:
    There was no intention to compare the products with this benchmark. The goal was to focus on a very narrow and specific scenario that is the heart of every cache evaluation, reading data from the client cache assuming it is there when there are multiple concurrent threads. The classic scenario with any web / app server. We went for a small data set to allow everyone to run this test on his/her machine.
    This benchmark focused on the scalability behavior of the client cache. This is the most important aspect of every cache solution. If this one does not scale well , it does not worth the effort and time testing more complex scenarios such as write , remove , update, big memory , partitioning , LAN replication , WAN replication , elasticity , Query , security , reliable asynchronous persistency to RDBMS and NoSQL DB, Batch operations , schema evolution , ..Net/C++/Java interoperability, Map-Reduce API, Event Driven API , Cloud enabling , etc. You are more than welcome to test both products and compare. We will be happy to publish the report.
    Shay

  • Vladimir Rodionov

    I would prefer to have numbers for something larger than couple KB of data. How about 100GB cache instance for starter?

  • JC

    I am reading your testing program LocalCacheVsEHCacheBenchmarkMain.java

    can you explain why curTP = (curTP + tp) / 2;?
    I am evaluating your product and see if we can use that in our production. Thanks

  • shayhassidim

    JC,

    I’ve placed an updated benchmark code here:
    https://dl.dropboxusercontent.com/u/7390820/localcacheVsehCacheBench.zip

    See how results looks like:
    https://dl.dropboxusercontent.com/u/7390820/xap97vsehCache.jpg

    The numbers speak for themselves.

    Benchmark output:
    Terracotta ehCache benchmark started!
    1 Client threads started
    Total TP for all Threads:6666666.67— Average TP for all 1 threads:6666666.67 read/sec–
    2 Client threads started
    Total TP for all Threads:3809523.81— Average TP for all 2 threads:1904761.9 read/sec—-
    3 Client threads started
    — Total TP for all Threads:4797368.36— Average TP for all 3 threads:1599122.79 read/sec–
    4 Client threads started
    — Total TP for all Threads:5929678.48— Average TP for all 4 threads:1482419.62 read/sec–
    5 Client threads started
    — Total TP for all Threads:6661480.91— Average TP for all 5 threads:1332296.18 read/sec–
    6 Client threads started
    — Total TP for all Threads:7202600.47— Average TP for all 6 threads:1200433.41 read/sec–
    7 Client threads started
    — Total TP for all Threads:7540341.72— Average TP for all 7 threads:1077191.67 read/sec–
    8 Client threads started
    — Total TP for all Threads:8146094.97— Average TP for all 8 threads:1018261.87 read/sec—

    GigaSpaces cache benchmark started!
    1 Client threads started
    — Total TP for all Threads:9090909.09— Average TP for all 1 threads:9090909.09 read/sec—-
    2 Client threads started
    — Total TP for all Threads:18181818.18— Average TP for all 2 threads:9090909.09 read/sec—-
    3 Client threads started
    — Total TP for all Threads:27272727.27— Average TP for all 3 threads:9090909.09 read/sec—-
    4 Client threads started
    — Total TP for all Threads:36363636.36— Average TP for all 4 threads:9090909.09 read/sec—-
    5 Client threads started
    — Total TP for all Threads:45606060.61— Average TP for all 5 threads:9121212.12 read/sec—-
    6 Client threads started
    — Total TP for all Threads:48939393.94— Average TP for all 6 threads:8156565.66 read/sec—-
    7 Client threads started
    — Total TP for all Threads:60722610.72— Average TP for all 7 threads:8674658.67 read/sec—-
    8 Client threads started
    — Total TP for all Threads:69813519.81— Average TP for all 8 threads:8726689.98 read/sec—-

    Regards,
    Shay