Author Archives: Guy Sayar
VBA GSIterator
VB Space API v1.2 is released.
A nice addition to the release is the native support for GSIterator, which enabled reading large sets of data in a manageable manner. The ReadWrite example file includes a usage scenario.
-Guy Continue reading
Realtime Aggregations
Quartet Financial provide a very appealing product called ActivePivot.
ActivePivot offloads pivot calculations and aggregations from the client process to a backend server. It exposes XMLA interface (among others), which means that if you are an Excel user, you simply define a pivot table and point the data source to the ActivePivot server. From that point onwards, Excel interacts with ActivePivot natively.
The result is a very thin Excel spreadsheet which only displays the aggregated result, and all the number crunching and aggregations take place at the server side. When the user interacts with the pivot table, Excel queries ActivePivot and display the next level. Very nice!
ActivePivot provides hooks for the correlation and aggregation execution, so custom logic can easily be applied.
ActivePivot stores the aggregated cube in memory, which allows it to update the cube and respond to market events, and by that provide intra-day aggregations.
The real time bit is where GigaSpaces fits in
Market data feeds are written into a space using one of GigaSpaces’ APIs (JMS, Remoting, JavaSpaces, etc’), and ActivePivot connect to the space and register for space events. When a tick is updated in the space, the space sends a notification to ActivePivot which in turn re-aggregates the relevant cube branch. This means that the aggregated cube is always updated and reflects the latest market state. In addition ActivePivot queries the space for the raw data when Excel sends a drill-down request, which allows ActivePivot a very quick response time.
ActivePivot can be applied at different areas and provide real time P&L, real time Risk, etc’
-Guy
Deployment Predictability
My colleague Uri raised an interesting point in his post. I completely agree with Uri, and would like to give an example.
I have been involved in a project for a mobile operator in the UK during the second half of last year. We built a scale-out SOA activation platform for a new mobile device launch using GigaSpaces. The GigaSpaces platform replaced an existing system.
The original system was built using JBoss as a backend server. Predictions were for a huge increase in activation-requests the system had to handle due to the new device launch. While the system worked fine using the JBoss as a platform, there was no way these guys could predict how many instances of JBoss they would need to run in order to cope with anticipated load. They started to do some benchmarking and performance testing to figure out where the system’s limits were, but they soon found out, that the process was leading them nowhere. This is mainly because the JBoss’s were inconsistent and they hit the scalability ceiling using a few (very few…) nodes. When adding more instances, the overhead of synchronising the JBoss cluster grow exponentially as suggested by Ahmdal’s Law, so the gain in TP that each instance added varied depending on the cluster size and other nodes’ load, which kills predictability all together.
JBoss is just an example in this case. It’s not a JBoss specific flaw, but rather a tier based approached which imposes a limited architecture.
They then came to us to resolves the predictability challenge.
We did an exercise to figure out how the deployment would look like using GigaSpaces, and came up with a linear formula of the HW and number of instances needed to support the given load. More than that, they knew that if the business predictions had been pessimistic, supporting extra load would mean simply deploying more spaces… On top of that, their back office systems did not support HA, and would explode if load increased suddenly, so GigaSpaces also provided HA and throttling for the backend servers. During one overnight test the database failed for about 4 hours, and the system was fully functional and completed users’ requests, while completed requests waited for the database to be brought up again to complete the archiving process. The customer was truly impressed!
Needless to say that the launch went flawless, and there were no issues what so ever with the GigaSpaces based system.
So, yes – Uri makes a good point. GigaSpaces’ customers can predict and properly plan ahead the deployment needed to support their business.
-Guy
Continue reading
Let’s play…
I have been involved in a nice gaming project in the last couple of weeks.
In the first meeting, giving an overview, the prospect presented their challenge:
“We store game data in memory as we need very fast response time, and we are reaching our limits in terms of supporting our business. We are launching games which support many thousands of users and we are not able to store a game’s data in a single server.
We tried using a database and running our aggregations against it. The response time we got was not acceptable. We want to store game’s data across multiple JVM, but how will which JVM stores which pieces of the data?”
Well, aggregations and data partitioning go very well together if you have the right technology in hand. Nati Shalom wrote a good post on this subject.
The interesting thing is that GigaSpaces supports aggregation since way before the Map/Reduce term was even thought of… When using JavaSpaces’ readMultiple API against a clustered space, the read request is broadcasted to all the cluster members, who execute the request (locally) in parallel, the client side receives back an array of responses which it then needs to aggregate (Reduce). With OpenSpaces, we have Springified this and it is also supported through Spring Remoting with a pluggable reducer at the client side.
A couple of hours after the dev was kicked off, we were able to demo how a single game can span multiple JVMs, aggregate the game’s data in a simple fashion, and display the aggregated result. The system, which was limited to x10,000 concurrent users, can now scale to x100,000 and x1,000,000 linearly using GigaSpaces.
Bingo!
-Guy
Continue reading
VB Space API Alfa Release
An Alfa release of the Space VB API is now available at OpenSpaces.org.
There are a number of changes in this build. Main ones are a new installer, a fix for default value in read and take APIs, and a more comprehensive example and documentation pages.
You can read more about it in the project pages, but one thing I want to specifically mention in my experience with the installer program.
It has been very long since the last time I had to compile an installer program. Actually, it was around 1998, where I created a setup for a university project. It was based on InstallShield, which made the entire process long and painful! This time it was very different. I used a utility called Inno Setup, a very lightweight and simple installer. It took me no more than 10 minutes to create the first run of the setup, and another half an hour or so to customise it with my own specific stuff (making sure GigaSpaces is installed, invoking the example at the end of the setup, etc’), which was also straight forward to do thanks to the detailed help files provided.
Now, surely InstallShield could do everything I needed and much more, but it would have been much harder and a longer process… Kudos to Jordan Russell for the well designed and implemented installer utility!
Enjoy!
-Guy
