Monthly Archives: March 2009
Another successful Enterprise EC2 deployment
Last week was a pretty busy week for me. As well as closing out business with a major international bank who are going to be using GigaSpaces to speed up their trading systems (and the results of this in itself are impressive, speeding up Trade queries from 30 minutes to between 15 to 30 seconds) we also had a major Telco go live with their first service hosted on Amazon EC2.
The client in this case chose to use GigaSpaces on EC2 because:
(i) Security: The Telco’s services were exposed over a secure SOA layer hosted in an internal Data Centre. This, coupled with a secure EC2 instance in which all public cloud /private cloud connections were done over SSL gave them the ability to consider hosting front end and new application services outside of the corporate data center
(ii)Flexibilty and Agility: The difference in building out the service in EC2 and doing this in-house was vast in terms of timescales. Difficulty in procuring machines for the reference environment and for Test and Development would have meant the service could not have been deployed in the timeframe that the business required.
(iii)Cost: Upfront costs were zero to minimal. Just the costs of renting EC2 units for reference, test and development
(iv) Little or no impact on the development process, other than speeding up ! The use of GigaSpaces Cloud tooling meant that the development team used their normal agile approach and used the GigaSpaces cloud tools to automate deployment to the cloud.
(v) Testing Scale: The ability to procure instances immediately to test scale meant that the robustness of the service to handle load could be validated early.
(vi) Choice: As GigaSpaces can be used inside the corporate data centre or on the cloud (unlike some vendors I could mention) then the organisation could make the choice as to the most effective way to make the deployment based on cost and speed.
If you would like to have a look at the high level architecture of this application here.
Also, if you did not get a chance to view it, GigaSpaces held a webinar on “Deploying your existing applications to the Cloud”. I’ve embedded the video below so you can view this.
Twitter getting bigger – Scaling becomes more of an issue
An informative post on YES Cloud outlines the infrastructure implications of tweeting using Twitter. Using Twitter stats from tweetstats.com shows that Tweeter are responsible for nearly 2 million tweets per day. With Twitter currently popularity soaring and it coming out of it’s current niche use it is frightening to think how many Tweets it is going to have to handle.
To get a better sense of the infrastructure needed Prashant Gandhi, the blogger from Yes Cloud, did some calculation in the original post as to what Twitter may have to handle:
Assumptions:
Average Tweet Size: 100 bytes
# of Tweets: 10 per tweeter per day
# of Tweeters: 1 billion worldwide
Infrastructure Requirements:
Tweet Rate: 10 billion tweets per day
Tweet Storage: 100 Gigabytes per day (with 10:1 compression)
Storage needs appear to be quite manageable also – 100GB/day means ~37TB/year
Each tweet is essentially an HTTP transaction (request and response). The tweet rate of 10B/day translates to ~115K HTTP transactions/sec for tweets uniformly distributed throughout the day. Assuming that the compute infrastructure (aggregate of web, application, database servers) can process 1000 transactions/sec/server, about 115 servers are needed. If a peak to average ratio of 3:1 is assumed, then about 350 servers are needed.
Now given that Twitter has more than enough problems scaling it’s current infrastructure do you really think it would be able to handle these volumes of scale ? Or will it/ has it re-architected ? (I’m waiting for that call you Twitter execs…..)
Now GigaSpaces is recognized as a technology that can scale and handle huge loads. It has proved this numerous times on platform tests, in banks handling huge amounts of market data, and by handling such dynamic platforms as the iPhone launch. I am starting to see more and more dynamic media, retail and innovative web 2.0 type platform vendors using GigaSpaces to handle the dynamicity of Peak loads that they otherwise cannot. GigaSpaces is the difference between scaling out your platform in real-time and servicing your customers, or not. Simple as that. Throw in the cloud to be able to handle break out scaling that the existing data centre cannot and GigaSpaces becomes even more compelling.
Oh, and if you want to read more thoughts about scaling Twitter with GigaSpaces have a look at Nati Shalom’s blog on the subject and also Guy NIrpaz’s presentation on Scaling Twitter.
Scrum is Infectious
Today, on my way back from Sweden, I stopped in the
Netherlands to visit Tricode, our strategic partner. On the way from the airport
we’ve chatted as always, and I realized that they started to implement scrum as
well in their projects. This is not the first (not the second and even not the
10th) GigaSpaces’ partner who’s adopting scrum, directly influenced by
GigaSapces. As our product development is being done with Scrum.
I wonder why is that, what makes them adopt scrum?
Well, it is optional that it has nothing to do with GigaSpaces.
Scrum is a popular agile methodology and coincidently we use scrum as other
organizations. This may very well be the case, but I doubt that.
When asking some deeper questions, it appears that the level
of service GigaSpaces’ partners and customers are getting because we are able
to respond quickly to on-going requirements and the fact that our early access
program allows our partners to influence a product release while it’s in the
making is perceived as a great advantage. This opens a whole new game of
collaboration between the software vendor (GigaSpaces in this case) and our partners
and customers. They are now part of the team. I believe this is the level of
cooperation and participation any product team would like to be able to provide
to its end users, and so do they. I think (humbly) that this is the main reason
for the aforementioned question.
Furthermore, it opens a whole new discussion channel between
us. We not only speak about the functionality provided by our product, we also
share experiences and advices on scrum implementation. This is the great part
of being a development team which creates a product that other development
teams use.
It is truly a great feeling. We’re adding value not just by
delivering our product per se. We also create a lot of value by incorporating
our customers and partners into our development process, and by sharing
experiences and helping each other out.
I may very well be wrong with my conclusion, but it does
feel good!
Is Cloud Computing a revolution or an evolution?
Over the past few months I occasionally got into heated debate with different colleagues regarding whether cloud computing is a revolution or an evolution. Will it change our world, or is it going to be more of the same thing… Continue reading
4643 is the magic number for Amazon EC2 reserved pricing
As Kamran Yousaf and I found out when looking at when EC2 reserved pricing makes sense, the simple answer is 4643.
Amazon recently announced new pricing option where you can reserve an instance for one or three years and then have discount on the hourly rate. The table below describes the cost per year if the instance is up for the whole year.
| Instance Type | Cost/Year On Demand instance $ | Cost/Year for 1 year reserved instance $ | Cost/Year for 3 Year reserved instance $ | |||
| Small |
|
|
|
|||
| Large |
|
|
|
|||
| Extra Large |
|
|
|
|||
| Medium High CPU |
|
|
|
|||
| Extra Large High CPU |
|
|
|
The above prices were calculated using Linux instance and US pricing
Cost/Year On Demand instance$ = cost per hour * 365*24
Cost/Year for 1 year Reserved instance $ = 1 Year reserve price + (cost per hour * 365*24)
Cost/Year for 3 Year reserved instance $ = (3 Year reserve price /3)+ (cost per hour * 365*24)
Now this brings the interesting question when is it cost efficient reserve an instance ?
We can calculate the minimum usage hours over which the cheaper per hour price for a reserved instance starts to decrease the overall cost using the following formula
Instance Reserve Price/(On Demand Instance Hourly Price – Reserve Instance Hourly Price)
e.g for a small one year reserved instance
350/(0.10 – 0.03) = 4642.8 hours
For small 3 year reserved instance
500 / (0.10 – 0.03) = 7142.8 hours
The above calculated numbers are same for all instance types. So you should reserve a one year instance if you intend to use more than 4643 hours or the instance is up 53% of the time in one year. With a 3 year reserved instance you can save money if over 3 year period you use 7143 or more hours or the instance is up 27% of the time over a 3 year time period.
(adapted from our original post on Cloudiquity.com)
