Hacker News Reader

Comment by qhoxie | Hacker News Reader

We have a highly trafficked and large dataset that we aimed to move from our datacenter to EC2. This is all in MySQL. Upon doing so, we had to spend significant effort trying to get our system stable and we ended up narrowing it down to EBS simply not being able to handle the IOPS we needed.

We explored all the options available and worked with both Percona and Amazon on the issues, but it was clear that EBS and EC2 just are not meant to handle such load. Our database is currently back running in our datacenter.

There have been a number of reports with similar results using EBS. It all depends on your particular IO profile, but since you said it was high, I encourage you to tread lightly.

If you want more information you are welcome to email me.

View on HN

The pain point in this particular case (I worked with qhoxie and bjclark on this) wasn't so much that the EBS performance was consistently bad, but that it was very unpredictable. Servers would suddenly, for no apparent reason, start performing very poorly.

That's been a consistent theme I've seen with EC2 instances--it's hard to predict how fast something is going to be, and once it's running, you don't know if its performance is going to change. That's one reason to take benchmarks of EC2 (and possibly other cloud providers) with a grain of salt.

I've heard anecdotes of people starting up n EC2 instances, running benchmarks on each, then killing all but the fastest ones.

We don't go that far, but we did choose the data center that had the faster machines. It had upgraded hardware that the others didn't. It won't be the fastest forever, but you can certainly shop around internally to find faster machines.

Just to add to what QHoxie said (I worked with him on this project), the site is doing Alexa top 750 traffic. Every page view would hit the db atleast a few times. And it's not that the site just couldn't run off EBS volumes, but we would see massive performance spikes where EBS IOPS would slow to a crawl for a few minutes to even a couple hours.

This is a good clarification to make. It was not that our entire application ground to a halt when we flipped the switch. We would experience intermittent and extreme performance issues. We dug deep with tools like cacti and ended up being able to point only at the 'unknowns' of the shared resources and EBS.

I'm beginning to thing that MySQL does something very different with disks than Postgres, because everyone I've talked to that said they had a problem had been running MySQL, and everyone who has been successful has been running Postgres (myself included).

This really would not be all that surprising, but it also would not surprise me if the average Postgres setup was better tuned/architected than the majority case of MySQL uses.

Our setup was surely not ideal, but even with significant tuning, it was not sufficient.

It is really good to hear successes like Reddit's, though. How long have you folks had your databases in EC2?

We had some of databases at EC2 since November of 2008, and moved the rest of them over in May 2009.

Maybe. I know when we migrated Chesspark from native machines to EC2, the postgresql database performance was at least an order of magnitude worse, and we spent two weeks doing caching and index hunting that we never had to worry about before.

Reply on news.ycombinator.com