Comment by BurritoAlPastor | Hacker News Reader

Comment by BurritoAlPastor | Hacker News Reader

BurritoAlPastor

The author's takeaways include moving disks to io1. This is a bad bargain in most cases, and particularly bad in the ~500 IOPS range (which is what I'm seeing in the Grafana screenshot there).

gp2 disks get 3 iops per gig "free", bursting up to 3k. (They don't burst after 1 tb, because your baseline performance is higher than the burst rate.) io1 is 25% more expensive per-gb, and you pay by the IOPS on top of that.

A 175gb gp2 disk will give you 525 IOPS baseline, at $17.50 a month. I'm guessing his volume is about 40gb, doing math backwards from his bottlenecked IOPS; a 40gb io1 with 500 IOPS will cost you $83.75 a month! And on top of that, AWS will cap you hard at that 500 IOPS; the gp2 can still burst if needed.

I know of two general cases where io1 disks are worthwhile: if you need more than 10k IOPS, or if you have very high IO requirements but very stable disk space usage (e.g. high-performance RRDs). Crack open Excel and do the math; it's worth the five minutes to check.

(Also, your burst balance is available as a Cloudwatch metric, as I recall! Set alarms on that shit!)

That's what we typically do to scale most EBS volumes (critical disk-intensive workloads run on io1 anyway). Instead of switching directly io1 we over-provision space which solves most IOPS issues.

We recently upgraded one of our etcd clusters from v2 to v3 and it turned out that etcd v3 uses significantly more IOPS than v2, using EBS burst credit during periods with normal load. Our solution was to increase the EBS volume sizes from 16GB to 50GB (100 IOPS -> 150 IOPS [1]). Our costs went from $1.6 per volume to $5, instead of $11.75 for io1, for a comparable performance for this particular application.

1: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolume...

Agreed. After a lot of struggles trying to find a good average IOPS on AWS for our database, we just increased disk to 1tb with gp2 and got rid of the problem. It gets 3000 iops/sec all the time and you never have these problems. It's still much cheaper than io1.

It's a trade you must account for when choosing cloud providers, nothing is free. For now it's still cheaper to pay for servers than spend time setting things up ourselves, but we're always measuring cost x benefits and thinking if we need to leave aws. The time is coming, but for now it's still working for us.

> we just increased disk to 1tb with gp2 and got rid of the problem. It gets 3000 iops/sec all the time and you never have these problems. It's still much cheaper than io1.

shockingly, tons of people don't know about this. Sure, there are use cases where it probably isn't cost effective to do this, but my default policy towards using gp2 is "will this server write to disk even a moderate amount? if so, its get a 1TB volume." Especially for something like a node in a k8s cluster where your workload is non-deterministic, this should be the default.

> Also, your burst balance is available as a Cloudwatch metric, as I recall! Set alarms on that shit!

That's my takeaway as well. If you can track and alert on it, you can provision whatever you want and only buy the expensive io1 when it makes sense.

Reply on news.ycombinator.com