EBS Storage Improvements | Hacker News Reader

EBS Storage Improvements

aws.amazon.com

73 pointsyaalok55 comments

And as always no word and no link to costs. EBS is not perfectly cheap, so in real life you need tedious studies to understand whether you want or can afford the nice options available. And understanding things in advance is close to impossible. You need to test and wait for enough billing data becoming available.

It’s expensive and a nightmare of cost complexity which leads to an entire meta-industry of cost management which I’m sure is a net loss.

I’m really more concerned about the absolutely miserably shitty performance of snapshots myself which always goes unmentioned and unmaintained. So I’ve got a 4TiB volume online attached to a node which is blasting out fragmented writes galore (Jenkins - ugh).

So we need to back up this volume for DR. Snapshots right? Well not when it takes 28 hours to snapshot that volume. So off to rdiff-backup it is. Back in the last infrastructure cult of virtualisation this was a 20 minute turnaround.

Good job it’s 5 nines now. That means we only risk a 3 day recovery window 2000x less often. Actually that hurt writing that.

I’m sure someone will now immediately sell me EKS to run Jenkins now which is ten fresh layers of eye gouging pain between idea to delivering software, something which I barely remember being able to do efficiently at this point.

Heh. I feel your pain. EKS is a nice platform, but I have seen the light and try not to run anything that requires a persistent file system on it. I feel like I'm shouting in the wilderness though- just adout every day is a "can't we deploy mysql here?"

Yes I’m a big proponent of stateless stuff. In fact 20 years ago I was crowing this to a deaf audience. Instead I find queues in databases that didn’t even need to exist. And our solutions, kubernetes for example, are complicated Rube Goldberg machines rather than things that make us sleep easy.

However really the software industry is all about state and where to keep it because that’s where the trolls collect their gold coins.

So what can we do?

I find gin helps to numb the pain...

So where would a queue go in a distributed system if not in some kind of database/file system? How do you handle resilience if the queue is not stored on disk somewhere? Paxos?

Well the point is more that a lot of stuff doesn't even need to be in a queue. It's just cookie cutter engineering. "I did this before and it worked so I'll do it again" without any consideration on what it does and why it does it.

Is this an EKS problem or a Kubernetes problem?

Both. Containers are meant to be ephemeral, they don't play well with permanent storage by design. People have been trying to run databases anyway and hitting roadblock after roadblock.

Here's an old rant mentioning broken container filesystems, a bit old from 2016 but storage didn't change much https://thehftguy.com/2016/11/01/docker-in-production-an-his...

Unless they are sharded and replicated distributed databases...

> Both. Containers are meant to be ephemeral, they don't play well with permanent storage by design. People have been trying to run databases anyway and hitting roadblock after roadblock.

Really? Can you share details? I'm asking because I know people who run databases in Docker for years and have no issues. Just store data on a volume.

> Here's an old rant mentioning broken container filesystems, a bit old from 2016 but storage didn't change much

Maybe not, haven't had a storage issue with Docker myself since maybe 2017. That article feels like a big rant but yes, it's ... old.

Docker volumes are great. I've also run databases in docker without issue. But they're different from kubernetes "persistent volumes" and "persistent volume claims"

The issues come when you move into high availability (specifically in a Kubernetes context). K8s may decide to move your database to a different node (server) on a whim, and suddenly your data is gone. You have to solve this by having mounted, block-level storage available to your cluster (eg AWS's EBS; the product being discussed) so you can use persistent volumes. That comes with added complexity, and usually the need to incorporate db operators into your stack.

As a result, a lot of k8s admins (myself included) choose to pay crazy amounts for RDS and S3 to avoid the complexities of persistent data.

I'm just learning all this; but couldn't you run a DB Cluster persistently on EC2, managed separately from K8s, and VPC it to the K8s cluster? Is there transfer charges involved and/or is the latency to night to consider a setup like this? Is this cost effective to RDS?

If you have a DB cluster running on multiple EC2 instances, you can definitely do that. There are advantages/disadvantages compared to just paying for RDS (eg managing backups, system complexity, and failover) but if you already have that expertise in-house, I can imagine a scale where there might be cost savings.

My comment was specifically discussing the challenges of keeping block storage-dependent, stateful apps within k8s (as opposed to using RDS or rolling your own HA cluster on EC2 or elsewhere). Apologies to GP if I incorrectly assumed they were discussing container volumes more broadly.

Yes, hence „volume”, not „docker volume”. For k8s, always a PV. rexray for mesos.

That article is about broken container filesystems, as in, the root filesystem of the container, but that filesystem is not where you would ever store the data of a database. In kubernetes/eks you would use a PersistentVolumeClaim with an ext4 or xfs filesystem. This would not share any of the overlay filesystem problems and StatefulSets are quite good at managing these.

We run over 900 very stateful MySQL database clusters and several hundred zookeeper clusters in kubernetes on ec2 with ebs successfully. Sure, we've hit our share of problems over the years, but storage itself is solid.

Indeed. The reliability of the system depends on the reliability of the block storage. There are many flaky solutions and many good solutions, and they are unrelated as to whether or not you are using containers.

Meanwhile arguably the whole reason Kubernetes took off compared to other container orchestrators is support for StatefulSets and Persistent Volumes.

Lots of people run their databases on Kubernetes, but it’s best to remember that Pod ephemerality makes it a more complicated orchestrator than them on VMs. That’s said, the market seems to have decided that’s what they want.

Funny enough, I agree and often hear of statefulset as a pro of kubernetes, yet I've much more rarely seen companies actually using the statefulset let alone migrate their databases to kubernetes.

There is a gap between liking a technology and migrating a hundred critical production databases onto it.

These things take time. Having sold a stateless container PaaS for many years, the main technical objection is they couldn’t run a database on it, you still needed VMs (even if they were on demand).

People caught in tech fads get obsessed with mechanism over outcome.

I was with you until >entire meta-industry of cost management which I’m sure is a net loss.

I am paid hourly and I have literally, in the past week, billed ~14 hours to save ~$150,000 per month. We work for a client and we strive to make sure we're not spending more money in labor than we're saving in AWS costs. I don't know exactly the rate I'm billed at that the client sees, but I know for sure that on aggregate, aside from the actual productive work that I know adds value, I have justified every penny I've been paid in the last ~2 years through my cost savings work alone.

Maybe some folks overcharge or undersave, but I take exception to your comment that you're "sure is a net loss."

Edit: our monthly AWS spend is in the 7 figures. We go through cycles where our priorities change, but when we're in cost savings mode, we regularly discover cost savings measure that are a) implementable in double digit hours and b) save six figures per month.

I’m not saying that it doesn’t result in a saving but that the fact that industry has to exist to keep costs constrained is a net loss.

Imagine if pricing was transparent, calculable and easy to rationalise the cost of a change. But it isn’t unless you pull a third party in.

My older volumes are of type "standard", but that type is not on the price list. I cannot even send a link to the price list, because the region is not in the URL. https://aws.amazon.com/ebs/pricing/ Complete lack of transparency.

It's not all that hard to find older generation pricing really... it's the same on every AWS product page. Many AWS products have thousands of older generation offerings which they don't want cluttering up the marketing pages.

https://aws.amazon.com/ebs/previous-generation/

Also easy to see on the pricing calculator

"standard" volumes are legacy. If you search for "previous generation" on a given service pricing page you'll usually find a link to the old stuff. In this case, https://aws.amazon.com/ebs/previous-generation/

Unless you're specifically looking for high-IOPS or massive HDD volume, you should just default to "gp2" SSD volumes these days.

"Today we are launching the io2 volume type, with two important benefits, at the same price as the existing io1 volumes"

The same price as something that has been difficult to grasp before does not really help.

My comment was not about the pricing of io2 in particular, but about the pricing of EBS in general.

Eh, it's not so complicated.

io1/io2 is about $12/month per 100 GB, and $60/m per 1000 IOPS.

So 200 GB and 10,000 iops would be $624

With the caveat that (io2) cannot support more than 500 IOPS per GB.

gp2 is $10/m per 100 GB, and you get 3 IOPS/GB (with potential to burst to 3000 IOPS for volumes under 1 TB).

GP2 IOPS max out at 16,000 per volume.

io1/io2 IOPS max out at 64,000

Basically, use io2 if you need more than 16,000 IOPS. or you need a large amount of IOPS for only a small amount of storage.

io2 also has a 0.001% annual failure rate, while the others have a .2% failure rate.

E.g., to get 10,000 IOPS with gp2, you would get a 3.33 TB volume, which would cost $333/m.

Local storage is far, far faster (up to millions of IOPS!)...but not as reliable, and does not have all the EBS features. https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/storage-...

> io2 also has a 0.001% annual failure rate, while the others have a .2% failure rate.

Is failing once every 5 years really different from failing once every 1000 years? You should have disaster recovery plans anyway and configuration errors are the most likely cause of failure, not the underlying storage system.

For companies that have a few hundred of these running at a time, it sure does! Also, it's a 2000x ratio, not 200..., and 1/500 years vs. 1/1,000,000 years.

AWS’s claimed 99.999% ratio is 1/100,000 years. Claimed improvement of 100x in the article.

Hmmm, it said "2000x more reliable than a commodity disk drive" in the article which I misunderstood (I guess that is a comparison to a local HD) + plus my inability to do math late at night :-)

Thank you for the correction!

Well, I am playing in a different league. I have single digit budgets per volume per month. Of course they are smaller and have more modest performance requirements. But still, exceeding my budget by 30% is not acceptable over longer time. And EBS (as well as all other Amazon stuff) makes it really easy to exceed it by 300%.

For standard volumes you pay by IO operations (absolute calls). For the newer types you pay by provisioned IOPS (a max rate). That's a particular difficult decision to make if you move in that area. The less advanced competition has simpler pricing, but then you might be more unsure about what you get. Again for high requirements you know that if someone offers you 25 or even 50 GB for 1€ per month you know it's too good to be true. For modest requirements that's something you need to consider.

Sounds like AWS may not be the right place for your workloads.

Often it is not.

But smaller/cheaper competitors might have more downtime (hard to quantify in advance), problems to keep their kernels maintained, less predictable capacity and product lifetime. So it's not without risks going for cheaper options.

AWS Lightsail might be a better fit for you.

Thanks for that tip, they don't seem to advertise that a lot!

Launched 11/16 and somewhat reasonably priced since 08/18. Still more expensive than some others I also use, but at these prices it might be an easier decision if you want to avoid risks with the more aggressively priced competitors.

Edit: Even with a terraform provider, nice!

They actually give you an incredible discount on bandwidth, if that's important...it's a a known AWS "hack" :-)

Data transfer hasn't been an issue for me in those low end work-type scenarios.

However, as a person having connections to several countries I use to spin up an instance in a suitable country to work around geo-blocking of TV stations occasionally. That way you very quickly accumulate a couple of GB. Looks like for this use case this lightsail should be much cheaper, even if my bill is not a problem in absolute terms.

Literally in the first sentence of the new volume type intro: "… at the same price as the existing io1 volumes".

The cost isn't hard to find, Google "ebs pricing". There's plenty of things to complain about AWS billing, but EBS pricing is one of the simplest things to deal with. But if you aren't sure if you need these, you probably don't.

"ebs pricing" search does not find standard, at least not in top rank, because standard is not on the ebs pricing page. You need to know to add "previous generation" or "legacy" or something like that to your search terms. Easy when you remember it, but not great transparency from their side.

> But if you aren't sure if you need these, you probably don't.

True. When inspired by this discussion checking out what I pay for a moment I had this oh shit experience for a moment, I pay twice as much as the cheapest option. When trying to convert the first volume I noticed that the minimum size for that is 500 GB, that's 100 times my volume size, 50 times the prize, so I really don't need it.

However, as someone pointed out somewhere else in this discussion there might be a more suitable option on the low end called AWS lightsail.

For anyone curious about the durability of gen1 (io1) I had to look it up to compare to this new tier.

“All other EBS volumes are designed to deliver 99.8% - 99.9% durability”

Summary - the new offering (called io2) jumps to five 9’s.

Great job AWS

Not sure how they determine that though? At that level, drive failures or single host failures (for which they have plenty of data available) won't cause issues. The risk remaining are botched deployments or config changes that bring down whole data centers or services. They are very unlikely, but can you really quantify their likelihood and expected downtime?

5 9's to me sounds mostly like marketing. In the end you still have to have disaster recovery plans if the service fails and you also need to test your DR plans regularly. That extra 9 won't really save any money here.

I tend to look at it in terms of how safe you can be while being lazy. We should have replication, DR , backups at various intervals, snapshots (as someone else mentioned take forever with big disks). When we have less than what we “should” I think the 9s paint a picture of how risky we are playing. Keeping in mind, less than should is always lazy. We gotta call it what it is.

This is the same company that declared s3 has 11 9s of annual durability. They might have enough objects that they can verify that number now.

In the ebs outage of 2011, volumes were unable to remirror after losing 1 mirror. That increased the probability of losing both mirrors, possibly reducing the durability below the designed rate. I.e. the designed rate assumes everything is working as designed.

Somebody know what is the architecture behind those volumes? How do they work?

For RDS, the “trick” to getting max iops in gp2 cheaply is to get over 5334GB of storage, giving you the baseline of 16,000 iops (no burst at that level)[1].

This costs roughly $615 a month (double for multi az), way cheaper than io2 at that level.

It looks like 16000 provisioned iops in RDS is more like $2266 for 5334gb (double for multi az) a month. Varies by storage amount of course - 500gb storage is $1600 for single-az at 16k iops.

I’m not sure this change affects RDS storage pricing tho? The pricing calculator still calls Provisioned IOPs "IO1".

[1] https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_...

.. so it's cheaper to provision more disk and get free iops than to get a smaller disk and pay for provisioned iops?

I recently bumped iops to 12k on one of my RDS instances, and I had to cringe when I did it.

Bit of a one-way thing though, you can't reduce the amount of storage allocated on an RDS instance without downtime (at least with Postgres)

Will this new storage layer be available for AWS services like RDS and ElasticSearch? I checked the dashboards and the SSD options are still just "General Purpose SSD" or "Provisioned SSD".

My understanding is that these services use EBS under the hood but I guess there is no way to check if the attached storage is using io1 or io2 for now?

Provisioned IOPs are io1 still according to the pricing calculator (to my knowledge, it is EBS under the hood).

Max IOPS is still only 64,000 though?

This is correct, according to the documentation [^1].

I could not find any explanation of the differences between io1 and io2 that would justify the increase in IOPS per GiB and durability on the EC2 user guide [^2] or on Amazon Builders Library [^3], though.

Note that 64k is the maximum IOPS per volume, not per instance the volume is attached to. Therefore you can use software RAID configurations to increase your performance or fault tolerance further [^4]

Links:

[^1]: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volu...

[^2]: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-ug.p...

[^3]: https://aws.amazon.com/builders-library/

[^4]: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/raid-con...

Reply on news.ycombinator.com