Continuous reinvention: A brief history of block storage at AWS
allthingsdistributed.com
allthingsdistributed.com
> Compounding this latency, hard drive performance is also variable depending on the other transactions in the queue. Smaller requests that are scattered randomly on the media take longer to find and access than several large requests that are all next to each other. This random performance led to wildly inconsistent behavior.
The effect of this can be huge! Given a reasonably sequential workload, modern magnetic drives can do >100MB/s of reads or writes. Given an entirely random 4kB workload, they can be limited to as little as 400kB/s of reads or writes. Queuing and scheduling can help avoid the truly bad end of this, but real-world performance still varies by over 100x depending on workload. That's really hard for a multi-tenant system to deal with (especially with reads, where you can't do the "just write it somewhere else" trick).
> To know what to fix, we had to know what was broken, and then prioritize those fixes based on effort and rewards.
This was the biggest thing I learned from Marc in my career (so far). He'd spend time working on visualizations of latency (like the histogram time series in this post) which were much richer than any of the telemetry we had, then tell a story using those visualizations, and completely change the team's perspective on the work that needed to be done. Each peak in the histogram came with it's own story, and own work to optimize. Really diving into performance data - and looking at that data in multiple ways - unlocks efficiencies and opportunities that are invisible without that work and investment.
> Armed with this knowledge, and a lot of human effort, over the course of a few months in 2013, EBS was able to put a single SSD into each and every one of those thousands of servers.
This retrofit project is one of my favorite AWS stories.
> The thing that made this possible is that we designed our system from the start with non-disruptive maintenance events in mind. We could retarget EBS volumes to new storage servers, and update software or rebuild the empty servers as needed.
This is a great reminder that building distributed systems isn't just for scale. Here, we see how building the system in a way that can seamlessly tolerate the failure of a server, and move data around without loss, makes large scale operations (everything from day-to-day software upgrades to a massive hardware retrofit project) possible that just wouldn't be possible in a "simpler" architecture. A "simpler" architecture would make these operations much harder, to the point of being impossible, making the end-to-end problem we're trying to solve for the customer harder.
At the time each volume had very inconsistent performance, so I would launch seven or eight, and then run some each write and read loads. I'd take the five best performers and then put them into a Linux software raid.
In the good case, I got the desired effect -- I did in fact get more IOPS then 5x a single node. But in the bad case, oh boy was it bad.
What I didn't realize was that if you're using a software raid, if one node is slow, the entire raid moves at the speed of the slowest volume. So this would manifest as a database going bad. It took a while to figure out it was the RAID that was the problem. And even then, removing the bad node was hard -- the software raid really didn't want to let go of the bad volume until it could finish writing out to it, which of course was super slow.
And then I would put in a new EBS volume and have to rebuild the array, which of course it was also bad at because it would be bottlenecked on the IOPS for the new volume.
We moved off of those software raids after a while. We almost never used EBS at Netflix, in part because I would tell everyone who would listen about my folly at reddit, and because they had already standardized on using only local disk before I ever got there.
And an amusing side note, when AWS had that massive EBS outage, I still worked at reddit and I was actually watching Netflix while I was waiting for the EBS to come back so I could fix all the databases. When I interviewed at Netflix one of the questions I asked them was "how were you still up during the EBS outage?", and they said, "Oh, we just don't use EBS".
Hey! We also did that! It turned out, that eventually you hit the network bandwidth limit. I think, the performance topped out at around 160 megabytes per second for most of the instance types back then.
One interesting tidbit is that during the period this author writes about, AWS had a roughly 4-day outage (impacted at least EC2, EBS, and RDS, iirc), caused by EBS, that really shook folks' confidence in AWS.
It resulted in a reorg and much deeper investment in EBS as a standalone service.
It also happened around the time Apple was becoming a customer, and AWS in general was going through hockey-stick growth thanks to startup adoption (Netflix, Zynga, Dropbox, etc).
It's fun to read about these technical and operational bits, but technical innovation in production is messy, and happens against a backdrop of Real Business Needs.
I wish more of THOSE stories were told as well.
Can anyone explain why?
https://www.allthingsdistributed.com/images/mo-manual-ssd.pn...
I think we got SSDs installed in blades from Dell well before that, but I may be misremembering.
I/O performance was a big thing in like 2010/2011/2012. We went from spinning HDs to Flash memory.
I remember experimenting with these raw Flash-based devices, no error/wear level handling at all. Insanity, but we were all desperate for that insane I/O performance bump from spinning rust to silicon.
Secondly it reminds me of the time when it simply made sense to ninja-break and rebuild mdraids with ssds in-place of the spinning drives WHILE the servers were running (sata kind of supported hotswapping the drives). Going from spinning to ssd gave us a 14x increase in IOPS in the most important system of the platform.
That's one of the reasons why I think we should have a professional license. By requiring an apprenticeship under a master engineer, somebody can pick up incredibly valuable knowledge and skills (that you only learn by experience) in a very short time frame, and then be released out into the world to be much more effective throughout their career. And as someone who also interviews candidates, some proof of their experience and a reference from their mentor would be invaluable.
You'd also see service providers start selling pre-PE stamped services - which is hardly different from the whole "go buy your SaaS from X who is (compliance regime) authorized" today. Just like today when you build a house, your general contractor goes and buys a pre-engineered truss and hires a bunch of laborers to put it up per spec.
> While the much celebrated ideal of a “full stack engineer” is valuable, in deep and complex systems it’s often even more valuable to create cohorts of experts who can collaborate and get really creative across the entire stack and all their individual areas of depth.
Otherwise, great article, illustrating that it's queues all the way down!
There's usually enough for a GPU, SSD or two... and that's about it. I don't like having to spend so much for fast IO, dangit.
Can sometimes find boards that do switching to appease :/
The ATX spec allows for 7 or fewer physical slots. That's not what's letting me down!
How many lanes are actually available and their division changes over time. Everything from USB to SATA has internally been moving to PCI-e, there's more 'internal consumption', too.
Gen5 is kind of a victim of success. Each individual lane is very fast, but providing them to a fair number of add-in cards/devices isn't practical. Simply physically, but also bandwidth between lanes.
Pretty much all of gen4 is nice, those systems are perfectly viable. More lanes that were slower offered me more utility. It's multi-faceted, I overstate it too. I like hyperbole :)
A B650(E) board has enough lanes to give you a fast GPU slot, four CPU x4 slots running at 8-16GB/s each, and four chipset x1 slots running at 2GB/s each.
Why wouldn't they have?
E.g.
> Why do servers have redundant power supplies?
So if you lose power you don't lose the whole server? It's even more important for cloud providers that have huge servers with high density. You connect the different power supplies each to an independent power feed so 1 can go down. Would you rather have 2x the capacity instead?
This is commodity hardware.
It pretty much works off ARM spec. Besides AWS owning it and offering it cheaper it's not in any way faster / better than Intel / AMD / something else.
We've had custom motherboards, server cases, etc long ago even before clouds.
If Apple silicon happens in the cloud then maybe...
Now the focus is on cost-efficient machines in clusters, which is why we have the specialization.
"EBS is capable of delivering more IOPS to a single instance today than it could deliver to an entire Availability Zone (AZ) in the early years on top of HDDs."
Dang!
Otherwise instance drive and sync over S3.
Instance storage seems to only be available for (large) Linux EC2 instances.
There are no operating system limitations that I'm aware of, however. I was just able to launch a Windows m6idn.2xlarge to verify.
My use case was to bring up a Windows instance using instance storage as the root device instead using of EBS which is the default root device.
I wanted to run some benchmarks directly on drive C:\ — backed by an NVMe SSD-based instance store — because of an app that will only install to drive C:\, but it seems there’s no way to do this.
The EC2 docs definitely gave me the impression that instance storage is not supported on Windows as a root volume.
Here’s one such note from the docs: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/RootDevi...
”Windows instances do not support instance-store backed root volumes.”
Really nice that you are engaging with the comments here on HN as the article’s author.
(For others who may not be aware, msolson = Marc Olson)
Before we launched the c4 instance family the vast majority of instance launches were from EBS backed AMIs, so we decided to remove a pile of complexity, and beginning with the c4 instance family, we stopped supporting instance storage root volumes on new instance families.
Also, throughput is going to be limited by your instance type, right? Though that might also be the case for EBS. I can't remember. Part of the problem is AWS performance is so confusing.
EBS connects to EC2 via a separate pipeline, different from the EC2 instance Networking bandwidth. This is true for all Nitro instances. EFS / FSx connects to EC2 via Networking bandwidth. So you should refer to that if you are looking for the bandwidth information.
While instance storage is ephemeral nothing really stop you from using it as a local cache in a clustered filesystem. If you have a somewhat read intensive workload then you might see performance close to matching that of using instance storage directly.
There are some fundamental limits to how fast a clustered filesystem can be, based on things like network latency and block size. Things like locking is an order of magnitude slower on a clustered filesystem compared to locally attached storage.
It's difficult to innovate by just repeating what's been done before. But everything you learn along the way helps shape that innovation.
> In retrospect, if we knew at the time how much we didn’t know, we may not have even started the project!
I mean, literally every problem they outlined.
>Compounding this latency, hard drive performance is also variable depending on the other transactions in the queue. Smaller requests that are scattered randomly on the media take longer to find and access than several large requests that are all next to each other. This random performance led to wildly inconsistent behavior. Early on, we knew that we needed to spread customers across many disks to achieve reasonable performance. This had a benefit, it dropped the peak outlier latency for the hottest workloads, but unfortunately it spread the inconsistent behavior out so that it impacted many customers.
Right - which we all knew about in the 90s, and NetApp more or less solved with WAFL.
>We made a small change to our software that staged new writes onto that SSD, allowing us to return completion back to your application, and then flushed the writes to the slower hard disk asynchronously.
So a write cache, which again every major vendor had from the beginning of time. NetApp used NVRam cards, EMC used dedicated UPSs to give their memory time to de-stage.
Etc. etc.
>network attached block level storage at AWS's scale hasn't been done before.
This is just patently false. It's not like EBS is one giant repository of storage. The "scale" they push individual instances to isn't anything unique. The fact they're deploying more pods in totality than any individual enterprise isn't really relevant beyond the fact they're getting even greater volume discounts from their suppliers. At some point whether I'm managing 100 of the same thing or 1,000 - if I've built proper automation my only additional overhead is replacing failed hardware.
Downvote away, watching HN think that re-inventing the wheel instead of asking someone who has been there already what the landmines are seems to be a common theme.
Hahahah surely this is a joke, right?
If it’s so easy and you already had solved all these problems, why didn’t someone already build it? Why didn’t you build EBS, since you apparently have all the answers?
Regarding a durable/asynchronously destaged write cache, I think EMC Symmetrix already had such a feature in the end of '80s or 1990 (can't find the source anymore).
Well among implementers, vendors were mostly locked into the vertical scaling model.
I ran a SGI cluster running CXFS in 2000 as an example, and by the time EBS launched, I was spending most of my SAN architect time trying to get away from central storage.
There were absolutely new problems and amazing solutions by the EBS team, but there was information.
Queue theory was required for any meaningful SAN deployment as an example and RPM/3600 had always been a metric for HD performance under random.
Not that everyone used them, but I had to.
People often talk about standing on the shoulders of giants. To extend that metaphor, the only ones standing there are the ones who climbed the giant. If you hire a college grad and stick them on that giant's shoulder, it's not the same.
Craft comes before art.
The downward slide of most great tech companies starts with managers finally losing patience with the engineers. Check out Boeing as a recent example. It’s not about art or craft, it’s about treating people as more than widget producers.
Your comment elsewhere about NetApp solving all known problems with WAFL. Hahahaha. Have you tried deleting a 5TB file in a filesystem at 95% capacity with snapshots enabled?
I guess it's a good thing I didn't say NetApp solved "all known problems with WAFL" - but critical reading would've required you to respond to the content of the post, not provide an inaccurate summary to make a point that wasn't there.
What I DID say is that NetApp solved the issue of spewing random small writes all over the disk, resulting in horrendous performance on subsequent reads from spinning disk.sace reclamation takes a while. What's your point, assuming you had one? Because deleting a 5TB file on a filesystem 95% full with lots of snapshots is a daily workflow for... nobody? And if it is: there are countless ways to avoid that situation, but I assume you knew that too?
VMware tried something like that with vCloud Air [0], which attempted to build a public cloud on the ESXi hypervisor with hardware and expertise sourced from storage vendors. It failed.
There were a number of reasons for the failure but perhaps the most prominent was that VMware vastly underestimated the amount of engineering necessary to build a multi-tenant public cloud. As the article pointed out you need to optimize along the entire I/O path from client request down to sectors on disk. It's a significant mindshift from running VMs over a SAN, which is where VMware came from.
(I was involved in the project in its later stages. These are personal observations.)