Comment by justinsb | Hacker News Reader

Comment by justinsb | Hacker News Reader

Parent Full thread

What I'd like to see is a storage product that aggregates the unused storage space on EC2 instances.

* By default instance storage isn't attached, so there's probably a lot of completely available capacity.

* Even if attached, it's rare that the full capacity would be used, so thin provisioning would leave some space available.

* Some host machines won't be fully allocated.

I imagine taking this pool of capacity and using erasure coding and replication to build reliable storage. As your volumes come and go, you need to make sure it remains available, which is why I imagine erasure coding across a large number of customers. By integrating with the guest -> host assignment function you can ensure that you never lose data, if need be delaying scheduling until you've copied data elsewhere.

You'd have to throttle reads & writes to ensure that the guests weren't unduly impacted (easier with SSD's predictable IOPS), and the splitting and erasure coding would make for slow reads & writes as well. But this makes the economics a lot more attractive (free!).

One thing that suggests Glacier could be doing this: if I was AWS, and I was doing this, I would not be in any rush to tell EC2 customers that I was "stealing" their unused capacity!

I don't see why Amazon would want to buy enough disks to support 100% of what they promise and then sell the unused capacity at below cost when they could just buy fewer disks instead. Either way they'd be in trouble if their EC2 customers suddenly started wanting to use all the space they were promised.

It's a good point: a cloud provider could choose to try to under-provision disks instead. The problem though is that (some) disks are local to the machine; it's not easy to move physical hard disks around when the calculations are wrong (and if you have to get it right on a per-host basis, it's more likely to go wrong). It is however easy to move chunks of data around, particularly if you use something like erasure coding to give you a huge amount of flexibility.

In short, your way is a good alternative, but my guess is that buying the full capacity and selling the surplus is probably roughly cost-equivalent, and considerably less likely to end up with you not being able to sell the full capacity of any given host.

Maybe for unattached instance storage this is possible (thought not a good business decision as others have noted). But you are forgetting how Amazon sees those attached instance disks: As raw block devices from the hypervisor.

They see a computer with an attached disk of, say, 1 TB. They don't see the filesystem at all. They don't know if a sequence of zeros is unused space, or are literally zeros in data. There is no way that Amazon can use slack space because they don't know what is slack space.

This incomplete view of how disks (both instance and EBS) are used is also evident by what metrics CloudFront can track: CPU usage. Raw I/O stats for disk and network. To get richer metrics, you have to install scripts/agents running inside your EC2 instance which than beacon the info to CloudFront via the API.

In short, Amazon's current setup means they cannot use the slack space inside of attached block devices, even if they wanted to.

The way thin provisioning normally works (e.g. on ZFS or LVM) is that only non-zero blocks are stored; an unstored block is read as zero; a block written as zeroes could be "unstored". It doesn't matter whether a block of zeroes is stored or not: it still reads as zero. So a cloud provider only needs to store non-zero blocks, and thus doesn't need to provision the full capacity to back a volume.

As to whether EC2 does this I don't know - based on your observations they may not. I think it's an interesting way to build a storage product like Glacier, even if Amazon may have chosen to do something different.

Reply on news.ycombinator.com