Hacker News Reader

Comment by pjungwir | Hacker News Reader

I saw the section on EBS, but it didn't offer much advice. Getting good performance on networked storage is the biggest challenge to me. The last time I asked about that here [1], I got this answer:

    nasalgoat 161 days ago [-]

    The secret to EBS is to use General SSD,
    not Provisioned, but use a RAID stripe.

    The reason this works is because IOPS are provisioned
    per EBS drive and by the size of the drive. So a RAID0 
    stripe of, say, ten General SSD drives will outperform 
    the more expensive PIOPS single drive.

That sounds like a great approach, although I haven't had time to try it out yet. I'm curious if anyone else has done anything like that.

[1] https://news.ycombinator.com/item?id=12609172

View on HN

Citus Cloud uses that approach. I designed it with the arithmetic from the page about gp2, http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolume....

A useful inflection is the "minimum size for max throughput" at 214GiB. That same disk would also offers 38 minutes at the maximum throughput at 3000PIOPs. After that, performance will drop to about a fifth: that's the baseline for a 214GiB disk. Another important thing to consider that large contiguous reads and writes on EBS are up to 256KiB per IOP: most bulk transfers take up far fewer IOPS than people normally calculate.

For most workloads, Burst Is Good: for a fixed amount of money, you'll be able to obtain meaningfully better responsiveness in a program. Few workloads are pushing the maximum IOPS all the time. If you need more baseline IOPS you can make gp2 disks bigger or add more disk, depending on whether you find bandwidth has been saturated or not.

Are Citus Cloud customers generally IOPS constrained rather than throughput constrained? You're also constrained by the EBS-optimized interconnect, right? http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSOptimi...

Also, have you considered using i3 instances for their superior price per throughput/IOP? Obviously, you can't rely on EBS snapshot based backup/restore workflows on those instances, but maybe the performance gain is compelling enough to make the exercise worthwhile?

I have rejected this on the basis that stopping and starting an instance is too useful for solving problems. Even with a few striped EBS disks, instance issues solvable by re-attaching the disks via stop/start are probably ten to one hundred times more likely to occur than a disk failure.

Still, I know some people do things this way, relying on archives and HA for redundancy, and it can be pretty good.

As a random sample of operating time, probably latency constrained, where something like a local disk would be nice. We do have occasional spikes for bulk writes and reads, and the good throughput of large IOPS EBS are instrumental there.

Use multiple big* SSD volumes in RAID 0 on big* instances.

* IOPS and bandwidth are proportional to the size of volumes

* instance types have a hard cap on network and IO, proportional to the size of the instance.

Otherwise, you can try local SSD instances but there are only a few and they are incredibly expensive.

I'm working on an approach using ZFS on K8s. The idea is to write a custom "flexvolume" driver that takes set of EBS volumes an options, attaches them, create a zpool, set options, etc. In tandem, however, I'm adding to each node host a systemd service to create dm-crypt+LVM devices atop of each NVMe drive (as in the i3 instance types). The flexvolume driver will support adding SLOG and/or L2ARC devices by provisioning LVM volumes from those.

Something I'm still unclear, is whether EBS does any data checksums, in which case I could disable ZFS' to spare CPU cycles (but keep compression, of course.)

Sounds great. Is this approach somehow related to ClusterHQ's?

> I'm adding to each node host a systemd service to create dm-crypt+LVM devices atop of each NVMe drive

That sounds cool; can you describe how you're using systemd for this in a bit more detail? I do this with an Ansible playbook when the machines are provisioned.

Reply on news.ycombinator.com