Hacker News Reader

Comment by jchw | Hacker News Reader

This is probably a big deal, if it's anything like the local SSD storage on GCE. The performance of local SSDs on Google Cloud is nearing absurd compared to anything else you can find right now.

That being said, I think one of the less compelling parts of this is that it'll probably vary per instance type quite a bit, being limited to C5 to start, so if you have a workload needing way better disk I/O than CPU performance, you might have to waste. That's one thing GCP really does have on AWS, better granularity.

View on HN

> being limited to C5 to start

I3 and F1 instances also have NVMe SSD instance store volumes https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ssd-inst...

Here's the full list of instances with instance storage: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Instance...

Nice, didn't know that.

GCE local SSD was the fastest thing around when first released in 2014, but hasn't improved and has since fallen behind. GCE local SSD 4k read IOPs max out at around 200k/volume and 800k/instance w/4 volumes. I haven't measured c5 local nvme yet, but on i3, local nvme volumes run at 100-400k/volume and up to 3.3 million IOPs on the largest instance type (i3.16xl).

Hey, you seem to know a lot about this. Can you explain why the docs seem to claim they max out at 680k/volume read IOPS and 360k/volume write IOPS? They seem to be talking about 4k blocks.

https://cloud.google.com/compute/docs/disks/performance#type...

Is Google being disingenuous? I haven't benchmarked it, but I will say that anecdotally the GCP local NVMe SSDs feel very fast for things like building software compared to other setups I've done on other clouds (AWS, UpCloud.)

The sequential write performance of local NVME on GCP is... not great.

Would love some benchmarks, if you happen to have them. Frankly, I don't have any good benchmarks directly comparing GCP and AWS, but my current heuristics is that GCP "seems" to have better I/O and AWS "seems" to have higher frequency CPUs. Could be way off.

I did some benchmarks of AWS vs GCP storage [0] (NOT including local storage), and the short version is that Google's slow storage is slower than AWS's, and Google's fast storage is faster than AWS's. And I tested NVMe, but on vSphere, and it was very, very fast.

0: http://engineering.pivotal.io/post/gobonniego_results/

With more instance types and storage classes, EC2 offers a much wider spectrum of performance capabilities for CPU, storage (ephemeral and durable), and networking.

I don't see how this has to do with my question about benchmarks of sequential disk I/O on GCP instances with local NVMe SSDs. Did you respond to the wrong message?

Either way, I use both GCP and AWS but I am finding that I'm spending less on GCP. I don't believe every workload will be cheaper in GCP, but it is clearly competitive at least.

I'm responding to your comment "GCE seems to have better I/O and AWS seems to have higher frequency CPUs" - this isn't true. You can get faster I/O, networking and CPU with EC2 if you select the correct instance/storage and are willing to pay for it.

For seq read/write I've observed up to around 15000/2900 MB/s on i3.16xl and 2750/1400 MB/s on GCE w/4xlocal SSD

This is lacking in details:

- What OS image(s) were used? This would aid in reproducibility.

- What tool was used for benchmarking? Preferably with the configuration so I could reproduce it.

- What GCE instance is used? I assume you will bottleneck on something other than I/O if comparing a very small GCE instance to a very large AWS instance.

I will probably benchmark this on my own at some point now that my interest is piqued, but if you have already done the work to benchmark this I'd love to hear more. So far though it's too vague for me to use meaningfully. Is there a blog post or other material that is publicly accessible? I've Googled for such before but come up surprisingly empty handed.

I've just tried to benchmark, but I'm not sure how. I tried using dd to measure sequential speed and this was giving pretty good results on Google Cloud:

> dd if=/dev/zero of=/dev/disk/by-id/google-local-ssd-0 bs=8192

> 63356186624 bytes (63 GB, 59 GiB) copied, 36.6496 s, 1.7 GB/s

> dd of=/dev/zero if=/dev/disk/by-id/google-local-ssd-0 bs=8192

> 63356186624 bytes (63 GB, 59 GiB) copied, 17.5434 s, 3.6 GB/s

...But I'm perplexed by why it cuts off at 64 GB.

On Amazon I couldn't figure out how to get decent speeds at all. Comparable to my Google Cloud box, I set up an i3.xlarge, which had 4 CPUs and 1 SSD. While the dd operation was working fine, it slowed down to a paltry 430 MB/s quickly and never recovered. I doubt this is the true sequential performance of the drive, so I gave up.

I would've attempted to do multiple SSD configurations, but those seemed even harder. mdadm-based RAIDs were clearly bottlenecking somewhere other than hardware, because they were slower than writing or reading directly.

tl;dr: It turns out these new-fangled NVMe instance store setups are too complicated for me to understand.

You'll need to apply some SSD specific methodology to correctly characterize their performance. Writing zeros to some SSDs will trigger special logic in their flash translation layer (FTL) that will prevent the writes from hitting the flash media. They simply note in the metadata that you wrote some zeros, and reading them back will also be impossibly quick.

This isn't the case only on new-fangled NVMe devices...

Reply on news.ycombinator.com