Hacker News Reader

Comment by rtkwe | Hacker News Reader

Yeah how many people are running apps on servers served at all or even partially by NVMe SSDs? Where I work for our on prem stuff it's basically all network storage.

View on HN

You can do pretty amazing things with well designed (onprem) networked storage with NVMe drives/arrays.

I replaced the company I was working with at the time's traditional "enterprise" HPE SANs with standard linux servers running a mix of NVMe and SATA SSDs that provided highly available, low latency and decent throughput iSCSI via network.

Gen 1 back in 2014/2015 did something like 70K random 4k read/write IOP/s per VM (running on Xen back then) and would just keep scaling till you hit the clusters 4M~ IOP/s limit (minus some overhead obviously).

Gen 2 provided between 100K and 200K random 4k to each VM to a limit of about 8M~ on the underlying units (which again were very affordable and low maintenance).

This provided very good storage performance (latency, throughput and fast / minimally if at all disruptive fail-over and recovery) for our apps, some of them were written in highly blocking Python code and needed to be rewritten async to get the most out of it, but it made a _huge_ (business changing) difference and saved us an insane amount of money.

These days I've moved into consulting and all the work I do is on GCP and AWS but I do miss the hands on high performing gear like that.

Old stuff now but the links are https://www.dropbox.com/s/rdojhb399639e4k/lightning_san.pdf?... and https://smcleod.net/tech/2015/07/24/scsi-benchmarking/ and there's a few other now quite dated posts on there.

I hope it's worth noting here that a increasing variety of formerly vertically integrated storage systems management layers and more capabilities have become available in VM form with per GB licensing models.

IF you touch health care or finance outside of the trading rooms, Hitachi Data Systems (Ventara but I am not seeing the Hitachi name disappear it's too important I think I'm mindshare I know) the most surprisingly good and common installation. HDS wasn't scared to use OSS and build open platforms.

Big edit sorry

I wandered away from concluding with my own little dream to decouple the block implementation from the fs how DAOS does only with a full selection from the commercial file systems available, paying for usable capacity and not raw installed drive specifications. Paying for peritoneal at enterprise mark-up sucks. Meanwhile not too many people seem to be aware that you can run very small budget and scale file systems that were only available in multiples of house prices only a couple of years ago. The latest all flash NetApp filer is 20k and I don't imagine many people who have the knowledge to debate about the issues can't economically justify that even in a home lab. Executive care options can cost less only it's not many acquisitions that can cost you more just owning but with orders of magnitude difference between the costs.

For network storage some of his points are even stronger. Sure, the page cache becomes more useful as latency goes up but it also becomes more important to send more I/O at once, something that is hard to do with blocking APIs like read(2) and write(2). The page cache is pretty good at optimizing sequential I/O to do this, but not random I/O or workloads where you need to sync().

There are a series of hardware updates and software bottlenecks to work through before access becomes more common, the performance will bubble up from below starting with more widespread NVMe devices, then faster NVMe over fabrics hardware will become more common, and drivers, hypervisors, filesystems, and storage apps will likely have to rethink things to re-optimize. That means different times in terms of showing up in the cloud/on-prem/etc.

This is starting to change a bit because of things like the DPUs that companies are making. Basically it's an intelligent PCI-e <=> network bridge that lets you emulate/share PCI-e devices on the host while the actual hardware (NVMe storage, GPU, etc.) is located elsewhere. This lets you reconfigure the host in software without having to physically change the hardware in the servers itself. It also lets you change the way you have things in the server rack since everything doesn't need to be able to physically fit into every other server case.

EDIT: informative article about DPUs, https://www.servethehome.com/what-is-a-dpu-a-data-processing...

The linked article states that DPUs behave like a PCIe device implementing the NVME protocol but instead of directly connected storage it can forward requests over network (fabric) via NVMeoF.

This doesn't look like a generic PCIe over network/fabric bridge. Did I misunderstand you or did I fail to locate that information in the linked article?

It's specifically NVMe over network, not PCIe over network.

The Fungible DPU mentioned in the article is capable of it, it's only very briefly mentioned in that article but it does link to a more in depth article about it. https://www.servethehome.com/fungible-f1-dpu-for-distributed...

IBM research put out a bunch of white papers a few years back about the huge performance difference between spinning rust, SATA SSDs, and NVMe. The performance gains were amazing, especially with things like parquet.

We did a big upgrade upgrade a couple years ago moving all of our DBs onto NVMe SSDs. We get significant improvements to our query times.

Fast SSDs are pretty cheap nowadays.

not apps and not running there, but we bought a NAS for our really small compute cluster and just put 16 cheap (=marketed as read-intensive) NVME drives in (well, originally 8 – but then there was money left, which had to be spent fast), because IOPS are magnitudes better than HDDs and for around 3 times $/TB the hassle of configuring some caching just wasn't worth it in our opinion.

Performance on par with typical Desktop-SSDs over NFS (multiple clients possible). Definitely better than the old 1TB/disk-RAID5.

There's more to life than "apps on servers".

People doing audio/video work with lots of input streams can also max out disk I/O throughput (quite easily before NVMe SSDs; not so much anymore).

> network storage

Do you mean cloud storage? as in, other people's computers?

I conjecture he means SANs, iSCSI, NFS, Fibre Channel and other on-prem, but still not local to the server where the compute is running.

It probably means NFS.

It's a pretty common pattern to have a fleet of big beefy VM hosts all backed by a single giant SAN on a 10gbe switch. This lets you do things like seamlessly migrate a VM from one host to another, or do a high availability thing with multiple synchronized instances and automatic failover (VMWare called this all "vMotion"). In any case, lots of bandwidth to the storage, but also high latency, at least relative to a locally-connected SATA or PCIe SSD.

So yeah, if that's your setup, you don't have much of an option in between your SAN and allocating an in-machine ramdisk, which will be super fast and low latency, but also extremely high cost.

Why not consider nVME in this case then as cheaper than RAM, slower than RAM, but faster than network storage? I don't know how you handle concurrency btwn VMs or virtualize that storage, but there must be some standard for that?

I think a lot of it depends what the machines are used for. I'm not actually the IT department, but I believe in my org, we started out with a SAN-backed high availability cluster, because the immediate initial need was getting basic infrastructure (wiki, source control, etc) off of a dedicated machine that was a single point of failure.

But then down the road a different set of hosts were brought online that had fast local storage, and those were used for short term, throwaway environments like Jenkins builders, where performance was far more important than redundancy.

I’m laughing a little bit because an old place I used to work had a similar setup. The SAN/NAS/whatever it was was pretty slow, provisioning VMs was slow, and as much as we argued that we didn’t need redundancy for a lot of our VMs (they were semi-disposable), the IT department refused to give us a fast non-redundant machine.

And then one day the SAN blew up. Some kind of highly unlikely situation where more disks failed in a 24h period than it could handle, and we lost the entire array. Most of the stuff was available on tapes, but rebuilding the whole thing resulted in a significant period of downtime for everyone.

It ended up being a huge win for my team, since we had been in the process of setting up Ansible scripts to provision our whole system. We grabbed an old machine and had our stuff back up and running in about 20 minutes, while everyone else was manually reinstalling and reconfiguring their stuff for days.

Ha, that's awesome. Yeah, for the limited amount of stuff I maintain, I really like the simple, single-file Ansible script— install these handful of packages, insert this config file, set up this systemd service, and you're done. I know it's a lot harder for larger, more complicated systems where there's a lot of configuration state that they're maintaining internal to themselves and they want you to be setting up in-band using a web gui.

I experienced this recently trying to get a Sentry 10 cluster going— it's now this giant docker-compose setup with like 20 containers, and so I'm like "perfect, I'll insert all the config in my container deployment setup and then this will be trivially reproducible." Nope, turns out the particular microservice that I was trying to configure only uses its dedicated config file for standalone/debugging purposes; when it's being launched as part of the master system, everything is passed in at runtime and can only be set up from within the main app. Le sigh.

Lol. Hope may not be a strategy, but you need a strategy if you don't want to really learn that lession the hard way: if it hurts do it more often.

It depends on what you want to spend your labor dollars on. I run a complex system on HPE storage which is all SSD or NVME and is super fast. You pay, but the vendor mostly takes care of it operationally — I pay for about 20% of a SAN sme, mostly to do maintenance on the storage fabric.

Reply on news.ycombinator.com