benlwalker

On very recent Linux kernels you can open the raw NVMe device and use the NVMe pass thru ioctl to directly send NVMe commands (or you can use SPDK on essentially any Linux kernel) and bypass whatever the fsync implementation is doing. That gives a much more direct test of the hardware (and some vendors have automated tests that do this with SPDK and ip power switches!). There's a bunch of complexity around atomicity of operations during power failure beyond just flush that have to get verified.

But the way you tested is almost certainly valid.

NVMe is certainly not more complex than either AHCI or SAS from a driver or hardware interface standpoint. And I've written drivers for all three that run in production today. Now maybe the controller design is more difficult - I wouldn't know - but there are several vendors that sell NVMe controllers out there so it can't be all that bad.

And as noted in the other comments, this is about simplifying support for things like connectors, LEDs, expanders, and software in data centers where they're mixing NVMe SSDs and HDDs today. It's just simpler to have fewer parts and software differences.

To expand on the batching details, I haven't seen exactly what he's doing here, but historically the numbers quoted are from benchmarks that submit something like 128 total queue depth in alternating batches of 64. The disks hit max performance with a much lower queue depth than 64. So it's very unrealistic perfect batching that you'll never see in a real application. The workload is only a useful tool for optimizing the code, which is exactly what he's using it for.

There's one more aspect that I don't think gets enough consideration - you can batch many operations on different file descriptors into a single syscall. For example, if epoll is tracking N sockets and tells you M need to be read, you'd make M + 1 syscalls to do it. With io_uring, you'd make 1 or 2 depending on the mode. Note how it is entirely independent of either N or M.

I don't have any formal comparisons handy but there's lots of NVMe-oF benchmarks here: https://spdk.io/doc/performance_reports.html

NVMe-oF can do both TCP (akin to iSCSI) and RDMA (akin to iSER). There are likely several reasons NVMe-oF is faster, but one big one is that each connection within a session shares state in iSCSI, so you either process all connections for a session on a single thread or you take locks. In NVMe-oF, it's possible to keep every connection entirely independent, so the NVMe-oF implementations scale better.

There's more cool stuff coming in this area too. For a long time there's been the virtio family of protocols for shuttling IO to something outside QEMU to handle. Originally that was always KVM and the implementation is called vhost. Then later it became clear that these same messages could be sent to another user space process to handle instead (called vhost-user). These work great for creating virtio devices in the guest. But operating systems like Windows don't have virtio device drivers in-box, so it's a little annoying.

Recently, a new protocol to replace virtio has been defined. It is modeled on vfio ioctls and currently only can forward to another user space process, so we're calling it vfio-user. With this protocol, it's possible to emulate any PCI device rather than only virtio devices. Projects like SPDK (what I work on) can now use this to present fully emulated NVMe devices into guests and back them with whatever actual storage is available (a file, something over the network, a real NVMe SSD, etc). This allows an OS, including Windows, to boot from the virtual disk using it's in-box NVMe driver. This hasn't quite made it into a QEMU release yet, but it's close!

eBPF the bytecode is not particularly limited. You can parse complex formats like Arrow or Parquet even. The Linux kernel overlays a verifier on top which adds all sorts of draconian restrictions (for good reason). When people talk of eBPF they don't always mean to include the Linux verifier limitations as well. In particular, that nvmexpress working group link in the parent post does not say one way or the other.

They don't actually say that the lambda executes any closer to the data than any other instance. It would make sense for these to operate on the S3 nodes of course, but the description doesn't say one way or the other.

I think, as with anything, it can be used in bad faith. By taking advantage of the fact that many of the stakeholders are only able to give an idea some basic consideration due to time constraints, it's often possible to build consensus around an idea that isn't actually the best one. And by the time someone with a better idea comes around, the idea that has already built consensus can use that power to squash it.

But that assumes the person using this process is acting in bad faith to begin with (they're not pursuing the best idea, but rather their idea). If this technique is used in good faith with an open mind, it's one of the most effective ways to deal with large organizations.

Since the project I work on (https://spdk.io) largely produces a set of executables as output, it was most natural to write the tests in bash. There's one top level bash script that kicks off the full suite of tests and thousands and thousands of lines of tests all written as bash scripts stringing together calls to these executables.

One of these tests is to run shellcheck against all of the scripts in the repo. We don't allow any modifications to scripts without shellcheck giving them the green light now. The quality of our tests has increased dramatically since this was instituted - it's a really great tool.

Whether you get more IOPs with smaller I/Os depends on a number of things. Most drives these days are natively 4KiB blocks and are emulating 512B sectors for backward compatibility. This emulation means that 512B writes are often quite slow - probably slower than writing 4KiB (with 4KiB alignment). But 512B reads are typically very fast. On Optane drives this may not be true because the media works entirely differently - those may be able to do native 512B writes. Talk to the device vendor to get the real answer.

For at least reads, if you don't hit a CPU limit you'll get 8x more IOPS with 512B than you will with 4KiB with SPDK. It's more or less perfect scaling. There's some additional hardware overheads in the MMU and PCIe subsystems with 512B because you're sending more messages for the same bandwidth, but my experience has been that it is mostly negligible.

The benchmark builds to build/examples/perf and you can just run it with -h to get the help output. Random 4KiB reads at 32 QD to all available NVMe devices (all devices unbound from the kernel and rebound to vfio-pci) for 60 seconds would be something like:

perf -q 32 -o 4096 -w randread -t 60

You can specify only test specific devices with the -r parameter (by BUS:DEVICE:FUNCTION essentially). The tool can also benchmark kernel devices. Using -R will turn on io_uring (otherwise it uses libaio), and you simply list the block devices on the command line after the base options like this:

perf -q 32 -o 4096 -w randread -t 60 -R /dev/nvme0n1

You can get ahold of help from the SPDK community at https://spdk.io/community. There will be lots of people willing to help.

Excellent post by the way. I really enjoyed it.

The system that was tested there was PCIe bandwidth constrained because this was a few years ago. With your system, it'll get a bigger number - probably 14 or 15 million 4KiB IO per second per core.

But while SPDK does have an fio plug-in, unfortunately you won't see numbers like that with fio. There's way too much overhead in the tool itself. We can't get beyond 3 to 4 million with that. We rolled our own benchmarking tool in SPDK so we can actually measure the software we produce.

Since the core is CPU bound, 512B IO are going to net the same IO per second as 4k. The software overhead in SPDK is fixed per IO, regardless of size. You can also run more threads with SPDK than just one - it has no locks or cross thread communication so it scales linearly with additional threads. You can push systems to 80-100M IO per second if you have disks and bandwidth that can handle it.

Plug for a post I wrote a few years ago demonstrating nearly the same result but using only a single CPU core: https://spdk.io/news/2019/05/06/nvme/

This is using SPDK to eliminate all of the overhead the author identified. The hardware is far more capable than most people expect, if the software would just get out of the way.

I tried to switch to Signal the other day. In the US on Android, most people use the default messaging app and that means a lot of MMS/RCS messages. All the messages between myself and my wife are RCS, for example. Signal failed to import these - it only did SMS. I'd love to see someone tackle this problem much like this WhatsApp migration. It's essentially a deal breaker for people in this situation.

Much of this is written in the context of the new vfio-user proposal he linked at the end. This is a new, more flexible mechanism for implementing device emulation in a separate process. That emulation code can certainly be written in rust, providing a gradual path toward conversion.

As background, vfio-user was at least partially born of some discussions and a proof of concept last summer for using SPDK to provide virtual NVMe devices into QEMU. These are intended to be high performance devices.

Except SPDK is written in C. So maybe the title should be rewrite SPDK in Rust!

I think QUIC-the-transport is potentially an interesting transport for storage, but the data there is often already encrypted, so the double encryption is a waste.

Is it possible to only encrypt the QUIC headers but leave the data unmodified? That idea just popped into my head and I'm not sure it makes any sense, but maybe that would provide the best of both worlds.

SPDK's RocksDB integration really hasn't gotten a lot of love. There's really two main challenges we hit and then never revisited.

First, the IO threads in RocksDB are a thread pool that assume they perform blocking operations. That doesn't jive with SPDK's async model (nor io_uring's). We're having to message pass to an async thread and block on semaphores in the IO threads.

Second, RocksDB was heavily reliant on the kernel page cache to make it fast.

Both of these things could have changed since we last worked on the integration. I haven't kept up with RocksDB development recently.

source: am SPDK maintainer

io_uring is a fantastic development for the kernel, and I really can't praise it enough.

However, there's still lots of reasons to use SPDK. Performance is still significantly better[0], and you can directly access all the of the NVMe features on the device without going through any abstraction layers.

[0] https://spdk.io/news/2019/05/06/nvme/

DPDK runs on Linux and FreeBSD officially. All of the momentum is on Linux though*

*I was recently part of an effort to add FreeBSD testing to DPDK because SPDK's FreeBSD test agent kept failing when we updated DPDK. I'm a core maintainer for SPDK, which is basically DPDK for storage.

I assumed so, but then why all the comparisons to file systems that are designed to run on top of a flash translation layer? I would have expected something in big bold letters at the top saying this is designed to run without an FTL, whereas the other filesystems rely on one to wear level. For example, this is right at the top:

"Writing to flash is destructive. If a filesystem repeatedly writes to the same block, eventually that block will wear out. Filesystems that don't take wear into account can easily burn through blocks used to store frequently updated metadata and cause a device's early death."

This isn't true. It's only true for filesystems that aren't running on a regular block storage device. It's even more egregious to use the word block because almost everyone assumes that means a logical block. They must be using it to mean a NAND page, which has entirely different behavior and does wear out.

All of the flash SSDs I am familiar with have no fixed relationship between a logical block and the actual NAND media backing it. In other words, the device automatically wear levels internally and the only control the user has over wear is the total number of writes they perform.

Given that, I don't understand the discussion of wear leveling in this FS. Maybe someone can explain further.

Jens' benchmark for SPDK quoted there is far off from the numbers we (the SPDK community) measure. We are able to replicate his io_uring numbers though, so we agree that the new interface is a large improvement. We're working to make full benchmarking data available shortly.

I'm the technical lead for the Storage Performance Development Kit (http://spdk.io). I have one of these in my development system and we're hoping to post benchmarks fairly soon. SPDK further reduces the latency at QD 1 by several microseconds. It's an impressive device!

I'm one of the authors of SPDK (which includes an NVMe driver and an IOAT driver). If the community wants to add rust bindings to those two components I'd be very supportive.

Submitting and completing a 4k I/O using SPDK is about 7 times more CPU efficient than the equivalent operation with libaio, which is opening a raw block device with O_DIRECT.

Said another way, on a recent Xeon CPU you can expect to drive somewhere around 3 million 4k I/O per second with SPDK per core. With libaio, you can do about 450,000 per core off the top of my head.

SPDK has no locks or cross core communication required, so it scales linearly with additional CPU cores. Blk-mq in the kernel also helped the kernel scaling problem significantly, but I'm not sure if it is perfectly linear yet.

Most applications need something like a filesystem to function - there is no denying that. Using SPDK requires applications to implement at least the minimal set of features in the filesystem that their application needs. Many databases and storage services already bypass the filesystem or use the filesystem as a block allocator only, so it is not a big leap from there to SPDK.

Sharing CPU resources between user space drivers is certainly a challenge. The best way to view this is that the kernel provides a general purpose solution for sharing resources with some associated overhead. Tools like DPDK and SPDK let you opt out of that, but now you are responsible for intelligently sharing the hardware.

You, as the application developer, have a distinct advantage though - you only need to solve the problem for your application, and using that knowledge can often lead to more efficient solutions. This may mean dedicating cores to the network or disk, or it may mean working in fixed sized batches, etc.

Dropping in to plug SPDK (http://spdk.io), which is like DPDK but for storage devices. This will become increasingly relevant as SSDs become much faster with next generation media, and storage has the added benefit that all SSDs have one of a couple standardized interfaces and can share drivers.

The equivalent layer of a TCP/IP stack for storage is probably a filesystem, and the kernel filesystems and block layers are at least as inefficient as the network stack for similar reasons.

http://pmem.io - for byte addressable form factors

http://spdk.io - for block device form factors