Linux Achieves 5.1M IOPS Per-Core with AMD Zen 3 and Intel Optane | Hacker News Reader

Linux Achieves 5.1M IOPS Per-Core with AMD Zen 3 and Intel Optane

phoronix.com

124 pointsksec21 comments

Wendell from Level1Techs has been beating his drum about pushing IOPS on these newer hardware platforms, and has some great videos about his journey and what is fundamentally broken under the hood with how software interacts with and fails to anticipate hardware as fast as the new Optane drives.

https://www.youtube.com/watch?v=RfrMnVpPuVw - The initial video attempting to clear 20 million IOPS on Threadripper

https://www.youtube.com/watch?v=WEMszbMUj68 - 15 million IOPS on Xeon 8380s (you can see how this is going)

https://www.youtube.com/watch?v=9Gu_rT8N0-U - Discussion with Allyn Malventano of Intel

My broad interpretation of this is that the software layer was not ready for things like 64+ core systems on a single socket, and PCI-E 4 drives moving data this fast. Nice to see it's starting to catch up.

It is not only the software layer that was not prepared for so many. Some NVMe devices support only 64 I/O queues (this is how many you need for Windows compatibility certification). Some NICs only have 32 receive queues. It's now trivial to build a box with 256 threads, so these numbers don't look great.

> Some NVMe devices support only 64 I/O queues (this is how many you need for Windows compatibility certification).

You're talking specifically about server environments there, right? Because consumer NVMe devices started out with far lower queue counts (eg. 7) and even the top consumer SSD product lines have struggled to keep pace with growing Threadripper core counts. 64 queue support is still something of a luxury among consumer NVMe devices.

That sucks. I can't recall having personally seen one that didn't support at least 64 submission queues.

Samsung's current flagship 980 PRO consumer drive supports 128 queues, but the previous generation (970 PRO/EVO/EVO Plus) only supported 32. Their first two generations were limited to 8 and 7 queues. I wouldn't be surprised if these limits also applied to their entry-level enterprise SSDs that used the same controllers.

How can we find this information out easily (not just for Samsung, but for others)?

Even Anandtech only tested 32 queues on the 980 Pro [0].

[0] - https://www.anandtech.com/show/16087/the-samsung-980-pro-pci...

Number of queues is not the same as the number of operations in a queue.

Supporting more queues means that you need less communication and synchronization between CPU cores; ideally, you have at least one queue per core so that each CPU core can submit commands to the SSD independently.

I didn't test queue depths higher than 32 when I reviewed the 980 PRO, because it's a consumer drive and consumer workloads hardly ever produce double-digit queue depths. Even going all the way up to QD32 is fairly unrealistic.

Server workloads can be quite different, and some of them genuinely can result in a large number of simultaneous IO requests. Consumer workloads have barely even started to evolve to take advantage of SSDs, and hard drives aren't quite entirely gone from the consumer PC market.

As for how to check how many queues a NVMe drive supports: the easy way is to install the drive in a system with a very large number of CPU cores, and see how many queues the Linux NVMe driver creates (check the drive's sysfs directory). My understanding from reading the NVMe spec is that you can't necessarily query the drive itself for the number of supported queues after the operating system's NVMe driver has already done that and set up its queues.

Is this not the correct way to do it?

  $ sudo nvme get-feature /dev/nvme0 -f 7
  get-feature:0x7 (Number of Queues), Current value:0x7f007f

That's the number of queues requested by the host system. If you ran that on a system with four CPU cores (and no SMT), you'd likely see that same drive return 4 instead of 7.

Set-feature for that value is how the host system indicates how many queues it is requesting. A subsequent get-feature returns how many queues were actually allocated. This can be less than the maximum supported by the drive, such as if the OS didn't request very many queues (due to you not having very many CPU cores), or if the drive is being virtualized and some of its queues are already allocated to a different virtual function.

>Consumer workloads have barely even started to evolve to take advantage of SSDs, and hard drives aren't quite entirely gone from the consumer PC market.

I really wish there is some industry push towards this. But last time ( I think around 6-7 years ago ) I try to push for this the answer was Information Security and Data erase / recovery would not allow SSD to be used in certain industry and domain. It has to be HDD.

I wonder if this has changed in the era of Smartphone and Tablet where NAND are now the default solution.

The latest generation of video game consoles come with NVMe SSDs as standard components, and game developers are starting to target solid state storage as a baseline requirement. So there's some visible progress, with support from GPU hardware vendors, and Microsoft bringing new storage APIs to the table (including a clone of io_uring).

I think any security concerns that might have once privileged hard drives over SSDs have been settled. There are plenty of options for FIPS 140-2/3 certified self-encrypting drives, and broad support for Sanitize operations that instruct the drive to wipe all user data even from storage areas that may not be accessible to the host system (ie caches, spare blocks. Any holdouts still preferring hard drives for security reasons are most probably deluded, and certainly are not influencing the storage market in any meaningful way.

I think the most fundamental way in which the software ecosystem is unprepared doesn't require Optane storage or 64+ core systems to demonstrate.

When CPU core counts first started growing, storage was still effectively single-threaded: a hard drive can only work on servicing one request at a time (though you can get a small throughput increase by keeping several requests in the queue), and scaling up to arrays of many drives was much more effective at increasing sequential throughput than random IO performance.

With flash-based storage, the SSD now has far more parallelism than your CPU. Having one pending IO operation per CPU core isn't enough to keep your SSDs busy. Having one pending IO per software thread and doing lots of context switches to run more threads than you have cores still won't keep your drives busy, unless you're only doing IO in large chunks. So synchronous, blocking IO APIs are unable to make full use of your storage performance. Optane's low latency doesn't save you from this problem; it just highlights that context switches are also too expensive.

Funny that, some people argue that the future of I/O is synchronous memory-mapped devices.

That can potentially work, but only by getting the filesystem/block storage/PCIe layers of the IO stack out of the way, and even then it's only plausible for Optane or similarly low-latency storage. It'll never work for flash, because the latency of flash will never be low enough.

I need some context.

Linux developers have made improvements to specific parts of the IO sub system. These changes will give Linux better IOPS performance.

Prior test are quoted at 3,8 IOPS. They ran the old code on a MD Ryzen Threadripper 3970X at 3,8 IOPS.

Now he got a better / fast CPU AMD Ryzen 9 5950X and it achieves an impressive 5.1M

It seems to me that the new test does not make it clear how much code changes helped and how much a new CPU helped?

Running the old version on the new CPU or the other way around would make it easier to compare?

My understanding from Jens Axboe's tweets is that he ran the same software test on his old hardware and his new hardware. Separately, he's been using this test for quite a while to track the progress of software optimizations.

Because hes only using a single CPU core I'm not sure how much the CPU would have helped but you're right he hasn't quantified this.

He's the lead developer of io_uring a new kernel mechanism for doing I/O that is much more effecient for these large IOPS workloads and it itself has been getting lots of optimisation in every recent kernel so it could be both.

Linux dominance in the datacenter is about to grow even more. Awesome news!

I genuinely want to know more. What specific model Optane drive was used? A lot of the newer stuff is greatly hampered by the QLC backend.

There are no products that mix second-generation Optane with QLC. The only two products that mix first-generation Optane with QLC are consumer drives that are don't work on AMD platforms. Jens is testing using the second-generation server Optane drives (P5800X), and getting performance that's completely impossible with any of the consumer Optane products, because all of those are still limited to PCIe gen3 speeds.

Reply on news.ycombinator.com