Hacker News Reader

Comment by tanelpoder | Hacker News Reader

I tested various fio options, but didn't notice this one - I'll check it out! It might explain why I still kept seeing lots of interrupts raised even though I had enabled the I/O completion polling instead, with io_uring's --hipri option.

edit: I ran a quick test with various IO batch sizes and it didn't make a difference - I guess because thanks to using io_uring, my bottleneck is not in IO submission, but deeper in the block IO stack...

View on HN

I think on recent kernels, using the hipri option doesn't get you interrupt-free polled IO unless you've configured the nvme driver to allocate some queues specifically for polled IO. Since these Samsung drives support 128 queues and you're only using a 16C/32T processor, you have more than enough for each drive to have one poll queue and one regular IO queue allocated to each (virtual) CPU core.

That would explain it. Do you recommend any docs/links I should read about allocating queues for polled IO?

It's terribly documented :(. You need to set the nvme.poll_queues to the number of queues you want, before the disks are attached. I.e. either at boot, or you need to set the parameter and then cause the NVMe to be rescanned (you can do that in sysfs, but I can't immediately recall the steps with high confidence).

Ah, yes, shell history ftw. Of course you should ensure no filesystem is mounted or such:

    root@awork3:~# echo 4 > /sys/module/nvme/parameters/poll_queues
    root@awork3:~# echo 1 > /sys/block/nvme1n1/device/reset_controller
    root@awork3:~# dmesg -c
    [749717.253101] nvme nvme1: 12/0/4 default/read/poll queues
    root@awork3:~# echo 8 > /sys/module/nvme/parameters/poll_queues
    root@awork3:~# dmesg -c
    root@awork3:~# echo 1 > /sys/block/nvme1n1/device/reset_controller
    root@awork3:~# dmesg -c
    [749736.513102] nvme nvme1: 8/0/8 default/read/poll queues

Thanks for the pointers, I'll bookmark this and will try it out someday.

> I tested various fio options, but didn't notice this one - I'll check it out! It might explain why I still kept seeing lots of interrupts raised even though I had enabled the I/O completion polling instead, with io_uring's --hipri option.

I think that should be independent.

> edit: I ran a quick test with various IO batch sizes and it didn't make a difference - I guess because thanks to using io_uring, my bottleneck is not in IO submission, but deeper in the block IO stack...

It probably won't get you drastically higher speeds in an isolated test - but it should help reduce CPU overhead. E.g. on one of my SSDs fio --ioengine io_uring --rw randread --filesize 50GB --invalidate=0 --name=test --direct=1 --bs=4k --numjobs=1 --registerfiles --fixedbufs --gtod_reduce=1 --iodepth 48 uses about 25% more CPU than when I add --iodepth_batch_submit=0 --iodepth_batch_complete_max=0. But the resulting iops are nearly the same as long as there are enough cycles available.

This is via filesystem, so ymmv, but the mechanism should be mostly independent.

Reply on news.ycombinator.com