Hacker News Reader

Comment by api | Hacker News Reader

What's funny about io_uring is that the blocking syscall interface was always something critics of Unix pointed out as being a major shortcoming. We had OSes that did this kind of thing way back in the 1980s and 1990s, but Unix with its simplicity and generalism and free implementations took over.

Now Unix is finally, in 2021, getting a syscall queue construct where I can interact with the kernel asynchronously.

View on HN

Maybe Linux will get scheduler activations in the near future, another OS feature from the 90s that ended up in Solaris and more or less nowhere else. "Let my user space thread scheduler do its work!"

There's already plans for a new futex-based swap_to primitive, for improving userland thread scheduling capabilities. There was some work done on it last year, but it was rejected on LKML. At this rate, it looks like it will not move forward until the new futex2 syscall is in place, since the original API is showing its age.

So, it will probably happen Soon™, but you're probably still ~2 years out before you can reliably depend on it, I'd say.

Scheduler activations don't require swap_to.

The kernel wakes up the user space scheduler when it decides to put the process onto a cpu. The user space scheduler decides which user space thread executes in the kernel thread context that it runs in, and does a user space thread switch (not a full context switch) to it. It's a combination of kernel threads and user space (aka "green") threads.

It might! I'm not sure if it's an exact fit or not but the User Managed Concurrency Groups work[1] Google is trying to upstream with their Fibers userland-scheduling library sounds like it could be a match, and perhaps it could get the upstreaming itcs seeking.

[1] https://www.phoronix.com/scan.php?page=news_item&px=Google-F...

I think some of the *BSDs have (or had) it. Linux almost got it at the turn of the millennium, with the Next Generation Posix Threading project, but then the much simpler and faster NPTL won.

We talked about adding that to Linux in the 90s too. A simple, small scheduler-hook system call that would allow userspace to cover different asynchronous I/O scheduling cases efficiently.

The sort of thing Go and Rust runtimes try to approximate in a hackish way nowadays. They would both by improved by an appropriate scheduler-activation hook.

Back then the idea didn't gain support. It needed a champion, and nobody cared enough. It seemed unnecessary, complicated. What was done instead seemed to be driven by interests that focused on one kind of task or another, e.g. networking or databases.

It doesn't help that the understandings many people have of performance around asynchronous I/O, stackless and stackful coroutines, userspace-kernel interactions, CPU-hardware interactions and so on are not particularly deep. For example I've met a few people who argued that "async-await" is the modern and faster alternative to threads in every scenario, except for needing N threads to use N CPU cores. But that is far from correct. Stackful coroutines doing blocking I/O with complex logic (such as filesystems) are lighter than async-await coroutines doing the same thing, and "heavy" fair scheduling can improve throughput and latency statistics over naive queueing.

It's exciting to see efficient userspace-kernel I/O scheduling getting attention, and getting better over the years. Kudos to the implementors.

But it's also kind of depressing that things that were on the table 20-25 years ago take this long to be evaluated. It's almost as if economics and personal situations governs progress much more than knowledge and ideas...

Actually, I think the biggest obstacle is that as cool as scheduler activations are, it turns out that not many applications are really in a position to benefit from them. The ones that can found other ways ("workarounds") to address the fact that the kernel scheduler can't know which user space thread to run. They did so because it was important to them.

    It's almost as if economics and personal situations 
    governs progress much more than knowledge and ideas...

That has always been the case and will probably always be the case.

Solaris had this over a decade ago now with event ports which are basically a variant of Windows IOCP:

https://web.archive.org/web/20110719052845/http://developers...

So at least one UNIX system had them a while ago.

People keep saying this but IME that's not how the Solaris Event Ports API works at all. The semantics of Solaris Event Ports is nearly identical to both epoll+family and kqueue. And like with both those others (ignoring io_uring), I/O completion is done using the POSIX AIO interface, which signals completion through the Event Port descriptor.

I've written at least two wrapper libraries for I/O readiness, POSIX signal, file event, and user-triggered event polling that encompass epoll, kqueue, and Solaris Event Ports. Supporting all three is relatively trivial from an API perspective because they work so similarly. In fact, notably all three let you poll on the epoll, kqueue, or Event Port descriptor itself. So you can have event queue trees, which is very handy when writing composable libraries.

I remember IPX has similar communication model. To read from the network, you post an array of buffer pointers to the IPX driver and can continue to do whatever. When the buffers are filled, the driver calls your completion function.

Do you have some references?

io_uring is Linux specific. BSD offers kqueue instead, introduced in 2000.

I believe both are limited to specific operations, mostly IO, and aren't fully general asynchronous syscall interfaces.

The entire point of the article is that it is not just for IO.

The OP said mostly IO not only.

> The entire point of the article is that it is not just for IO

I went through the list of operations supported by `io_uring_enter`. Almost all of them are for IO, the remainder (NOP, timeout, madvise) are useful for supporting IO, though madvise might have some non-IO uses as well. While io_uring could form the basis of a generic async syscall interface in the future, in its current state it most certainly is not.

The article mostly talks about io_uring enabling completion based IO instead of readiness based IO.

AFAIK kqueue also supports completion based IO using aio_read/aio_write together with sigevent.

> AFAIK kqueue also supports completion based IO using aio_read/aio_write together with sigevent.

If you can point to a practical example of a program doing it this way and seeing a performance benefit, I would be curious to see it. I did some googling and didn't really even find any articles mentioning this as possibility.

kqueue is widely considered to be readiness based, just like epoll, not completion based.

What you wrote sounds like an interesting hack, but I'm not sure it counts for much if it is impractical to use.

I don't know if it offers any practical benefits for sequential/socket IO. But AFAIK it's the way to go if you want to do async random-access/file IO.

Afaik, Nginx used this hack for aio on FreeBSD many years. It is superrior to aio on Linux. That is why Netflix used nginx together with FreeBSD.

Now nginx could use threads for file io, and linux file performance became ok as well.

It feels like it's been ten years since I fiddled with this on FreeBSD but EVFILT_AIO worked surprisingly well with aio and kqueue on spinning disk.

POSIX AIO uses `struct sigevent` to notify completion. On FreeBSD you can set ->sigev_notify to SIGEV_KEVENT, and ->sigev_notify_kqueue to a kqueue descriptor.

As for your "performance benefit" question and nobody using it over readiness based approaches: Well yes. No idea how it compares with a traditional readiness based socket program. But it's kind of not the point either. AFAIK the scenario where POSIX AIO makes most sense is files on disk.

There's nothing stopping io_uring from gaining more non-IO operations, and indeed that appears to be the plan.

kqueue is not really a replacement for io_uring. That said, FreeBSD can make a POSIX AIO request signal completion to a kqueue, which is a similar model of async I/O.

But as has been pointed out elsewhere, io_uring is also about making other syscalls asynchronous.

Reply on news.ycombinator.com