In general, what are the advantage of a thread-per-request model? Better load balancing between cores?
You are essentially asking the operating system to do the scheduling for you. But the OS will never be able to do it perfectly as it has no knowledge of what your application is doing.
The main advantage of OS scheduling is that you get pretty good results without having to think about it at all. Pretty good, but never perfect.
From LMAX presentations, it looks like they want you to split your application into tasks [1], define a graph of task dependencies, have each core process a particular kind of task and have task processors communicate their producers via a ring buffer.
In particular, the allocation of tasks is static. The use of a ring buffer means that there is very little contention, and task processing is very efficient, but some cores might end up underutilized.
On the other hand, if you have a thread per request, and allow them to migrate between cores, idle cores can steal tasks from busy ones. So in theory you could get better utilization, but task processing is less efficient since you need to share more data between cores.
("threads" don't need to be OS threads, they can be green threads)
That said, I am not sure if GP meant this by thread-per-request, or "legacy" applications that use a thread pool, or something else.
[1]: https://www.slideshare.net/trishagee/introduction-to-the-dis...
Static allocation is a special case of scheduling. You decide which parts of the process run on which core -- the scheduling in this case is done at design or configuration time.
> On the other hand, if you have a thread per request, and allow them to migrate between cores, idle cores can steal tasks from busy ones. So in theory you could get better utilization,
Migrating your tasks between cores is nothing you can't design into your application. For example, in a typical event driven architecture where you have worker threads running each on separate core, there would be something to decide where the task is queued and usually the logic will take into account how busy particular worker thread is. The operating system does nothing in this case, what it sees is a number of threads each running on its separate core that don't need to be preempted (hopefully).
Wouldn't that work well only if the time taken by each task is predictable? I.e. you mention working on a trading system. But in a trading system you want to run the same branchless code path regardless of the kind of incoming event and whether you are sending an order or not after running the trading logic. So the individual "task" is very predictable.
On the other hand think of a task like "return all the comments for a certain page". The time taken by an individual task is unpredictable, proportional to the number of comments. So you'll regularly get one core getting enqueued a bunch of tasks with no comments, finishing quickly and then staying idle.
With work stealing, after finishing, that core would get a chance at "stealing" tasks from other threads' queues.
(of course, the architecture I am describing would be awful for a trading system)
> The operating system does nothing in this case, what it sees is a number of threads each running on its separate core that don't need to be preempted (hopefully).
Btw, I agree that pinning OS threads to each core and then layering something of your own on top of it is going to be faster. It is just that you can layer on top a green thread system (like Go), and get something thread-for-request -like.
... not that I think anyone is doing this outside of maybe some hobbyist building their own HTTP server for fun.
Of course they do. There are loads of appropriate applications. Heck, people still run CGI programs unironically.
Also, why I love the new virtual threads in Java. Will they work as good as hoped? No idea. Probably not for all use cases. But the direction to say: You know what, threads are great to program and debug compared to the alternatives, maybe let's find a way to make their performance better instead of putting up with async , is so refreshing.
Here's an example difference: in thread-per-request, any global state can be accessed from "anywhere", and thus you end up with locks, reference counts, GC, and what not. In thread-per-core, global state is sharded across cores and never accessed "from the outside", and thus needs no locks/atomics (beyond the messaging primitive).
...or possibly it's accessed from everywhere, and locks are needed again. What you're describing is kind of an ideal application architecture, not a hard rule about thread pinning.
"Possibly accessed from anywhere" is a bad design in general, and unacceptable in realtime processing, where you need to know access patterns exactly.