Why should you use Tokio vs. threads in Rust? | Hacker News Reader

Why should you use Tokio vs. threads in Rust?

twitter.com

82 pointswspeirs47 comments

Rust Async WG member here: This question comes up pretty regularly when people first learn about async Rust. I suspect it’s because performance is often emphasized as the main benefit of async Rust.

In my opinion performance is always relative and dependent on the workload. Instead it’s worth looking more at which capabilities async Rust provides which aren’t present in non-async Rust. This includes ad-hoc cancellation, ad-hoc concurrency, and combining the two into things such as timeouts.

I wrote a post explaining this in detail a few months ago: https://blog.yoshuawuyts.com/why-async-rust/

Does async have an impact on tail latency? I feel like async + iouring should provide substantially better consistency for latency (+ lower) but I haven’t seen any measurements confirming this yet.

For me it's also ability to run multiple concurrent actions on shared state without need for synchronization, Arc or dealing with Rust lifetimes. A simple loop with select! is hard to overrate.

You can multiplex lightweight threads or async over kernel threads.

If you're serving a single connection with 1 thread then you're wasting that thread unless you do enough computation to fill that logical hardware thread's capacity.

In other words it's a good idea to use both together.

I wrote a 1:M:N scheduler in Rust, C and Java. It multiplexes N lightweight threads over M kernel threads and has 1 scheduler thread.

https://GitHub.com/samsquire/preemptible-thread

Nginx has multiple worker processes and an event loop in each I think.

There's no reason your event loop couldn't be on each thread.

I'm also thinking of separating recv and send on different threads so you can send and receive simultaneously.

Maybe 20 years ago Sun changed the Java threading model from m:n to 1:1. Because of problems.

Loom seems to be a new implementation of m:n Threading for Java. How does your implementation compare?

---

Could not find references from 20 years ago, but this paper from 2014 is nice:

"Earlier M: N was considered best among all because of its performance benchmarks. Currently the Linux and Solaris have shifted from M: N to 1:1 thread model despite of the advantages of M:N. Our emphasis will be on 1:1 Model semantics for shifting. This paper gives a thorough review of operating system thread models. We have researched factors which compelled different OS platforms to shift to 1: 1 Model."

https://www.researchgate.net/publication/270217184_Thread_mo...

> Loom seems to be a new implementation of m:n Threading for Java.

Kotlin successfully introduced green threads (viz. coroutines) to the JVM world way back when. Kotlin constructs for structured concurrency are especially super nice.

Scala has had coroutines (async+await) before Kotlin existed, so not sure if "introduced to JVM" is the right word here. But anyway, I agree they are nice in Kotlin.

Thank you for this!

I am wondering whether to include S for socket in the syntax for scheduling.

So if you have M:N:S multiplexing you can share a socket over multiple lightweight threads which are themselves assigned to one kernel thread at a time.

My implementation is a toy compared to Loom. Loom is a serious API adaptation which causes the Thread API to work as lightweight threads, I am not sure if it handles process starvation as my M:N scheduler does.

if you put while (true) {} in a lightweight thread, I think that thread shall either be preempted by Loom if Loom implements that or it shall keep the lightweight thread assigned to that kernel thread, causing starvation of other lightweight threads.

So you might have

thread1:light1:socket1recv thread2:light2:socket1send thread1:light1:socket2recv thread1:light2:socket2send

My understanding is that Java didn't have an M:N thread scheduler before, it had an N:1 scheduler (I believe that all virtual threads were multiplexed onto a single shared operating system thread).

I believe that Loom is the first official M:N thread scheduling implementation for the JVM.

Maybe Sun was simply wrong. Many other languages/runtimes have had great success with M:N threads, including the Erlang runtime.

In Linuxland 20 years ago "Native POSIX Thread Library" an 1:1 implementation won as well (1).

Go's M:N Threading is a great success. I guess M:N threading is harder to implement at the OS Level.

(1) https://en.wikipedia.org/wiki/Native_POSIX_Thread_Library

For me it comes down to simplicity of the sharing model. With async you can basically ignore sharing and write stuff that works against the same data structures because you know they’re mutually exclusive. Once you move stuff into separate threads you need to be a lot more careful, using Arc and Mutex et al, or devise channel semantics. I tend to only use threads when I never want a future interfering with the scheduling of another future, either when one is really long or when low latency is absolutely critical. One example recently was I had very low latency network stuff happening on one thread and it was sending messages to a TUI thread. I absolutely didn’t want the networking code ever blocked by the TUI, even for short times.

I find it fascinating that LWT have come so far that this is a legitimate question. There was a time when async << threads << processes. But a lot of heroic good work went into bringing about this wonderful age.

But tokio is multithreaded, so you need to deal with all that complexity?

I believe its abstractions work the same way regardless of how many threads you do or don't give it

The multithreaded runtime requires that your futures be Send.

Which isn't a big deal in most situations as you can still use mutable variables without mutexes, even across await points.

Tokio has a single-threaded executor as well

That’s a runtime option, and I think it specifically matters when you’re using tasks. If you’re using network and io futures (I’m not sure of this) I don’t think it returns a thread backed future.

> With async you can basically ignore sharing and write stuff that works against the same data structures because you know they’re mutually exclusive.

How does that simplify things in rust? You don't get shared mutability either way, and RefCell is barely simpler than Mutex.

You don't need RefCell. You can just use mut variables as long as you keep everything local inside one task. But one task can actually run many things concurrently, unlike a thread, so this is very powerful.

From my limited experience, passing variables around between threads is quite painful in Rust, async makes sense therefore as it allows you to reduce shared ownership of variables. A while ago I wrote a simple multi-threaded networking library in Rust and things that would be trivial in C/C++, like accepting a network connection and passing the socket handle to a thread with some extra information were really tricky (maybe just because I was inexperienced in Rust) and took some quite ugly code to solve. In the end the program I produced had significant performance bottlenecks and was quite hard to understand.

I rewrote the same thing in Golang afterwards, took me maybe 10 % of the time and the code I produced was very readable and extremely fast. After that I'm a little burnt regarding Rust and have a hard time buying all the hype around it. I guess it has potential for embedded systems and other memory-constrained environments though, as Golang can't compete there right now due to the large size of the runtime, though there are projects like Tinygo that might eventually fill that gap as well. So I'm probably more hyped about Zig than Rust for now when it comes to next-gen languages.

I'm working my way through the Rust book right now, and that's sort of the sense I'm getting too.

The compiler is quite good at catching some issues relating to memory access predominantly. There's many ways to solve multi-threading issues (&, &mut, Rc, Box, Arc, ...).

If you do it the right way the compiler won't get in your way much, but if you do it the wrong way it will be a cat and mouse game where you do your best to confuse the compiler into believing your code is fine-ish.

> (maybe just because I was inexperienced in Rust)

Given the one consistent thing I hear about Rust is it's steep learning curve, I'd almost certainly chalk it up to that. Go let's you ignore details that you can't ignore in Rust, and being forced to think about and organize data in a certain way to eliminate that pain is a skill that no doubt takes practice.

In my experience, many things that are trivial in C/C++ are unfortunately subtly wrong, especially involving threads.

It's a shame that your C or C++ experience doesn't translate to Rust. For me a lot of things just clicked in Rust. Although there certainly was the occasional "why can't I just do this?" that took much longer to reason all the way through. It's maybe something Rust tutorials need to be better at, or maybe there ought to be tutorials specifically for e.g. C++ folks?

For better or worse, Rust gives you the choice (& vs Arc, etc), and it's yours to make. These choices are fatiguing to make if you aren't used to them (I believe C/C++ is similar for newcomers). In the long run, making those choices explicit is where Rust really shines in codebases IMO.

Nothing wrong with using Golang if it works for you. I sometimes wish for a middle ground between Python and Rust, but Golang isn't for me.

‪Main reason for me to use async instead of threads is when I have a very high number of concurrent connections, that don’t do much in terms of CPU, but will require a lot of RAM because of the minimum stack size per thread. If I don’t need that, then threads are fine. ‬

Does Rust have a high minimum stack per thread? I know managed languages like Java do, but for Rust couldn't it be a page or two of actually-mapped memory?

Last I checked, the kernel allocates a bunch of address space per thread but it doesn't consume actual memory unless you touch it and page fault.

It seems pretty likely that, at some point in the lifetime of a thread, it will consume a lot of the stack and thus touch those pages. And there isn't a mechanism for deallocating those pages again, right?

Tying your memory consumption heavily to the max stack depth also seems like it would be difficult to reason about IMO.

for me this is the big point. stack-based memory allocation works great as long as your usage and sharing follows the control flow. but when it doesn't, rust really leaves you in a bad spot and you get to play with:

   - Arcing everything
   - using a fixed-sized ancillary array and passing indices
   - creating threads that correspond to the lifetimes you care about
   - fighting your way through adding explicit lifetime allocations and
   - delaying reclamation of _everything_ above it on the stack
   ...

and while workable, none of those things are particularly nice

IDK, it's pretty easy to just Arc something if it has to be shared across threads. That's what you do in other languages. It's rare that I care to deal with the thread lifetimes themselves.

Rust defaults loosely follow platform’s libc defaults for improved interoperation. On major platforms the default stack size is 2 megabytes of mapped address space. You can ask for less, but it gotta be explicit for every spawned thread.

That’s mapped space, so depending on the OS it could be more or less friendly to its memory subsystem. Linux won’t bother finding you actual resident pages until you do need them. But if the stack drops back down, it will be swapped out as any other dirty memory. Rust does not do any MADV_DONTNEED magic behind your back to aid with reclaiming memory.

As far as I'm aware, the stack size per thread typically doesn't actually matter. It's virtual memory until each page is written to

It matters actually, if your each tasks use 10KiB stack for execution, and you have 1000 tasks, with threads that will consume 12MiB just for the stacks (+2KiB per task due to page alignment). If you use async runtime as Tokio the stack is un-rolled on each suspend point. So if you have 16 worker threads, that means the stacks will consume 192 KiB.

This is true. After I wrote my initial reply I did some measurements. After spawning 4096 threads on my M1 mac, I saw 80MB of reported memory usage. The equivalent tasks in tokio after spawning 100,000 of them reached only 100MB memory. This is roughly 20x less memory.

One common thought I see a lot though is that each thread uses 1MB (or more) in stack space alone. This just isn't true with modern memory paging

Use tokio (async) when you wait a lot, use rayon (threads) when you compute a lot. These are two almost orthogonal concurrency use-cases.

Does rayon offer the ability to pick which CPUs are available for distributing work to?

I want to pick N cores for I/O and the rest to do compute but I didn’t see a way to let me pin the CPUs rayon could use for its compute threads.

Rayon operates on top of the thread pool. You can control its size [0] and limit it to a fraction of your available cores leaving the rest for I/O.

[0] https://docs.rs/rayon/latest/rayon/struct.ThreadPoolBuilder....

Am I wrong for following the general rule of thumb:

"If your tasks spend most of their time waiting, use async. If they spend most of their time running, use threads"

I think async concurrent code is just much easier to reason about, regardless of threading. Also, for intra-request concurrency (e.g. i need to grab data from N different sources to complete this request), it's much easier to arrange for that work to be done concurrently. Also easier to implement highly responsive applications with async vs. threading.

There is an alternative here with coroutines. There is no fundamental reason to take on the function coloring problem.

If one was to benchmark cpp coroutines vs rust async would the performance difference ever pay off?

C++ coroutines[0] are also "function coloring", unless I'm missing something?

[0] https://en.cppreference.com/w/cpp/language/coroutines

Rust async functions are basically coroutines that suspend when you use await. Except that you can't return intermediate results (although once Generators are stabilized, that should resolve that use case).

Personally have found async rust to be both ugly and not very useful. Instead of async IO I just make my own event loop with mio and use non-blocking non-async libraries (like tungstenite for WS and mio_httpc for HTTP).

Sounds like you're almost re-implementing single threaded async

Yeah I suppose I am, but the perf is better (very important in this particular use case) and the code is simpler (imo).

But what makes the perf to be better? Better scheduling / less context switches?

I think using a simple single threaded executor gets you little context switching and such.

The issue is that async generator codegen currently is not as good as we want it to be. Async futures currently take more space than they need to and have some issues with drop tracking. They're sound, but not necessarily efficient.

These are not problems with the language or the feature, just the current implementation

Maybe check out glommio.

Reply on news.ycombinator.com