Hacker News Reader

Comment by danuker | Hacker News Reader

Why implement coroutines when you have multithreading?

Coroutines are to me a brain-twisty way to make the most out of a single core, because your language doesn't support multithreading. For example: JavaScript with a single thread in the browser, Python with the GIL.

C++ has threads which ACTUALLY run in parallel on the CPU. Why bother complicating the language further?

View on HN

Because native multithreading is expensive when it comes to (a) memory consumption per thread of execution and (b) context switching. Also, if your program spends a lot of time just waiting on outside resources (most notably slow/blocking IO), coroutine concurrency is bother faster and more lightweight. If your program is CPU bound, then coroutines/async have little-to-no value over native multithreading.

That being said, yeah, most async workflows are obtuse and difficult to follow and they force all consumers of your interface to also use the async primitives if they want the benefits. Go seems to be one of the few languages that does async/coroutines right.

See: https://eli.thegreenplace.net/2018/go-hits-the-concurrency-n...

What I like about Go is that each co-routine is entirely synchronous, and the only concurrency primitives you're given are channels and mutexes. This makes your code so simple and easy to understand.

I just looked up Kotlin's co-routines, and there are a number of annoying hoops you need to jump through:

1. Co-routines can only be built by calling runBlocking or coroutineScope.

2. Functions that are async buy should act like blocking calls in a co-routine need to be marked with `suspend`

3. There are a bunch of primitives you need to use like `launch` or `async`.

4. There are built-in options to make the co-routines lazy, but it makes the API feel sloppy.

5. Rather than locks, Kotlin encourages explicit thread dispatching. Some times you need to do this in Go too, like when using a C library, but in general it's rare.

The other consequence, however, is that Go coroutines only work with other Go code, and bespoke Go threads and stacks require the runtime to jump through a lot of hoops to do FFI correctly, with the result that the ecosystem tends to rewrite rather than reuse existing code in C and other languages.

In contrast, the approach with explicit promises and await can be mapped all the way down to a C struct with a function and a data pointer representing a callback. That is, it is C ABI compatible, and thus it can be easily composed across any boundary that respects that ABI.

The only runtime hoop I've ever had to jump through was to invoke runtime.LockOSThread() inside a go-routine. In a lot of ways:

    go func() {
        runtime.LockOSThread()
        // ...
    }()

Is simpler than creating a thread in other languages.

The runtime jumps through those hoops for you as needed, mostly, but there are consequences to that wrt performance.

`launch` is just the `go` keyword. `async` stores the result in a `Deferred` (or `Promise`), which is an abstraction for parallel decomposition.

Other than confining UI work to the main thread, I don't think I have seen thread confinement as an alternative to locks.

That's kind of my point. Go has the `go` keyword. Kotlin has runBlocking, coroutineScope, launch, async, and suspend. Go's simplicity feels nicer to me.

`runBlocking` and `suspend` is the coloured function point. Some hate it, I don't mind it. It has its benefits.

---

A quick search on SO gives this example of parallel decomposition. https://stackoverflow.com/questions/57662739/pattern-for-fet...

Starting several `async` tasks, then awaiting them is cleaner for me. But that's subjective.

---

Kotlin's structured concurrency handles the context stuff by default. When you handle those concerns (cancellation for example) in go, it's just as complex, but more verbose.

Go's context object is used for cancellation. I think this is pretty simple and has the benefit of avoiding any type of exception handling while still giving you the ability to clean up anything you were doing.

In general, if you like Kotlin, I can see why you'd like their approach to concurrency. Kotlin really likes a large standard library with generic blocks that act as syntactic sugar. I used to like that too, but as I've gotten older I've gotten lazier. Kotlin now how too much mental overhead for my old brain.

The coroutines support in the stdlib is here: https://kotlinlang.org/api/latest/jvm/stdlib/kotlin.coroutin...

You can build exactly Go's coroutines API on top of these primitives, by encapsulating the Continuation object in Channel and Mutex types, and you would not have to touch the `suspend` keyword (which is sugar for CPS-transforming the function).

I don’t think you can build Go’s coroutine API in Kotlin because Go ensures you can never starve the scheduler. For instance, blocking system calls in Go spawn new OS threads so that they can be pre-empted by other running go routines, and most of the standard API is built with stuff like select, poll, and epoll so you never have to worry about blocking the scheduler. That’s not how Kotlin works, right?

So, while you can have a similar syntax in Kotlin, the overall systems will have different semantics.

Project Loom will get us that behavior for the JVM, as long as the default dispatcher is modified to support virtual threads. For now, we schedule on an unlimited thread pool using Dispatchers.IO, which is not as cool.

The cost of context switching between threads sharing the same address space is much lower than between tasks, because there's no need for a TLB flush. It just comes down to the cost of saving/restoring register state (more or less), and that's a nice known fixed cost on any given processor.

Context switching between tasks has a variable cost, because the implications of the TLB flush depend on the working set size of the switched-to task. If it doesn't touch much memory, the variable cost is low; if it touches a lot, the variable cost can be much, much higher than the register save/restore (fixed cost).

> It just comes down to the cost of saving/restoring register state (more or less), and that's a nice known fixed cost on any given processor.

There's also the cost of cache eviction. Not something you have to manually manage, but it's a cost you pay nonetheless. Maybe.

Indeed, I ought to have mentioned the cost of cache misses post-context switch too. It too is working set size dependent, but the relationship looks different from the cost of TLB misses. However, my understanding is that an inter-task thread switch would also not cause cache invalidation (there's no need; the address space remains the same).

Would you need to flush a TLB on a regular context switch? That seems inefficient as threads are being scheduled per quanta in Linux. TLB entries are only evicted on page faults right?

> C++ has threads which ACTUALLY run in parallel on the CPU. Why bother complicating the language further?

Actual parallelism is surprisingly slow and resource heavy in practice. When everything is on one core, you keep things local in L3, L2, L1.

However, add on a 2nd core, then you suddenly have "ping ponging". Lets say core#0 has data in L1 cache, and now Core#1 needs to read-modify-write to it. That means the cores need to:

1: Core#1 begins to mark that cache-line as exclusive.

2. Core#0 needs to then respond by marking the line as invalid. Then Core#0 ejects the data from L1, and passes it to Core#1.

3. Core#1 can finally begin to work on the data.

4. If Core#0 uses the data again, Core#1 must do step#2 again.

--------

This is called "Ping-ponging". L1 cache read/writes is 1-nanosecond, but ping-pongs can be 30nanoseconds or slower, 30x slower for no reason.

You don't want to add another core to your problem unless you're _actually_ getting a speed benefit. Its more complex than it may seem at first glance. You very well could add a bunch of cores and then suddenly your program is way slower because of these kinds of issues.

Another problem: False sharing. Core#0 is working on "int x", and Core#1 is working on "int y", but x and y are on the same cacheline. So they have to ping-pong even though they never actually touch the same data.

Your example implies some combination of (a) excessive data sharing between threads (b) possible need for core pinning.

If thread A is going to need to read all the data touched by thread B, then it's unclear why you've split the task across threads. If there are still good reasons, then probably pin A and B to core N (never pin anything to core 0, unrelated story), and let them interleave there.

If that doesn't make sense, then yep, you'll have to face the cost of ping-ponging.

In my benchmark of a simple ledger I generate and do 25,021,365 (25 million) random transactions per second. (Withdraw and deposit over 80,000 accounts)

Paralellise it via sharding and I get 80,958,379 transctions per second.

If I remove the randomness, I get 134,600,233 transactions per second. If I parallelise with 12 threads I get 931,024,042 transactions per second.

My point being, if you paralellise properly and do not use shared data, then you can boost performance by a multiple of the number of threads.

https://github.com/samsquire/multiversion-concurrency-contro...

For the technique I use for this scalability, see my journal entries on sharded structs and sharded integers

https://github.com/samsquire/ideas4#565-sharded-struct-pseud... https://github.com/samsquire/ideas4#608-sharded-integers

This is not so likely in most workloads unless multiple threads are hitting very close or adjacent regions in memory. Most multithreading workloads work on memory not so close together. It's good to be aware of it, but to suggest it is the most likely outcome from multithreading is misleading.

Coroutines don't really have anything to do with threads. They are just a way for you to model suspending and resuming functions.

If you have every tried to implement a complicated iterator you will find that you model a for loop in your head but that the iteration of the loop is controlled from outside using the ++ operator. The ++ operator allows the iterator loop to go one more time round. Normally you have to manage this with state stored in some object. With coroutines you can actually write a loop and co_yield from the middle of the loop to implement the ++ operator.

Co-routines can be a great structured-programming tool for implementing state machines. A good article which explains this idea explicitly: https://eli.thegreenplace.net/2009/08/29/co-routines-as-an-a...

That said, overhead always needs to be considered; the overhead cost of using a CPP co-routine would be considerable for implementing a state machine that transitions rapidly.

This is actually really convenient.

In our code we have an abstraction that wraps a set using a variant. Instead of implementing an iterator/begin/end based on which data the variant holds, we have an iter() method that unwraps the variant and loops through and co_yields. Since you can consume a generator in a for-each loop, you use it the exact same way (except calling iter).

In our case it doesn't save that much complexity, but it's definitely some. I could imagine more complicated cases where it is even more convenient.

Everyone who ever programmed a bit in Python knows that. Generators simplify your code and remove unnecessary boilerplate.

However, async/await is real nice as well.

Real threads can too expensive sometimes. A single coroutine in Go just use few kb of memory. So we can easily spawn hundred of thousands of coroutines but not possible using threads

Your point is correct but your magnitudes are off.

IIRC, a thread takes up ~32kB or so worth of data in pthreads/Linux, or at least... somewhere around there. So on a rather cheap server with 64GB of RAM, you can easily fit 1-million threads. More expensive servers have 512GB, 1TB or more RAM and can easily handle this kind of load.

Of course, coroutines are even smaller/more efficient and can go into 10-million, 100-million or higher.

A bigger issue is creating and destroying threads, which can be costly if you don’t do a lot of work per each. Of course, people have worked around this problem for ages with thread pools, but then you’re already breaking up ownership, code flow and screwing up the stack for debugging and panicking. So something like coroutines or green threads is still motivated, it is just extremely different across languages and is almost always a leaky abstraction.

A thread on Windows is MUCH heavier than Linux. 1 MB stack from what I remember plus takes a lot longer to create.

dwStackSize can be set as low as 64kB though. So if you're in the habit of making millions-of-threads, maybe make them smaller on Windows?

It does default to 1MB, but these sorts of things are configurable.

Ugh I really need to finish my blog post where we had async with only a few dozen bytes of overhead per task.

Cooperative async runtimes are obscenely efficient.

> Why implement coroutines when you have multithreading?

Because coroutines are easier to think about than threads, and avoid entire (large) classes of bugs.

Because coroutines (typically) have a lower overhead than threads.

Because if you are IO bound and not CPU bound, and therefore you don't need to spread work across cores, threads may not provide a given workload with much, or any, benefit.

I've benefited from cooperative userspace context switching with CPU bound workloads as well. Imaging simulating a vehicle's worth of embedded software, with several dozen tasks originally intended to run at 100Hz asynchronously across multiple different computers. Now, imagine running them faster than real time, deterministically. Now imagine doing several thousand of those simulations. The overhead of OS threads, context switching and synchronization starts to add up, and is 10-100x slower for that sort of workload.

Cooperative multitasking systems having more deterministic performance profiles is often overlooked.

I'm working on a post right now covering this in detail, but I spent the last 2 hours being reminded that explaining concurrency stuff in simple terms is hard. :-D

It'd be nice if we had system-level "lightweight threads." The explanation I've heard before is that "user space knows more about the program." So people write clever runtimes that do kernel-like task scheduling multiplexed over CPUs, in user space.

I can buy that. Think about a system thread. It's pretty much its own process that shares an address space with others. Its interface is syscalls and memory. A user thread (coroutine, goroutine, greenlet, whatever) is... whatever you want it to be. It can be small and special-purpose.

I agree, but I think as you’re alluding to, the deeper issue is that the syscall boundary is too big of a fence to jump over for small operations. Hence, io_uring and similar, which pools multiple logical syscalls into one. The alternative way would be to have non-syscalls but kernel-aware libraries that share memory with the kernel, I guess? It’s a much bigger change, philosophically, so I think the io_uring model + user space scheduling will take the throne for the next decade or so at least. It’s a bit unfortunate because we’ll likely see a lot of bloated and fragmented language runtimes, but it’s still vastly superior to what we’ve seen over the last decade+.

Coroutines are about decoupling OS-level threads from logical threads (where OS-level threads are already decoupled from physical threads). The reasoning is actually very similar to why OS-level threads do not correspond directly to physical threads.

You may wonder why OS threads are not themselves sufficient, but it turns out there is a decent amount of overhead involved in managing huge numbers of OS threads. There are limits at which Linux does not perform well anymore, leading to problems like live locking. Also, if you can avoid context switching/cache invalidation by keeping a single OS-level thread busy, you get performance benefits even before you hit OS limits.

So the userspace scheduling and green threading that coroutines enable allow you to increase concurrency without hitting these limits.

Why use a hammer when you already have screwdrivers?

For an I/O heavy application (not in C++) I avoid threads for the main "loop" as there is a central cache that is accessed often. Synchronizing access to it is likely more expensive and error-prone than using just a single thread. It is also possible a few hundred or thousand tasks are running concurrently, which becomes very expensive very quickly with threads.

There are a few CPU-intensive tasks but those are easy to isolate and send to a threadpool.

Coroutines are for when you would normally use callbacks or a state machine, but handling the callback’s state or the state machine is complex enough that it becomes much simpler when expressed as a coroutine (simpler to reason about). It isn’t really an alternative to multithreading.

Co-routines (when used as an async primitive) are an an alternative to callbacks.

Callbacks exist to avoid the overhead of marshalling async results back to the original calling thread: instead of having the calling thread poll until the child thread(s) pass their results back, the calling function returns after kicking off the children, and one of the children directly resumes the work directly when the results are present. This is a significant performance gain.

However, callbacks have famously poor ergonomics, so many structured alternatives have been created: Promises/Futures, Async/Await, and CPP Coroutines.

Coroutines are when you don't want to run in parallel, because sequence matters. A typical example is iterators, you want the loop and iterator (routine and coroutine) to run in lockstep.

You could use threads instead, but then you risk getting in all sorts of concurrency problems. And if you synchronize everything, then great, you have reinvented coroutines. In fact, you can think of coroutines as threads with built-in synchronization, which may be how they are implemented.

To more elegantly write code that is async in nature like IO/Web requests? Let stuff happen and just await where needed?

Seems like different type of concern than what you need multiprocessing for.

Good for embedded devices (e.g. a single core ESP32).

even without any threads or async, coroutines are very nice for writing generators and super useful when you want generic code which can iterate, without leaking the container implementation to client code

Coroutines This is another can be used It as another way is still possible to language construct like manage object lifetime, and make some to use threads control flow much easier if you have coroutines to write. lambdas.

Reply on news.ycombinator.com