Hacker News Reader

> We all have multicore machines and it would be great if we could use more than 1/8 or 1/12 (or whatever high thread count of your beefy servers) of our hardware.

It is probably important here to realize that async solves concurrency, not parallelism. You can use async with a single threaded runtime for I/O concurrency and mix that with threads for computational parallelism for long running jobs.

That said, there may be some benefit a multi-threaded runtime would have for the typical I/O bound app (after working around lifetime limitations by adding Send/Sync to data structures). This is because I/O bound programs and those requiring computation are not mutually exclusive and there is always some amount of computation going on, so there may still be some benefit. I doubt a synthetic benchmark would answer this as those typically don't measure any actual work performed, but just "requests/sec".

> It is probably important here to realize that async solves concurrency, not parallelism. You can use async with a single threaded runtime for I/O concurrency and mix that with threads for computational parallelism for long running jobs.

In my experience, it's impossible to mix threads and async tasks. They can't communicate or share state. Threads need locks, while async tasks require an awake mechanism. If you just stick to unbounded channels that don't block on send, you can get far, but in 99% cases you will need to decide upfront on a specific approach.

This has not been my experience at all. Delegating compute-intensive tasks to rayon inside a tokio runtime is not particularly hard (assuming you can pipeline things to separate IO and compute effectively).

A pattern that has worked quite well for me is to use

``` struct ComputeTask { some_state: SomeState, completion: Sender<Output>, }

impl ComputeTask { fn run(self) { .. do you compute-intensive stuff

     self.sender.send(output);
  }

}

async fn do_stuff() -> Result<Output> {

let (tx,rx) = tokio::sync::oneshot::channel(); let task = ComputeTask { .., tx }

  rayon::spawn(move || task.run());

  rx.await

}

```

Shared mutated memory isn't necessarily a problem, because it still is unknown how often access is required to that memory.

E.g. having a thread that spends perhaps 1% of the time with state mutation vs 10 threads spending each 3% of the time with state mutation. You have smaller efficiency per thread, but still higher efficiency overal

This reminds me of the whitepaper Scalability! But at what cost?

http://www.frankmcsherry.org/assets/COST.pdf

Which I think is about how single threaded programs are faster than scalable but slow multithreaded systems.

I think you might be right and that's why there is ReadWriteLock or RwLock for single writer taking turns.

If you have a counter or a data structure you're mutating in every request then you'll hit lock contention.

If you're just using Arc<T> without any other parallelism primitives, then it's immutable and the cores can all read without blocking. The only thing it does is reference counting to know when to Drop.

Blindly using Arc<Mutex<T>> without considering access patterns is a software architecture problem, not a problem with Arc or Mutex.

Send + Sync is maybe not enough but wouldn't it be possible to say: as long as the memory location is read-only you can parallelize access to it. Send + Sync helps pass the read-only data through without synchronization to all threads, while the rest of the Send and exclusive mutability system flags the tricky points for you.

I can see that Send/Sync by itself does not tell you if the data is read only or just internally synchronizing mutation.

I still don't understand why async is faster. Sharding data can be as simple as a buffer per thread in a thread pool to catch incoming data. With async, don't you havev to allocate that input buffer each time? That seems hideously expensive.

I enjoyed this article from Cal Paterson my excolleague. It's about Python async not being faster:

https://calpaterson.com/async-python-is-not-faster.html

I think the idea is that while your blocking waiting for IO in one task you can serve a different task, potentially from a different user. Coroutines, green threads, communicating sequential processes as in Go or Occam.

Pure Python is a very slow language compared to Rust, with significant differences in orders of magnitudes of expenses. I would not expect information about Python performance to be particularly relevant to Rust without further evidence directly from Rust.

I think in retrospect, it makes sense to me that if you are io bound vs cpu bound (like my stuff usually is) that async could let you wait on more things at a time.

I think the whole "IO bound" thing has taken on a life of its own and attained a legendary status that is not always an accurate reflection of reality. People often seem to model things as if "waiting on the DB" is all their system does and the code they wrote executes in exactly 0 nanoseconds, but that's not how it works. It isn't actually that hard to talk to a relatively local database with some well-optimized query and be doing CPU work either comparable to the wait you spent on the DB, or even greatly exceeding it, at which point your language's performance in fact does matter, potentially even dominates.

async isn't typically 'faster' at all. It just lets do do more with fewer threads, and that has its own benefits.

async can actually _increase_ the number of syscalls your application performs.

Which performance metric are you looking at for "faster"? Async is cooperative multitasking applied at a different level of abstraction. Much like OS level multitasking it adds overhead, and reduces performance in terms of latency. On the other hand it improves throughput by allowing better resource use.

>don't you havev to allocate that input buffer each time?

Have to? No, you could pre-allocate and reuse buffers. It is less straight forward than the buffer per thread strategy, but possible.