Comment by morelisp | Hacker News Reader

Comment by morelisp | Hacker News Reader

Parent Full thread

I mean that's a nice theoretical argument, but the empirical results are not so clear.

https://dl.acm.org/doi/10.1145/3297858.3304069

Surprisingly, our study shows that it is as easy to make concurrency bugs with message passing as with shared memory, sometimes even more. For example, around 58% of blocking bugs are caused by message passing.

...

Our study found that message passing does not necessarily make multithreaded programs less error-prone than shared memory. In fact, message passing is the main cause of blocking bugs. To make it worse, when combined with traditional synchronization primitives or with other new language features and libraries, message passing can cause blocking bugs that are very hard to detect.

I'd also add that in the modern world we've "solved" many multi-threading problems by splitting the queues and data stores out into multiple services. Now we don't have any threading problems - only "distributed system" problems, and if there's a bug there that's the architect's fault, not the programmer's!

The results of concurrency bugs with message passing tend to be less bad than the results of concurrency bugs with semaphore/locking code; although this is often because message passing tends to not be shared memory semaphore/locking tends to be shared memory.

Blocking bugs are a service issue, but incorrect locking can lead to deadlock or unprotected writes that corrupt data.

Depending on what they mean by 'detect', I've found it's actually quite easy to detect blocking issues in a message passing environment (Erlang, specifically), if a process has a blocking issue, it will end up stuck waiting for a message that will never arrive; however its mailbox will be growing as new requests are added. Scanning for large mailboxes, as well as growing mailboxes is easily automated and detects the issue. Of course, that's runtime detection; it's hard to detect these with code analysis (because it's not exactly a code problem. It's a system problem, and the system is an emergent property of the running code, it's not explicitly described anywhere, at least in Erlang; this is simultaneously amazingly powerful and limiting)

Interesting - in other words you are saying that message passing concurrency bugs tends to result in liveness violations whereas locking bugs tend to violate safety?

Yeah, that's a nice restatement. Additionally, it's a lot easier to find liveness violations than safety violations. If your system deadlocks, it's obvious (not so much if it livelocks), and you can see which parts are deadlocked. If your system corrupts data, it's not always obvious, and even when it is obvious that there was corruption, it's not always obvious where it happened.

Although that's true, I found in the past that a pure queues/actors based approach didn't work well for some kinds of app. I had to introduce locks again and ended up with a hybrid solution.

The problem was that the app had a mix of low latency requests to access some parts of a large data structure, but the actor had to do a combination of fast and slow requests, so low latency requests would end up sitting in the queues for unacceptably long periods even when they needed access to only a small and stable part of the overall data. A related problem was the code size and complexity increase that came from having to make immutable snapshots of the underlying mutable data structures. Adding locking fixed all that and it didn't lead to any noticeable increase in bugginess. If anything the code was simpler, because the readers could simply access the data when it was in an immutable (locked) state, get what they needed and get out as quickly as possible, with RW locks ensuring good scalability and memory locality.

The actors approach also ran into circular deadlock when increasing the granularity of the actors, but unlike with locks there was no tooling support to find the issues (Guava has cycle-detecting lock classes).

The experience soured me on anything claiming to be a silver bullet for concurrency. It felt like the problems had been transformed into different shapes but not actually decreased in volume. Since then I use whatever approach feels intuitively right at the time.

I mostly agree with you, but it feels a bit handwave-y. That such bugs are less serious is maybe true for a particular class of idioms and a particular class of problems that, say, dominate CRUD application design.

Unsynchronized writes are also detectable at runtime, I would guess (having done no production Erlang work) with around the same order of false negatives as a stuck mailbox. TSan will only find the race if it occurs relatively close temporally; mailbox scanning will only find the stuck message if the send rate is sufficiently high. TSan is a lot more expensive but can be productively used during test suites. Mailbox scanning seems likely to fire false positives and only works in a complete running system.

In the end, only simpler program design can save us.

Yeah so many services APIs are blocking, it's hard to keep threads non-blocking. Often there is a way, but you have to dig for it.

Reply on news.ycombinator.com