Comment by toast0 | Hacker News Reader

Comment by toast0 | Hacker News Reader

Parent Full thread

The results of concurrency bugs with message passing tend to be less bad than the results of concurrency bugs with semaphore/locking code; although this is often because message passing tends to not be shared memory semaphore/locking tends to be shared memory.

Blocking bugs are a service issue, but incorrect locking can lead to deadlock or unprotected writes that corrupt data.

Depending on what they mean by 'detect', I've found it's actually quite easy to detect blocking issues in a message passing environment (Erlang, specifically), if a process has a blocking issue, it will end up stuck waiting for a message that will never arrive; however its mailbox will be growing as new requests are added. Scanning for large mailboxes, as well as growing mailboxes is easily automated and detects the issue. Of course, that's runtime detection; it's hard to detect these with code analysis (because it's not exactly a code problem. It's a system problem, and the system is an emergent property of the running code, it's not explicitly described anywhere, at least in Erlang; this is simultaneously amazingly powerful and limiting)

Although that's true, I found in the past that a pure queues/actors based approach didn't work well for some kinds of app. I had to introduce locks again and ended up with a hybrid solution.

The problem was that the app had a mix of low latency requests to access some parts of a large data structure, but the actor had to do a combination of fast and slow requests, so low latency requests would end up sitting in the queues for unacceptably long periods even when they needed access to only a small and stable part of the overall data. A related problem was the code size and complexity increase that came from having to make immutable snapshots of the underlying mutable data structures. Adding locking fixed all that and it didn't lead to any noticeable increase in bugginess. If anything the code was simpler, because the readers could simply access the data when it was in an immutable (locked) state, get what they needed and get out as quickly as possible, with RW locks ensuring good scalability and memory locality.

The actors approach also ran into circular deadlock when increasing the granularity of the actors, but unlike with locks there was no tooling support to find the issues (Guava has cycle-detecting lock classes).

The experience soured me on anything claiming to be a silver bullet for concurrency. It felt like the problems had been transformed into different shapes but not actually decreased in volume. Since then I use whatever approach feels intuitively right at the time.

I mostly agree with you, but it feels a bit handwave-y. That such bugs are less serious is maybe true for a particular class of idioms and a particular class of problems that, say, dominate CRUD application design.

Unsynchronized writes are also detectable at runtime, I would guess (having done no production Erlang work) with around the same order of false negatives as a stuck mailbox. TSan will only find the race if it occurs relatively close temporally; mailbox scanning will only find the stuck message if the send rate is sufficiently high. TSan is a lot more expensive but can be productively used during test suites. Mailbox scanning seems likely to fire false positives and only works in a complete running system.

In the end, only simpler program design can save us.

Interesting - in other words you are saying that message passing concurrency bugs tends to result in liveness violations whereas locking bugs tend to violate safety?

Yeah, that's a nice restatement. Additionally, it's a lot easier to find liveness violations than safety violations. If your system deadlocks, it's obvious (not so much if it livelocks), and you can see which parts are deadlocked. If your system corrupts data, it's not always obvious, and even when it is obvious that there was corruption, it's not always obvious where it happened.

Reply on news.ycombinator.com