Comment by tptacek | Hacker News Reader

Comment by tptacek | Hacker News Reader

Parent Full thread

Like you†, I've had the pleasure of working with some fairly large concurrent codebases and the character-building experience of tracking down deadlocks, random memory corruption bugs that turn out to be race conditions, and (my most favorite of all) unexpected serializations that randomly bring programs to a halt. Most of that experience has been in C++, with a little C and a little Java mixed in there.

Over & over I see language aficionados ding Golang for not taking advantage of immutability and for allowing shared data --- or, in your case, going a step further and reducing all communication among processes in Golang to instances of synchronized sharing.

What I'd like to know is: why don't all those hundreds of thousands of lines of concurrent Golang code out there, including all the library code I can just "go get" and whose authors have been encouraged by Rob Pike to use, basically, threads with near total abandon (watch his video about designing a lexer!) --- why don't all those libraries and programs randomly deadlock and corrupt themselves all the time?

Because my experience is that Golang code is quite a bit more reliable than, for instance, Python code.

What am I missing? The "share by communicating" model in Golang seems to work pretty darn well, especially given the extent to which Golang begs programmers to make programs concurrent.

† OK probably you more than me but still.

> why don't all those libraries and programs randomly deadlock and corrupt themselves all the time?

The simplest answer would be "they do." In aphyr's recent presentation on Jepsen, where he tested etcd (a Go database implemented on top of Raft), he noted that when he started using it he encountered a ton of easily reproducible races and deadlocks (which he sarcastically noted was surprising because he thought goroutines were supposed to make concurrency issues a thing of the past).

I am not saying that Go channels don't help the situation at all--and the inclusion of a race detector doesn't hurt either--but you still have plenty of ways to shoot yourself in the foot. The thing that probably helps most is that GOMAXPROCS is 1 by default, since data races are a multicore phenomenon in Go.

Distributed systems programming is its own special concurrency problem, and distributed systems also exhibit deadlock, races, and serialization, no matter what language they're implemented in. I'm not sure what finding a race condition in a distributed commit implementation says about a language; at the very least, it's nothing you couldn't say about Rust as well, which is also not a language that solves distributed systems concurrency problems.

Maybe I'm wrong and etcd was riddled with concurrency problems between the goroutines of a single etcd process?

In any case: as anyone who has worked on a large-scale threaded C++ codebase can tell you: Golang programs simply do not exhibit the concurrency failures that conventional threaded programming environments do. It would be one thing if Golang code only used concurrency for, say, network calls. But goroutine calls are littered throughout the standard library, and throughout everyone's library code.

It is not a black and white situation probably. Golang is better because it has built-in channels and encourages users to take advantage of them. It also has garbage collection. So those 2 things right of the bat help.

But there are better things out there -- isolated heaps (Erlang), borrow checkers (Rust), stronger type systems and immutability (Haskell) etc. There are no magic unicorns so those things often come at a price -- sequential code slowdown.

Getting back to go. One can of course say, "Oh, send only messages. We are all adults here. Let's just agree to be nice. Stop sharing mutable memory between goroutines!" But all it takes is "that guy" or "that library", doing it "that one time" and then there are crashes during a customer demo or during some critical mission. It crashes and then good luck trying to reproduce it. Setting watchpoints in gdb (or the equivalent Go tool), asking customers "Can you tell me exactly what you did that day. Think harder!" and so on.

Also, as others have pointed, with Golang though, often it is run with just one OS thread backing all the concurrency. So many potential races could be just be hidden.

There is also some confirmation bias involved. When something is broken, often authors don't write blogs about it, don't advertise. They fix it, and move on. So maybe a lot of programs are full of concurrency bugs but just nobody is blogging about it. They've invested time and energy into learning a new ecosystem and now they have to blog about its flaws and so on. That is hard to do.

Another observation is that when spending a lot of time debugging and handling segfaults, pointer errors, user-after free errors, concurrency issues, that becomes the default and expected view of how programming works. It becomes hard to imagine how it could work another way. It becomes obvious that weeks would be spent tracking one concurrency bug or having to add cron jobs to watch for crashed programs and restart them because the system is so complex and non-deterministic, replicating the bug is too hard.

How does Rust's borrow checker cause a slowdown of sequential code? It's a purely compile-time construct and allows for the elimination of a GC, so it's actually a net win in code execution speed.

> How does Rust's borrow checker cause a slowdown of sequential code?

It doesn't, it was just grouped with the other two. But immutiblity-by-default could lead to slowdown.

Not at all. In languages that are thoroughly immutable, it's copying, not immutability, that has a runtime cost. Rust has mechanisms for avoiding these costs (moves and mutable references).

Furthermore, since the compiler's knowledge of mutability is directly related to its knowledge of ownership, one could argue that immutability actually makes code faster by dint of providing greater aliasing information (e.g. `restrict` in C) to the optimizer (though the Rust compiler has yet to actually leverage this optimization).

Just out of curiosity - is there a problem with sharing data across goroutines when access to/mutation of said data is controlled by a mutex?

    func (*Mutex) Lock

    Lock locks m. If the lock is already in use,
    the calling goroutine blocks until the mutex is
    available.

http://golang.org/pkg/sync/

It seems to me this is a valid alternative to [rigidly] sticking to pure message passing.

In short: We (as a field) tried them for many many (many!) years now and they have been found lacking -- in practice they're just too hard to get right for large-scale systems.

EDIT: The mutexes themselves are easy enough to get right, it's the systems using mutexes that are too hard to get right.

In most languages, the language says nothing about what data is protected by the mutex. Modula and Ada did, and Java has "synchronized" objects, but C/C++/Go lack any syntax for talking about that. This typically becomes a problem as a program is modified over time, and the relationship between mutex and data is forgotten.

> In most languages, the language says nothing about what data is protected by the mutex.

Or the other way around, what mutex protects a piece of data (or even that a piece of data should be protected at all), so it's easy to forget it and just manipulate a bit of data without correctly locking it.

I was pleasantly surprised to discover that Rust's sync::Mutex owns the data it protects, so you can only access the data through the mutex (and the relation thus becomes obvious).

> is there a problem with sharing data across goroutines when access to/mutation of said data is controlled by a mutex?

Well aside from the deadlock or priority inversion of sorts that should work. Just like it would work in C/C++/Java etc.

The real problem is when the shared data is not controlled by a mutex, but should be.

I feel this is a symptom of people coming from dynamic, higher level languages, who may or may not have ever really learned concepts of CS, "switching" to Go for performance reasons. I don't intend to insult anyone or certainly the authors of the above. Go is type safe but doesn't keep you from shooting yourself in the foot. You NEED to read the spec to understand when sharing memory is generally OK. I'm glad not to be penalized in performance or boilerplate to accomplish this. The downside, of course, are stories like the above. I don't blame the language here, though. They give you the tools to be safe. Getting away from automobile analogies, let's try woodwork. I can give you a drill, a drillbit, a screwdriver, and a screw. Sometimes you should know when to make a pilot hole, and when it's ok to forego this. But folks looking for 'performance' or 'ease of use', or folks who come from languages that just give you a nailgun, will inevitably split the wood a few times.

Were there any code examples provided that show how to easily trigger races & deadlocks? I mean, the Go team needs to be aware of these problems and provide a fix or something.

The issues weren't with Go--which definitely allows for both data races and deadlocks and doesn't claim to eliminate either--but with etcd. And according to aphyr, the team was very responsive and quickly fixed the ones he found.

My point wasn't that Go is _worse_ than contemporary languages like C++ and Java when it comes to data races, only that it doesn't eliminate them. Which, again, it doesn't claim to. Rust does, and it is an important difference between the two languages. Because data race freedom with cheap mutable state requires a garbage-collection free subset of your language [1], I think it's unlikely that Go will ever guarantee this.

[1] as noted by Niko Matsakis at http://smallcultfollowing.com/babysteps/blog/2013/06/11/on-t...

> show how to easily trigger races & deadlocks? I mean, the Go team needs to be aware of these problems and provide a fix or something.

You mean file a bug like "issue #1935 -- stop sharing memory between goroutines" (I just made it up to be silly there is no such bug report)

In other words, they have explicitly designed in the ability to share memory between goroutines. You can certainly file an issue or bug report about, somehow I doubt that will lead to much but being kickout and laughed at.

One can also just have 2 goroutines wait on each for results and that's a deadlock. Maybe there is a tool to detect that would be nice. Do you know of one?

If all goroutines deadlock, then the runtime will panic and you'll get stack traces for everything. But yes, you can obviously deadlock one or more goroutines trivially. A simple select{} will just block one goroutine forever, for example. And no, there's no tools currently that will detect deadlocks, AFAIK.

It seems that Go's race detector (https://blog.golang.org/race-detector) can do that. see an example: http://pastie.org/9705392

I haven't looked into that (or googled), but how does one detect that everything has deadlocked - is it some kind of profiling/sampling being done, or is it something more system specific? Any pointers? Thanks!

http://golang.org/pkg/runtime/pprof/#Profile Provides a "blocking profile" that tells you what things blocked and for how long they blocked for. You can use it to find places where you've deadlocked as well as places where adding buffered channels might help performance.

"all goroutines are asleep - deadlock".

https://code.google.com/p/go/source/browse/src/runtime/proc....

How can you compare the ridiculous amount of Python libraries/code out there (with varying degrees of quality as you would expect from a friendlier language) with Go ? Of course you would find Golang code to be more reliable. The libraries you're relying on are by comparison quite primitive and missing many years of technical debt.

Reply on news.ycombinator.com