Clojure Concurrency Tutorial | Hacker News Reader

Clojure Concurrency Tutorial

purelyfunctional.tv

222 pointsmaastaar41 comments

I get to do Clojure about ~60% of the time at work now, and it never ceases to amaze me how much easier it is to deal with concurrency than in vanilla Java.

Futures and Promises and core.async don't allow you to totally avoid planning out your project (I've been bitten slightly by core.async's `go` blocks behaving differently than Go's goroutines), but they are a godsend compared to dealing with manual mutexes and semaphores.

I still need to play with manifold; I hear that it helps deal with the IO-heavy stuff that core.async chokes on.

What's the difference between core.async go blocks and goroutine please, I thought they would be the same to a reasonable extent?

To a reasonable extent, yes, they're the same. The biggest issue is that goroutines are (more or less) fully preemptive (for all intents and purpose) with their thread pooling and go blocks in core.async are not. Basically, in Go, you can have as many blocking IO thing taking millions of seconds to run, but it won't block the thread pool, and your other goroutines will run fine.

Since core.async go blocks aren't fully preemptive, long-running things can eat up the whole thread pool. core.async allocates "number of cores + 2" number of "real" threads to do its work.

For example, say I have a program that downloads hundreds of giant files running on an 8-core server. The naive implementation would have a goroutine/go block for each concurrent download. In Go, this would work more or less ok (assuming you have plenty of memory), but in Clojure it would block all the other go blocks after 10 until the download is done.

Normally, it's not an issue, but it's a small gotcha if you're not paying enough attention.

I'm a bit surprised if I understand you correctly. I though a goroutine would run and hold the thread until any kind of sleep.

edit: Oh but there might be no "core.async sleep" in most of clojure libraries like IO/net stuff, which would explain why async go blocks hold the thread more often than they should, whereas Go has "goroutines sleeps" everywhere.

As I recall it, goroutines are in fact cooperative with implicit yield points. The implicit behavior is what I believe go means by “for all intents and purposes”

Particularly, I believe all function calls are implicitly yield, which means you can usually pretend that its preemptive through normal coding practices; but you can still never yield through a while(1){} (i think there was a proposal to add implicit yield on loops? Not sure if it went through)

I would assume core.async does not have many, if any, implicit yields, leading to the different behavior. But technically, both are cooperative, go is just more convenient about it

The real magic is where you can call blocking functions from goroutines and have it Just Work!

The implicit yield points you're talking about is sleep or wait (noop in other terms). It's also not exactly a yield, it calls the scheduler to tell it to schedule (meaning the scheduler could decide to run again the exact same goroutine that noop-ed). Finally, goroutines are not just cooperative, they also run in parallel in a pool of threads.

>The implicit yield points you're talking about is sleep or wait (noop in other terms)

Isn’t sleep() usually defined as not giving up control? It’s semantically a busyspin for n seconds. That is, it’s explicitly not a yield point; it's a blocking call, and it doesn’t give control back to scheduler.

And isn't wait() usually defined as, semantically, a non-blocking spinloop with a condition (hence notify() or poll())?

Which makes sleep(), wait(), yield() and noops very different.

I suppose in go, if yield() is implicit on any function call, then sleep() will immediately call yield() before actually sleeping (but once started, it’ll hold control until finished)

I'm pretty sure the implicit yield points are actually yield(), though I haven't verified; it's the only one of the sleep/wait/yield semantics that would be sensible to be implicit as far as I can tell.

>It's also not exactly a yield, it calls the scheduler to tell it to schedule

As far as I was aware, thats exactly what yield means; it yields execution control back to the scheduler. I suppose in eg, a generator, it yields control back to the caller, but afaik thats fundamentally the same meaning (the caller is the scheduler). Can you expand?

>Finally, goroutines are not just cooperative, they also run in parallel in a pool of threads.

Sure, but is that relevant? I assume async.core also trivially supports thread pooling, and is presumably equivalent to go’s pool. The only real difference is go implicitly instiates the pool, while core.async presumably makes it an explicit function to call

In Go, time.Sleep don't hold the thread. That's the cool part. In Go a noop gives the control back to the scheduler which will then reuse the thread for any goroutine that needs cpu -- ie. one that isnt noop-ing. So if you sleep or wait for 10 seconds, 10 seconds of cpu will be given to other goroutines.

What's the advantage of downloading multiple massive files in parallel, assuming you can saturate the link with just one? Isn't it better to finish one file before moving on to the next?

If you can't saturate the link with one file, then you should consider something like bit torrent.

It's an example. Think of something else, eg an HTTP client call that sits waiting for a response, instead. The call parallelism is the issue being discussed.

I'm just saying that as a dumb example of something that isn't terribly CPU intensive, but would block up the threads, but even then, let's assume that the downloads are really slow, so that it's not anywhere near saturating my network speed. The go-blocks in core.async would still block the other go blocks. This wouldn't be an issue with Go.

I/O is exactly the kind of stuff that should happen smoothly in goroutines or go blocks. It's kind of the quintessential case because I/O has a lot of sleeps and waits (noop or stalls in other terms).

When you're downloading a big file for example, I'm not an expert of that but I expect that there's gonna be a lot of cpu stalls in there. Same for the network adaptor and disk.

This is true, but I think it's worth mentioning that core.async lets you freely mix "full threads" with work scheduled from a thread pool. You use the same channels for both. You do the lightweight stuff in go blocks, and the I/O heavy work in a `(thread)`, where you use `<!!` and `>!!` for blocking gets and puts, respectively.

I find this to be amazingly flexible. You are right in saying that it's a gotcha: you do not want to do I/O heavy work in go blocks in general.

It is also worth noting that core.async works just as well in ClojureScript, which when you thing of it is pretty amazing. No threads there, of course.

core.async is a macro. Call an already compiled function under a library that blocks and the whole system will come to a screeching halt. Goroutines don't have that limitation.

Please see my other comment about using the same channels and calling I/O-heavy (blocking) functions from a `(thread)` instead of from a go block. You indeed shouldn't call blocking functions from go blocks, but core.async gives you the option of freely mixing separate threads and go blocks.

Having to use full threads in some cases and core.async in others pretty much defeats the purpose. Core.async is just nowhere near as good as goroutines. It'll continue to be like that until Project Loom finally has a release.

Sorry, what is Project Loom? I'm not overly familiar with it and a quick search for `clojure project loom` seems to indicate that it's a graphing library.

https://wiki.openjdk.java.net/display/loom/Main

I think you misunderstood what I wrote.

Core.async lets you freely mix both. You use the same channels, the only difference being that your coroutine blocks will be wrapped in `go` and use `<!` (parks coroutine if it needs to block), while your full threads with be wrapped in `thread` and use `<!!` (blocks). It makes the tradeoffs explicit (the tradeoff always exists) and isn't a problem at all.

I was reading somewhere that the core.async lib has not been updated in awhile...is this correct? (generally curious if the lib is actively updated)

It is maintained, but not actively. Like the last year there was two issues fixed in it.

There isn't someone actively trying to enhance it or add more features to it.

So my understanding is only major issues gets addressed.

Which is a quite peculiar and reccurrent pattern in closure.

After a while libraries and tools stabilize and the community just maintains it.

Most Clojure libraries I've worked with, even the "larger" ones, are fairly small and concise by modern standards. I wonder if this has something to do with the lack of activity (in addition to being a small community). It feels a bit like the Unix philosophy - lots of small/medium libraries handling discrete problems, rather than many competing mega-frameworks. To me, this feels like a silver lining that comes from the limitations of a smaller dev community.

This is why I love functional programming as a whole. I feel like my code is really terse, without little/no sacrifices in expressivity, and usually no substantial sacrifice in performance.

To me, that sounds like a Good Thing.

Yes. You should consider core Clojure (including everything covered in this article) as a proof of concept that will never be revisited.

Wow. That will come as quite a surprise to the dozens of Fortune 50 companies out there using it as part of their core systems, let alone all the smaller companies and startups.

Source: Worked as a consultant almost exclusively in Clojure for the past 10 years.

I, too, have used Clojure as my primary language across three startups and 10 years, and despite my great affection for the language and community, I have told no lies here.

Clojure is far from perfect. But a lot of people are using it and will continue to use it effectively.

It sounds like you're just salty for some reason. I'm sorry you had a bad experience, but that has no bearing on the usefulness of the language.

I don't want to speak for the original author here, but I believe he's basically saying that Clojure has a bit of a tradition of "change no behavior!". Rich Hickey has said as much in one of his talks [1], and while I think Rich Hickey is objectively smarter than me, I don't actually agree with him on this point.

The difference between Clojure and nearly any other language, however, is that you don't need the core library; you can add any language feature you want due to the awesome macro system in Lisps. Even if they deleted the core library tomorrow, we could still build in literally any language construct we want as a library.

This is why Lisp is made of magic :)

But any language with any sort of traction has a bit of a tradition of changing no behavior, so Clojure isn't special here. So I'm not sure what exactly you disagree with here. It makes me think I don't quite understand what you actually intend to communicate.

Back up, keep in mind that I'm not the same person as before, I don't disagree with you.

I think Clojure's built-in library is great, and more than production-ready. Not entirely sure what the parent was talking about.

No, this wasn't my point. I don't lament them not making breaking changes, obviously. My point was that most of the work the core team does is on projects that they will never revisit in the future. The one exception to this is spec, which drags on. Once it's out of alpha I expect them never to touch it again.

If you believe everything in core is perfect and fully formed, and that all other needs are served by libraries that are actively maintained, that's great. Not my experience by a long shot, but I'm happy to be grumpy in a world full of happy Clojure programmers.

Nothing I said has anything to do with the usefulness of the language - I work with it every day and have for a decade, and I hope that I'm not completely crazy. I was just stating that anything that appears in core basically gets to 1.0ish and is never touched again. This seems demonstrably true to me, and not even necessarily a bad thing, but perhaps others disagree.

I read that message to mean clojure/core.[library] and not clojure.core. It's a criticism of the slower development pace and low visibility that libraries like match, logic, combinatorics, test.check, data.zip, rrb-vector, cache, etc. have.

The author can clarify but if that's the critique its not without at least some merit.

Yes, this is part of it (although I do include things like core.async, reducers/transducers etc in that). A lot of Clojure feels like a bunch of thesis projects tacked together and abandoned. Most of it was quite innovative and cool at the time, but it's not being iterated or elaborated on. That's nobody's problem, really, I just happen to find it frustrating.

I'd almost be encouraged by the rework going into spec if it wasn't happening at such a glacial pace.

A common pedantic comment I hear many times from clojure devs its that (the few)libraries are complete, just work and don't need any updates.

Far from the truth IME.

Having said that, Clojure is my favorite programming language for writing "business" applications.

I still do quite a bit of Java work. Nowadays, most high-concurrency Java apps use async tools like Vert.x [1], not explicit mutexes & semaphores. In that respect, modern Java is not that dissimilar from core.async, etc.

[1] - https://vertx.io/

That's a fair point; I admittedly haven't touched Java concurrency in quite awhile, and I actually haven't touched Vert.x since they had direct Clojure bindings :).

Data flow programming is a natural Concurrency programming. In addition, Clojure's STM mechanism comes from RMDB's MVCC. It is the easiest, most convenient, and most appropriate to support data flow programming. https://github.com/linpengcheng/PurefunctionPipelineDataflow

Reply on news.ycombinator.com