What's the difference between core.async go blocks and goroutine please, I thought they would be the same to a reasonable extent?
Since core.async go blocks aren't fully preemptive, long-running things can eat up the whole thread pool. core.async allocates "number of cores + 2" number of "real" threads to do its work.
For example, say I have a program that downloads hundreds of giant files running on an 8-core server. The naive implementation would have a goroutine/go block for each concurrent download. In Go, this would work more or less ok (assuming you have plenty of memory), but in Clojure it would block all the other go blocks after 10 until the download is done.
Normally, it's not an issue, but it's a small gotcha if you're not paying enough attention.
edit: Oh but there might be no "core.async sleep" in most of clojure libraries like IO/net stuff, which would explain why async go blocks hold the thread more often than they should, whereas Go has "goroutines sleeps" everywhere.
Particularly, I believe all function calls are implicitly yield, which means you can usually pretend that its preemptive through normal coding practices; but you can still never yield through a while(1){} (i think there was a proposal to add implicit yield on loops? Not sure if it went through)
I would assume core.async does not have many, if any, implicit yields, leading to the different behavior. But technically, both are cooperative, go is just more convenient about it
Isn’t sleep() usually defined as not giving up control? It’s semantically a busyspin for n seconds. That is, it’s explicitly not a yield point; it's a blocking call, and it doesn’t give control back to scheduler.
And isn't wait() usually defined as, semantically, a non-blocking spinloop with a condition (hence notify() or poll())?
Which makes sleep(), wait(), yield() and noops very different.
I suppose in go, if yield() is implicit on any function call, then sleep() will immediately call yield() before actually sleeping (but once started, it’ll hold control until finished)
I'm pretty sure the implicit yield points are actually yield(), though I haven't verified; it's the only one of the sleep/wait/yield semantics that would be sensible to be implicit as far as I can tell.
>It's also not exactly a yield, it calls the scheduler to tell it to schedule
As far as I was aware, thats exactly what yield means; it yields execution control back to the scheduler. I suppose in eg, a generator, it yields control back to the caller, but afaik thats fundamentally the same meaning (the caller is the scheduler). Can you expand?
>Finally, goroutines are not just cooperative, they also run in parallel in a pool of threads.
Sure, but is that relevant? I assume async.core also trivially supports thread pooling, and is presumably equivalent to go’s pool. The only real difference is go implicitly instiates the pool, while core.async presumably makes it an explicit function to call
If you can't saturate the link with one file, then you should consider something like bit torrent.
When you're downloading a big file for example, I'm not an expert of that but I expect that there's gonna be a lot of cpu stalls in there. Same for the network adaptor and disk.
I find this to be amazingly flexible. You are right in saying that it's a gotcha: you do not want to do I/O heavy work in go blocks in general.
It is also worth noting that core.async works just as well in ClojureScript, which when you thing of it is pretty amazing. No threads there, of course.
Core.async lets you freely mix both. You use the same channels, the only difference being that your coroutine blocks will be wrapped in `go` and use `<!` (parks coroutine if it needs to block), while your full threads with be wrapped in `thread` and use `<!!` (blocks). It makes the tradeoffs explicit (the tradeoff always exists) and isn't a problem at all.