Hacker News Reader

Comment by int_19h | Hacker News Reader

Green threads are great, right up until the moment you have to interoperate with something written in another language / using another VM. Then it's a mess (see also: Go).

The nice thing about async/await is that, because it's all just a bunch of syntactic sugar over callbacks, any language that supports some kind of callbacks with state, can be mapped to async/await - even C! On WinRT, for example, such interop works across C++, C#, and JavaScript.

This is largely why languages that push this model tend to form rather tightly closed ecosystems, IMO. Which, in turn, leads to their lower adoption - and thus, no particular implementation of green threads can become a de facto standard.

View on HN

I think I might be missing what you mean? You can write an async/await implementation from scratch in a few dozen lines of elixir if wanted, but you also have Tasks that provide that already written for you (with support for being included into supervision trees, etc)?

How does JS interoperate with something outside V8?

A JS library that wants to interop with something outside of its runtime (which is not necessary V8 - it isn't in WinRT, for example) can do so through FFI facilities. So long as said FFI supports all the same things that C does, it supports callbacks via function pointers. And if you can pass a function pointer + data pointer to some API, you have a stateful callback - i.e. a promise/future/task. And you can map any such abstraction to the same in another language.

But the moment you introduce green threads, the caller and the callee both have to be aware of that particular implementation of green threads. So even if you can do C FFI, and whatever you're trying to call can also do C FFI, you can't interop async calls across that boundary without some kind of callback arrangement.

Here I'm a bit out of my depth.

I think with Erlang you don't need to know >that particular implementation of green threads, but you need to implement the required specifics of the VM FFI (which in practical terms might be what you meant) - but this has a reason, in that, for the VM to provide its guarantees it needs to be able to count "cycles", refs, etc in order to preempt the execution of any function/process at given points, so that no single process can block the scheduler. It also needs to transform things from and into data types usable inside the VM.

I think you can also use "dirty" nifs, but here you might crash the VM so you really need to know what you're doing (and you still have to translate things to and from the vm).

And there's also ports, and C Nodes. These have a higher sync cost but can be treated by the VM as if they were processes/erlang nodes.

Lastly, you can also just shell from the VM to an OS level process, or use sockets, etc.

But I doubt this is anything 99% of the people writing JS do in JS? One of the things people using Elixir or Erlang is exactly because of the guarantees and programming model of the VM.

Again, might be missing what you mention because it's not an area I have explored in any meaningful way.

That's the thing - it works for Erlang/Elixir, because it tends to live in its own ecosystem with its own libraries etc. If you are working on something where there's an existing large ecosystem in, say, C++, you'll be reinventing the wheel. Or jumping through hoops with a multiprocess implementation (pipes, sockets etc), and having fun synchronizing those.

Now, not all projects are like that - but reusing libraries from other languages is common enough in large projects. Thus, languages that can't accommodate that use case, don't become truly mainstream. Within their niche, they can be much more pleasant to work with, though. So I don't think there's anything wrong with Elixir per se, and for some tasks, it makes perfect sense - but it's not a very general-purpose tool.

Note, by the way, that I'm not talking just (or even mostly!) about JS, but rather about async/await in general - e.g. also in C#, where that syntax originated, or these days in C++20. On Windows, if you write "modern" (UWP) apps, regardless of the language used, they make a lot of async API calls for stuff like UI - and the implementation is all in native code, running in the same process as your app.

Yeah, it kinda limits the scope of adoption/interest in a way and I think everybody working with it would like that it wasn't the case but I don't think that gap can be worked out easily (or in a practical way) because without the code being written to accommodate the requirements of the VM it can't do what it's meant to do.

If you link a piece of code that can crash the whole VM or steal the processors schedulers then the value in writing supervision trees, restart strategies, compartmentalising your runtime concerns into processes, plus the tradeoffs made in the vm/language design themselves go down, because a single invocation can throw it all out of the window.

(and note, this is not to say the "outside" code is of less quality or anything, is just that when writing "inside" the vm, if you don't account for an unknown problem that happens only sometimes but place it under a proper chain of supervision (and this is much easier to do than writing 100% bug-free I've covered all cases including heinsenbugs code), it will be contained and not bring down the entire VM along with everything it was doing at the time, like all open client sockets or work it was doing, or impact other users when it happens)

Elixir/Erlang has some of the most mature ways of interoperating with external code of any language.

It's literally what Erlang was invented for 30 years ago, building telephone switches where the main logic was on the Erlang VM, interfacing with logic in C which interfaces with the switch hardware.

There are multiple ways of doing it, each with different tradeoffs of safety and performance. You can write FFI code using NIFs, but the Erlang culture of reliability frowns upon it unless you have real performance needs. A good example would be doing cryptography. If the FFI code needs to handle concurrency itself, it can, communicating with Erlang/Elixir via messages. You don't need to care about the threading model. But that's generally not a great architecture. You should rely on the VM's processes to handle concurrency and make the FFI code just be libraries.

It's very common to write performance critical code in a compiled language such as C++ and talk to it over a "port", which spawns an OS process and communicates with it over stdin/stdout. An example is high frequency trading, where Erlang is used to supervise low level code.

You can also write a "C node" which allows a standalone program written in C or Java to talk the native Erlang network distribution protocol. So you just send messages to it.

This kind of thing is very common in Elixir embedded systems, and a joy to work with. See https://www.cogini.com/blog/elixir-and-embedded-programming-...

> Green threads are great, right up until the moment you have to interoperate with something written in another language / using another VM

You mean like this? It's not a problem in elixir.

https://www.youtube.com/watch?v=l848TOmI6LI

I don't know why green thread based languages do not export a pooling interface on their FFI.

Pooling much nicer interface to implement than callbacks, and a quite easy concept to plug one interface into another.

I'm not sure I follow. Can you explain how two runtimes, each with its own green thread implementation, would interoperate in this manner to make a call across the boundary? Say, X (written in A) wants to asynchronously call Y (written in B) and wait on it, passing it Z (also written in A), which Y needs to call several times to perform its task.

X does not wait on Y. It starts the task, and adds the polling interface for the task into its internal polling list. When something is available, calls the continue interface of Y as many times as it needs, just as it does with the OS data polling.

The same happens for Y calling Z.

By "wait" I mean logical wait for the result of the async operation - e.g. to use it as an input for another operation - not a blocking thread wait. What you describe sounds like callbacks to me?

Polling loops are completely different from callbacks.

What I described uses functions to read partial or complete results, and functions that indicate if there is any result to read (and yeah, there must be some blocking thing to indicate whether there is any result at all available). It's something much closer to async/await, except that there is something to tell you if await will block.

It is much more complex than callbacks, but it enables much higher parallelism and somewhat higher single-thread throughput. Also, runtimes with green threads already have it implemented anyway, because it's the preferred OS (any high performance OS) interface for async.

Async/await is just syntactic sugar over callback-based continuations, though.

And in .NET at least, you can easily tell if await will block or not, by looking at Task.IsCompleted.

Reply on news.ycombinator.com