Hacker News Reader

I started using worker threads for some batch processing and development tools recently. We have a couple of tricky bits that pull data from different environments to compare them, so the code has to run 'as' two or more environments and forcing a state change was bug ridden as hell, and spawning a child process was slow enough that we limited the scope of how much we scanned (running n worker threads for n environments dropped a lookup from 1.5s to 350ms).

Armed with those successes I turned to thinking about applying something similar to a production Node app, and every time I think about it I realize what I really want is OTP. Each type of processing we do has its own caches, and its own CPU uses, which could dovetail well with worker threads, but I have a chicken and egg problem with trying to move any of that work out of process because I don't get any benefit until 1) none of the workers can silently crash and leave the application nonfunctional and 2) before I'm handling half a dozen concurrent requests per Node process instead of one or two. and 3) before message passing overhead is cheaper

It's either too late or too early to rewrite this whole thing in Elixir.

Perhaps Node is the problem, rather than Elixir being the solution?

If node is the issue here that would make elixir a solution, no? Changing languages (i.e. node is the issue) to get the features described would put you looking at elixir (among other languages that can fill the described properties).

Maybe what you mean is that elixir isn't the only solution?

Yeah, that’s what I meant: “elixir being the (only) solution”. Guess it was worded ambiguously.

Sometimes your problem is that you're using a reciprocating saw when you should be using an angle grinder, sometimes your problem is that you bought Ryobi instead of Milwaukee or Makita.

Indeed, but sometimes you just need to use the right tool for the job. That could be a precision guided flying laser chainsaw axe. Usually it’s a bog standard trusty dumb hammer that everyone else manages to bash similar looking nails with quite effectively.

Would running Nodejs in WebAssembly along with something like Firefly or Lunatic help make the transition smoother?

Or… use the right tool - Elixir. Monoglotism is bad, people.

Here's an attempt at implementing OTP in Go: https://github.com/ergo-services/ergo

Why do you say "attempt"? Does it not work sufficiently well [yet]?

Edit: Thanks for the additional context, dangoor. Cheers.

I haven't tried it, it's just on my radar. It may be great! But OTP has been around a long time, so I'm not going to assume that they've nailed it unless I've either tried it myself or seen folks talking about using it successfully.

A used this a couple of times in production: https://github.com/asynkron/protoactor-go.

No problem launching a 100k actors on a laptop.

That looks cool. Thanks for the link!

> This collection of battle-tested concurrency patterns that has been in use for a long time, in a variety of settings, is called OTP.

OTP seems like a pretty overloaded acronym—what does it mean here?

OTP in an Erlang context refers to the Open Telecom Platform - a bit of a misnomer since you can do a lot of things with it that do not inherently relate to telecoms, but that's how it evolved.

For context, Erlang itself was originally created at the telecom company Ericsson — the name is short for "Ericsson Language". But as concurrency and networking became bigger parts of software in general, the qualities that made Erlang good for telecom software turned out to be generally useful.

> the name is short for "Ericsson Language"

But just for fun, it's also the name of a Chinese Buddhist god: https://en.wikipedia.org/wiki/Erlang_Shen

I think the name was a play on both "Ericsson Language" and the name of Agner Krarup Erlang, who pioneered queuing theory (before it was known as such) to address the fundamental question of inter-exchange capacity planning.

The unit "erlang", for "1 busy circuit/resource" is named in honor of that research.

https://en.m.wikipedia.org/wiki/Agner_Krarup_Erlang

Open Telecom Platform, I think?

Or Outlaw Techno Psychobitch for the Bananarama of Languages. https://www.youtube.com/watch?v=rRbY3TMUcgQ

> It's why you'll hear long-time Erlang folks talk about how, it's really about OTP.

Indeed, this didn't come as a surprise because I've heard Erlang folks rave about OTP. What was really nice about this article (complemented by some of the comments here) was that it gave me a really nice intuition for what OTP really is, beyond "supervisor trees, y'know".

Please, can you share the rust library??

Not sure this is what GP is talking about but to implement the actor model in https://letlang.dev I use tokio.

I don’t remember the name and I don’t remember if it is tokio or something else. However, I did find this: https://docs.rs/genserver/latest/genserver/

I will keep looking. Keep in mind, BEAM uses a preemptive actor rather than async, so it avoids the “storms” that can happen when an async reactor runs out of resources. EDIT: but I guess Tokio is capable of preemption through work-stealing too?

This looks interesting too, though no OTP out-of-the-box: https://docs.rs/axiom/latest/axiom/

Can you say more about these storms? Sounds pretty interesting. How does preempting solve it?

I don't know Python or Rust well, but I think this is the problem:

https://betterprogramming.pub/the-dangers-of-async-in-python...

And this is how some people are solving it:

https://async.rs/blog/stop-worrying-about-blocking-the-new-a...

This is how it works in BEAM:

https://github.com/happi/theBeamBook/blob/master/chapters/sc...

The commonality is to limit execution time and yield (preempt) so something else can run too.

I'm the author of something related https://github.com/lunatic-solutions/lunatic