Hacker News Reader

Comment by tombert | Hacker News Reader

Disckaimer: I have never used Elixir in any serious capacity, but I have done a good chunk of Erlang.

Concurrency in Erlang sort of frustrates me...not because it's bad, but because when I use it I start getting pissed at how annoying concurrency is in nearly every other language. So much of distributed systems tooling in 2023 is basically just there to port over Erlang constructs to more mainstream languages.

Obviously you can get a working distributed system by gluing together caches and queues and busses and RPC, but Erlang gives you all that out of the box, and it all works.

View on HN

    Any sufficiently complicated distributed program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Erlang

I've done this multiple times. I will not make the mistake again. Next big system is being born in Elixir.

Well, give me a call if you need help. Just looking for work again lately.

Feel free to shoot me your cv/resume to the address in my profile! We just hired two new engineers this week, so the timing is not perfect, but I would like to have your info around for a rainy day which might come sooner than later.

Just did. Thank you.

Same as @pdimitar. I’m looking for part-time work.

please see comment to @pdimitar and do the same ;)

Virding's First Rule of Programming:

http://rvirding.blogspot.com/2008/01/virdings-first-rule-of-...

That has definitely been my experience. There's been a good number of times I end up gluing things together with ZeroMQ and SQS and Redis where I start thinking "you know this would have been easier to get an equivalent (or better) product in Erlang".

Sadly, I've only ever had one job at a shady startup that used Erlang; I would love to work for a company that sees its power, but sadly it seems that everyone who has not drunk the Erlang koolade thinks that Go is a suitable replacement.

But go only has 3 or 4 ways you can shoot yourself in the foot at every turn with concurrency! Better than C++, I guess.

I don't really mean to crap on Go, I actually think it's a reasonably ok language all things considered; if nothing else (at least on one machine), it gives you nearly the highest performance-to-effort ratios.

It just annoys me that people see "Go is good at concurrency", and then see "Erlang is good at concurrency", and then assume Go ~= Erlang, which is not the correct conclusion.

This is coming from someone who likes Elixir. Not much for its distributed systems features, but mostly because of the language design. I keep hearing everyone talk about how Erlang/Elixir gives everything out of the box and you don't need to worry about Queues, RPC or whatever... But in reality, people don't really recommend using Distributed Erlang that much, on most Elixir gigs I worked, they didn't use Distributed Elixir at all, just plain old Kubernetes, Kafka, Redis and GRPC.

> they didn't use Distributed Elixir at all, just plain old Kubernetes, Kafka, Redis and GRPC

There must be a good rationale for that decision. Do you know what it is?

There may or may not be “good” rationale. Could just be that most people using elixir are coming from other languages/ecosystems where all of that is normal.

Also in my experience, most of the time, the infrastructure team doesn’t know anything about elixir.

There's a good article about BEAM + k8s by José Valim [0]

[0] https://dashbit.co/blog/kubernetes-and-the-erlang-vm-orchest...

You could certainly get away without some of the other stuff but, as another comment has mentioned, it requires some infra know-how. Like, you can't "just" use ETS as a Redis replacement without setting it up in a way that its data won't get blown away during a blue-green deploy.

Well, in big orgs, that adopt Elixir/Erlang, along with other technologies with poor concurrency stories, those other ecosystems still need Kubernetes, Kafka, Redis and GRPC, to get by. elixir isn't going to make ruby or python apps magically concurrent. So that make sense.

However, in orgs that are primarily Elixir shops, I don't see a lot of Kafka or gRPC. (Redis is just useful, its more than just a queue and K8s and Elixir/Erlang compliment each other, btw.)

>those other ecosystems still need Kubernetes, Kafka, Redis and GRPC, to get by

And what makes Elixir not need Kafka, Redis or GRPC?

Instead of Redis, you could use ETS for caching. But once you have 2+ instances of your app, you will need to have a centralized caching mechanism, otherwise, each instance will have its own ETS with its own memory, not sharing anything. Unless you decide to use Distributed Erlang and connect all the nodes of your application, which comes with a lot of trouble. Much easier to just use Redis.

And lets say you have multiple teams, each team has its own service(built with Elixir), and you need to have some async communication between those services. What native Elixir solution would you use instead of Kafka?

Same for GRPC. What's the alternative? Connecting all the nodes and using Erlang message-passing?

> Connecting all the nodes and using Erlang message-passing? Most of the time, yes.

I think Elixir/Erlang + Redis pub-sub + PostgreSQL is the sane minimal subset for distributed and scalable systems.

Just say no to Kafka.

Where kafka comes in though is no-code db following. These are super handy when you want to be informed of changes to a table, but don't know which "micro-service" is changing the data. KafkaConnect is very handy.

Though I will concede, it's a bit of a bazooka for a mosquito, sort of thing.

After using NATS I don't think I'll ever want to use redis pubsub again.

ETS is quite faster than using Redis though.

single node though, so you have to add distribution in some manner. For some situations, whether its the "right" way or not, redis ends up being an easier way to go.

ETA: As another couple of comments pointed out, ETS also dies with the node so you've got to handle that also when rolling it.

ETS is cool, but its not a panacea.

Nebulex has different adapters although I've only use it on a local node which uses ETS and with transient data so I can't comment on them too much.

https://github.com/cabol/nebulex

well my startup uses distributed elixir. we use it horde to distribute certain long lived processes accross our cluster. that does not exclusive to kuernetes. we user kubernetes to manage and handle vm crashes (its only happened twice ever) as well as porvide consistent network toplogy between the nodes.

that said, having the ability to send messages between machines without having to bring in an external dependency like redis is AWESOME from a maintenance perspective. one less thing to worry about and the system just works. our uptime has been pretty amazing given our small team.

Could you explain a bit more how you are using it?

our payment processor requires a persistent connection to their server for each of our clients. we use horde to make sure there's only one process responsible for each paying user accross our cluster.

> I keep hearing everyone talk about how Erlang/Elixir gives everything out of the box and you don't need to worry about Queues, RPC or whatever...

Many companies are using Distributed Erlang but not the way you described: they are using it to exchange messages between nodes running the same version of the software. Imagine that you are building a web application, you can directly exchange live messages between nodes without Redis/RabbitMQ [1]. If you are deploying a machine learning model, you can route requests through multiple nodes and GPUs without additional deps [2]. If you want to see which users are connected to your app, you can exchange this information directly between nodes [3].

In other words, there is a subset of distributed problems that Distributed Erlang solves very well out of the box: homogeneous systems working on ephemeral data. And some of the scenarios above are very common.

If you need to communicate between different systems or you need to persist data, then I 100% agree with you, I would not use Distributed Erlang (at best it would be one part of the solution). I think this distinction gets lost in many conversations and sometimes it leads to false dichotomies: "why Erlang when I have k8s/grpc/redis?" while in practice there is not a lot of overlap. I have written about Erlang/Elixir vs Redis [4] and vs k8s [5] for those interested.

[1]: https://hexdocs.pm/phoenix_pubsub/Phoenix.PubSub.html [2]: https://news.livebook.dev/distributed2-machine-learning-note... [3]: https://hexdocs.pm/phoenix/Phoenix.Presence.html

[4]: https://dashbit.co/blog/you-may-not-need-redis-with-elixir [5]: https://dashbit.co/blog/kubernetes-and-the-erlang-vm-orchest...

> In other words, there is a subset of distributed problems that Distributed Erlang solves very well out of the box: homogeneous systems working on ephemeral data. And some of the scenarios above are very common.

Speaking of which, I'm looking forward to using Broadway [1] in a new project here in my company. Here, people are using an enterprise integration engine specialized in the healthcare space [2], with built-in single-branch version control and all actions going through the UI.

As I come from a background of several years with Ruby on Rails, I really hope to convince people to use this great library/framework your company created, since RoR is severely lacking when handling heavy concurrency like when gluing multiple APIs in complex workflows. Software engineers are going to love it, but integration analysts are used to IDEs with GUIs, so we'll need to create a pretty admin dashboard to convince them to switch.

[1] https://elixir-broadway.org/ [2] https://rhapsody.health/solutions/rhapsody/

We use it across multiple versions of our software running in the same cluster. As long as you dark launch API changes, it's not much of an issue.

https://www.youtube.com/watch?v=pQ0CvjAJXz4

(Doh, just realized I was replying to Mr. Elixir himself! And you're familiar with our project anyway :)

Cargo culting happens regardless of language. It's true that fully meshed distribution won't fly for Google or Amazon scale. But 99% of companies will never get to that scale, despite what they'd like to believe. Fully meshed distribution works just fine for many use cases.

> It's true that fully meshed distribution won't fly for Google or Amazon scale.

I'm not sure I agree at product level: WhatsApp seems to be scaling rather well. I can't say if their use of Erlang is "fully meshed distribution" or not, but it seems to be flying just fine as the world's number 1 messaging platform.

Elixir/Erlang often simply replaces most of what Kafka, Redis and GRPC offer.

Also, have a look at the Phoenix Framework Channels examples, as it essentially replaces most simple micro-services architectures.

This recipe book covers the common design examples:

"Designing Elixir Systems with OTP: Write Highly Scalable, Self-Healing Software with Layers" (James Edward Gray, II, Bruce A. Tate)

One day, you too may get annoyed with IT complexity, and articles mostly written by LLM chatbots.

Happy computing, =)

Sasa Juric makes this point in 'Elixir In Action' and some of his talks, where in other languages you need to pull in these other technologies near the start, whereas in Elixir you can use the built-in primitives until you start running into scaling problems.

- https://youtu.be/JvBT4XBdoUE?si=Xo0QXgVSI2HCg8pj&t=2198

"just plain old kubernetes" is an oxymoron, at best.

I fell in love with Erlang pretty quickly, and it’s hard for me to enjoy writing other languages. Simple concise syntax, pattern matching, immutability, error handling without branches all over the place, concurrency… it’s hard to walk away from that.

Have you tried Elixir? So many blogs etc. talk about coming to Elixir from Ruby, which I don't understand, little similarity beyond def...do...end.

Elixir uses the same VM with a different, some including myself would say better syntax. Do you still prefer Erlang syntax?

I dislike Elixir’s syntax for two reasons: it seems unnecessarily verbose, and it’s too much like Ruby/Python.

When I see Erlang syntax, it helps me think in Erlang.

Anyway, I haven’t given Elixir a fair chance, but it’s hard to get past my knee-jerk dislike of all the extra verbiage. I’d rather use LFE if I wanted an alternative syntax with more powerful macros.

regarding concurrency, language plays an important role and pretty much dictates how code is written which is i believe where it has most of the frustration. in erlang it’s in a functional style, in javascript it’s in an asynchronous style. what i’ve come to realize is that it’s still better and more maintainable to think synchronously and have the core of the tech handle concurrency, for example golang with it’s goroutines or multi process in ruby/python. admittedly it’s not as concurrent/distributed as erlang but should be easier to work with.

thats the thing though, elixir you can write it in a synchronous style and to make it concurrent is usually very easy because the semantics of regular and concurrent code is essentially the same.

for example refactoring something like this:

    File.stream!("path/to/some/file")
    |> Stream.flat_map(&String.split(&1, " "))
    |> Enum.reduce(%{}, fn word, acc ->
      Map.update(acc, word, 1, & &1 + 1)
    end)
    |> Enum.to_list()

to be async, concurrent, (Flow uses GenServers under the hood, A GenServer is a process like any other Elixir process and it can be used to keep state, execute code asynchronously and so on.)

    File.stream!("path/to/some/file")
    |> Flow.from_enumerable()
    |> Flow.flat_map(&String.split(&1, " "))
    |> Flow.partition()
    |> Flow.reduce(fn -> %{} end, fn word, acc ->
      Map.update(acc, word, 1, & &1 + 1)
    end)
    |> Enum.to_list()

The point is the code isn't radically different and is easy to understand if you understand the first block of code.

Yeah— as a primarily python guy, I find concurrency much more palatable in elixir than in js. As a relatively infrequent js user, I have tripped on its asynchronicity for decades. After learning in Perl, shell scripting, and a smidge off C in the early 2ks, then PHP and python in the subsequent decade, I rage quit js every time I picked it up for anything significant until like 10 years ago. For the first few years, I always forgot basic facets of the language like the scope of "this" in anonymous callback functions.

While it's not a close analog, the way Unreal Engine's node-based "no code" Blueprints language approaches asynchronicity just feels so much more natural. Even hopeful pre-hello-world coders seem to conceptually get "well I can't get that because this hasn't happened yet." Having graphical representations of things both in nodes and in-game obviously helps in ways that wouldn't make sense in js, but being built from the ground up to handle it does show that it can be approached more intuitively.

Node's "experimental" stream api also leads to code with similar semantics. It gives some additional concurrency options, but of course no parallelism outside of waits.

    createReadStream("file.txt")
      .flatMap((chunk) => chunk.split(" "))
      .reduce((acc, word) => {
        acc[word] = (acc[word] ?? 0) + 1;
        return acc;
      }, {})

Can you give some examples?

If you want to dig into some code:

The Elixir 'Getting started'[0] guide has you building a concurrent, distributed KV store using nothing but the basics of OTP (effectively the std of BEAM).

0. https://elixir-lang.org/getting-started/introduction.html

Reply on news.ycombinator.com