Hacker News Reader

Comment by plainOldText | Hacker News Reader

For the record, I would prefer not to get into a programming language flame war. Having said that, I will add the following: It's much harder to reason in Rust, as there are many concepts one needs to keep in their head (borrowing, lifetime, pointers, etc). Plus Rust is much more difficult to read. Whereas Elixir, and I emphasize Elixir over Erlang here, is much easier to reason in, concise, and simpler to read.

At the end of the day, use whatever language you prefer, just keep in mind that software needs to be (1) shipped, (2) maintained (by multiple people who read each other's code) and (3) evolved. I would also add (0) experimentation; before one even ships any code, one ought to easily experiment with various ideas.

View on HN

I content Erlang can be very hard to reason about. I've seen some extremely gnarly Erlang where the author didn't write a -spec and it was almost impossible to tell what shape a tuple parameter would take. In Rust the compiler enforces all of this for you. Rust gives you:

- Maintainability: Rust signatures not only tell you the types of arguments, but also their lifetimes. A signature in Rust is an extremely strong abstraction. Traits can further constrain types making it essentially a game of plugging the right blocks in to the right holes. Erlang is like having the blocks but all the holes are under a tablecloth and you just have to guess how to fit them in.

- Correctness: Without a static type system Erlang does very little for correctness. Rust has pattern matching as well and can enforces exhaustion. Not even Haskell does that. Lifetimes guarantee safety even for shared pieces of data.

- Speed: Obviously, Rust is a compiled, manually memory managed language with near C++ performance.

On every metric, Rust offers a lot more than Erlang. As someone who has spent years with Erlang, Rust is simply better for professional development.

Rust however does not offer nearly the same level of power when it comes to runtime introspection that BEAM does.

Coming from a company that uses elixir heavily and has made a significant investment in rust, I don’t think we would ever use solely rust on our distributed systems. However, we have rewritten some code that elixir was too slow at in rust and exposed it as a NIF on BEAM - and that has worked well. (Blog post on that soon hopefully).

I do admit, we are also going to be ditching mnesia for one of our clusters for our own in-house simpler system (ETS replication with different consistency/netsplit guarantees for our use-case), we've had to write our own cross-node process monitoring solution (at peak we see 200M+ cross-node monitors on our cluster), and we've also had to overcome the limits of message-fanout on distribution as well (https://github.com/discordapp/manifold).

However, for operating at our scale (peak 9m ccu @ 5m events/sec fanout to clients), we run a surprisingly small number of servers for our real time system (~120).

EDIT: I can't reply to your post below, but I think the runtime introspection we run into is not dealing with OS level metrics, but application level introspection. Introspecting the state of processes, writing code in the repl to debug issues within the cluster, benchmarking to find hot functions or where specific processes are spending a lot of their time. Capturing traffic to replay on a test cluster to simulate production load, all becomes very trivial with BEAM.

We couldn't get enough introspection. Now we use Rust and we can use a much larger toolkit. We have BCC, kprobes, flamegraphs... When you use Rust the OS is your machine. With Elixir you need to understand the BEAM, a slightly esoteric platform with a rather small community working on it.

Yes, we can do that through flamegraphs. For any container running on our system, we can get the call stack of the running process, time taken, resources used etc. We can attach to containers, proxy traffic etc, all through Kubernetes.

I said this earlier, but we've essentially separated operation concerns from our applications, and that opens us up to relying more on knowledge of Linux which is easier to hire for, and we can reuse that knowledge and all our tooling with any other languages we want to use.

We run between 200k and 2m processes per beam VM. I don't know how it'd be possible to get as precise metrics as we need just from relying on linux utilities. And in the same cluster, some processes although identical in code have dramatically different workloads.

> and enforces exhaustion. Not even Haskell does that.

Maybe it doesn't throw an error message, but ghc does warn about it when you use the -Wall flag. I believe you can get the behaviour you want by turning on only that specific warning (I forget what it's called) and using -Werror.

Reply on news.ycombinator.com