Hacker News Reader

Comment by rozap | Hacker News Reader

I think these blog posts usually pretend it's simpler than it is. There's usually (almost always) durable storage required somewhere. But to respond to your example specifically, OTP offers quite a bit. To your points:

Queues: a coordinator genserver process per node, which uses poolboy to only run X jobs at a time

Tracking failures/sucesses: postgres

Exponential backoff: a function

Uniqueness across the cluster: stdlib global locking (https://erlang.org/doc/man/global.html#trans-2)

draining: remote console

periodic tasks: a simple library that knows how to cron

I built a sharded cron-like scheduler in a few hundred lines of code. It's not doing a billion transactions per second, but it's been running in production for a few years without much drama. It's not complicated because it's just postgres and some standard erlang/elixir stuff.

Erlang has plenty of drawbacks, but this isn't it.

View on HN

> There's usually (almost always) durable storage required somewhere.

Despite the name, mnesia can provide durable storage, without using anything outside of OTP and your nodes' (hopefully durable) filesystems.

Despite? It pretty much means "to remember", it doesn't have the alpha privative of amnesia, "to forget"

IIRC original name was Amnesia, but Joe was made to change it for marketing reasons (it's in one of his books or blog posts).

Anyway, I do not think Mnesia is a particularly good way to persist stuff, just use postgres.

At WhatsApp we had a bunch of mnesia-backed services. I can't say that it was the best experience. Pretty much anything non-trivial, such as healing a cluster after a partition, or expanding a cluster, or cross-DC replication was a complete PITA to do properly. There's also very few people that have operational experience with it and very little documentation about it compared to pretty much any other storage engine.

The main benefit was the ability to load and store Erlang terms natively, along with the fact that it's just a library. The fact that your compute is colocated with your storage lends itself to a different architecture than when you use, say, a stateless web service in front of a relational database.

(Edit: I realize now that I'm replying to a WA old-timer so I'd be curious to know what your assessment of mnesia is with the benefit of hindsight.)

I think we would have found any database to be a PITA at write volumes and data volumes we were using, with our expectations of availability. I don't think we would have been able to get that performance out of MySQL, or Postgres; and anyway, we didn't have a dedicated team for database management to make sure nobody wrote bad queries if we did. Sync, status and reg ran on MySQL and php when I started, and it was a lot easier to run them and scale them on Erlang, IMHO. Repartitioning MySQL like we repartitioned mnesia would be a lot harder, I think.

I'm not too worried about a lack of experts and documentation. If you operate at the limits, you'll just need to become an expert, and the source code will be your documentation; mnesia is all written in Erlang, and not too hard to follow; most people won't hit the limits of ets, I'd think. If you don't operate at the limits, you might not need an expert, and if you keep up with OTP releases and hardware releases, the limits get bigger every year.

Looking back, I think we should have spent more time earlier on using the hooks mnesia provides to make healing clusters take less effort; and maybe some automation on cluster expansion. Of course, WA never shied away from manual work by server engineers. :)

(Hmmm, now I'm trying to figure out you are, send me an email if you will :)

Mnesia is first and foremost and in-memory key-value configuration storage. Relying on it for anything else will damage you in the long run.

> I'm not too worried about a lack of experts and documentation. If you operate at the limits, you'll just need to become an expert.

Which is why I'm happy others have paved the way.

A huge amount of web apps can get by with a Postgres backed queue that won't break a sweat handling a few hundred writes per second on a low end cloud hosted VPS.

While leveraging something like Oban you don't even need to be an expert. You can just use the library, throw work into it with a reasonable amount of foresight / best practices and you can start developing features in your app.

Reply on news.ycombinator.com