Comment by lobster_johnson | Hacker News Reader

Comment by lobster_johnson | Hacker News Reader

Parent Full thread

lobster_johnson

Indeed. In my opinion RabbitMQ is essentially useless in clustered mode.

When Rabbit recovers from a network partition and has to decide between multiple potential master versions of a queue, it picks the largest one to become the new master, and discards the others. It's rather mind-boggling that it can't merge them instead; after all, if your application is capable of handling duplicate deliveries, then merging (which would potentially result in previously ACKed messages becoming visible again) would be a perfectly acceptable solution.

The only way to make it non-lossy is to turn off HA recovery and manually handle network partitions, but it turns out that's not practically feasible, because there are no tools to work with Rabbit queues at a low level; the only way to recover is to discard one or more nodes.

We've also found Rabbit's clustering to be very flaky in general, beside the lack of partition tolerance. We recently had a Rabbit crash where one Rabbit node (not the machine itself) went down, and things got really stuck; the only way to recover was to stop all the nodes, then start them again. After we did that, all the queues were empty. We've also had instances where suddenly bindings go missing, or the bindings are there but attempting to declare them from a client fails with an "bindings already exist" error. And many other weird errors.

The last year or so, after having to endure all of these issues, we've decided to ditch clustering altogether and run a single node. That's risky, but ironically it's a lot more stable than our previous three-node cluster.

In my opinion, Pivotal really needs to redesign RabbitMQ's clustering.

Has anyone successfully moved off Rabbit? ActiveMQ, NSQ? Disque [1] looked promising, but seems dead (last commit was 18 months ago) at this point.

[1] https://github.com/antirez/disque

For a stable RabbitMQ cluster, you want dedicated RabbitMQ hosts with sufficient CPU, disk & network throughput for your workload. Most RabbitMQ users don't know what their workload is, or what their hardware boundaries are. We, the RabbitMQ team, should make this easier - and we will, in due course.

A good default cluster is 3 x r4.large with 100GB GP2 for RABBITMQ_MNESIA_BASE & pause_minority. For queues that need HA, a good default is ha-mode: exactly, ha-params: 2, ha-sync-mode: automatic. As for the Erlang version, we recommend 19.3.6.2 which has important fixes relevant for RabbitMQ. Today we recommend RabbitMQ 3.6.11, and 3.6.12 as soon as it ships.

In the past 6 months, I have been focusing on RabbitMQ stability and operability on AWS, GCP & vSphere. Can you tell me more about your RabbitMQ deployment lobster_johnson? This will help: https://s3-eu-west-1.amazonaws.com/rabbitmq-share/help-us-un...

I wouldn't mind moving this discussion to rabbitmq-users mailing list, so that it can benefit more in the RabbitMQ community.

Thanks, Gerhard

Antirez said he plans on merging disque into a Redis module, now that such a thing exists. I'm pretty excited, and would love to migrate off of RabbitMQ to Disque or whatever the module version is named, as we're already successfully running redis instances.

As for merging queues after partition recovery, the RabbitMQ devs have been talking about implementing that for years. I understand it's a hard problem, or it would already be part of RabbitMQ, since it's the most obvious and desirable solution for applications that can handle duplicates.

We're doing the same thing wrt avoiding clustering and accepting the brief downtime when the single RabbitMQ instance fails.

Yes.

The only reason why we are not offering a clustering plan is because when you experience a network partition in such a case, you will loose data. And we do not want to be in a situation where we are explaining to our customers that there data has been lost through no fault of our own, but because that's the way RabbitMQ works.

We would rather have single node plans where failure modes are much easier to deal with.

We run our RabbitMQ cluster with pause_minority as the partition handling strategy. This should eliminate most message loss on partition, no?

Yes. A bonus is that if you use it in combination with a health-check-capable proxy (HAProxy, Kubernetes), clients can be routed to any non-paused node automatically. In fact, you'll need that, since a paused node will close its ports.

That said, the devil is in the details; I'd be interested to know if RabbitMQ is capable of reliably detecting that it's in a minority. We've had issues where nodes are having issues talking to each other, but the problem is not consistent on both ends (e.g. A can talk to B, but B can't talk to A).

Hi, even in pause_minority mode; when the node/s that was in the minority rejoin the cluster, then it will loose any messages that it had and assume the messages of the majority.

Reply on news.ycombinator.com