Hacker News Reader

Comment by djur | Hacker News Reader

I agree. The pattern I've seen more than once is is:

1) Adopt RabbitMQ without any experts on the team 2) Conceal Rabbit/AMQP functionality as much as possible behind a simplifying abstraction, often in multiple layers, often written by non-experts 3) Run into some intractable reliability or scaling problem 4) Have no idea how to solve it because you still don't have any experts 5) Throw a lot of money at the problem, fail 6) Decide to do a very expensive migration to a different system (SNS+SQS, Kafka, etc.)

At that point, you go back to step 1. If you're lucky, somebody has expertise in the new system and the migration can be pulled off successfully. Otherwise, you either end up repeating the whole process or everything goes off the rails when you're halfway migrated to the new system.

This same process happens for all kinds of stuff, not just RabbitMQ, of course.

View on HN

Kafka is at least a bit simpler than RabbitMQ, though both are very square-ish shaped pegs and are usually forced into very round-ish holes. People regularly think they need a message queue, when they really need a job queue, or message bus. Or even all three, but then they try to hack everything on top of Kafka (or RMQ) ... which can be done, of course, but "results may vary".

For tracking state at scale (but still per-job, per-thing) a Cassandra-like system works best (but preferably a better implementation, eg. SkyllaDB or AeroSpike or some other KV store).

Lol to “ Kafka is at least a bit simpler than RabbitMQ”. I’m sorry, what universe does this statement live in?

True, it’s actually a lot simpler than RabbitMQ. People seem to assume “giant ball of enterprisey Java means” that the experience using it will be complicated. Kafka is extremely reliable, simple to cluster, battle tested (I haven’t hit an actual bug in Kafka in ages), the self-healing is turnkey, and has way stronger guarantees for clients.

Where it bites people is that it’s not a queue and scaling is harder than just add more consumers.

I am curious if anyone has used confluent.io (kafka as a service). Or is it too cost prohibitive?

Java is a bit simpler than erlang

> People regularly think they need a message queue, when they really need a job queue, or message bus

This is one of the reasons I am a really, really big fan of Google Cloud's Task Queues. It allows the stupidest, simplest temporal execution of HTTP invocations.

Currently working on a project in AWS and it's stunning how complicated it is to achieve the same simple need of "I want to execute this HTTP call at this time in the future". It's either AmazonMQ -- using either ActiveMQ or RabbitMQ with plugins -- or hacking around SQS's 15 minute delay limit. In our case, we are going to end up wrapping our messages in an envelope with a delivery time and if it hasn't met the delivery time, we put it back into SQS.

GCP is highly underrated for how it simplifies control over execution of code. Pub/Sub and Task Queues both have HTTP delivery built in. Couple that with Google Cloud Run and it is a recipe for building almost any type of execution model with much less complexity and overhead

In case you want to avoid an extra envelope, you can also add custom headers to SQS messages. This can be handy if you want to implement that delay hack without parsing message bodies.

I looked it up but couldn't really find information, what would you say is the difference between a "message queue", a "job queue" and a "message bus"?

I just throw them on the wall based on my experiences, maybe they have some agreed upon precise definitions, but I'm not aware :o

message bus: firehose of events, (by default) no ACK. usually multi producer multi consumer. (see also DBus which is more of an RPC layer + service discovery + pub/sub via event listeners)

message queue: usually between components, ACK, but no selective ACK, backpressure, might even have support for "dead letters" (letters not ACKed by any consumer)

job queue: selective ACK, retry, etc.

(there's also the "enterprise service bus", which is similar, but mostly implemented on things like IBM MQ)

I ran into the exact same problem as the author and I fixed it by...reading the documentation.

In my case, a whole team of devs was using RMQ without knowing anything about it. Literally caused many sev-1 occurrences over the years until I resolved all of the issues. It took a datacenter migration that allowed me the opportunity to redesign the entire RMQ infrastructure before I was able to put the whole mess to bed.

Sounds exactly like the story for many adopting noSql: adopt RDBMS with no experts, hide it behind an ORM, run into scaling and performance problems, no idea how to solve it, scale the hardware vertically, move to NoSQL.

Reply on news.ycombinator.com