Hacker News Reader

Comment by moe | Hacker News Reader

I'm still hoping for a simple protocol with strong guarantees and persistence. RabbitMQ seems to be a very good backend implementation but its the choice of AMQP for the main protocol is a pity.

AMQP is a ridiculously over-engineered monstrosity and a saturn-sized ball on a chain for any project.

View on HN

Seriously. Roll your own over Redis.

Redis is faster than BeanStalkd, a tiny message queue server written in C that uses libevent and a simple memcached-like protocol. Redis kicks ass, specially the beta versions.

Redis is a k/v store, not a queue.

Key == mailbox/slot, and value == message. All you need are concurrency primitives, and broadcasting (in the case of a distributed queue)

People have modeled message queues with far more primitive solutions:

http://www.freeopenbook.com/php-hacks/phphks-CHP-5-SECT-18.h...

A Redis based solution might not be as elaborate as RabbitMQ, but it will be very clean and straightforward, not to mention blindingly fast, and will get the job done.

Well, I'm quite skeptical about the "blindingly fast" part (network overhead?) but agree that in the light of the current mq-options that's indeed one of the more appealing ideas.

That said, I'm not really in the mood to reinvent that particular wheel myself here, but would probably give someone else's shot a try.

Actually Redis is a bit more than a k/v store these days. It provides implementations of a variety of basic data structures, including lists that support queue operations. There's BLPOP[1] and BRPOP for blocking job/message requests and LPUSH and RPUSH for placing a job/message on the queue.

That said, I'd still like to see benchmarks showing Redis beating Beanstalkd. Of the two, I've only used Beanstalkd, but it's a great example of keeping things simple while still providing advanced functionality.

1 http://code.google.com/p/redis/wiki/BlpopCommand

By what metric is redis faster than beanstalkd?

Does redis has any job queue facilities for making a reliable, low-latency, safe job queue?

http://www.rockstarprogrammer.org/post/2008/oct/04/what-matt...

Yes, it does. See the LPOPRPUSH and BLPOP protocol commands. There's a number of threads on the Redis Google Group about using Redis as a job queue. There's also Resque by a githubber.

http://github.com/defunkt/resque

RPOPLPUSH does about half of what's needed and BLPOP does the other half, but neither does the whole job.

It's not interesting to discuss what you can use to hack a queue together. Lots and lots of really badly thought out queue services exist today built on top of memcached or a relational database or other things that appear to work in normal cases, but can go horribly wrong and start losing data when bad things happen.

beanstalkd is not really a message queue, its a job queue. I understand that job vs message can get into semantic nonsense, but in essence a message queue can and will send any given message to n people. A job queue will only send it to the first requester. Redis does kick butt, but a k/v store is not the same as a broacast (or 1toMany, or many to one) meassage store, is not the same as a 'give me the next job' (1toAny) server either.

How exactly would you do this with Redis? Redis does not push out data to N > 1 listeners/observers/subscribers. I suppose you could use either SYNC and pretend to be a replication slave, MONITOR and watch for RPUSH, or BLPOP. BLPOP hangs/blocks the client though.

Sure, the Redis C codebase is fairly approachable. Perhaps you were suggesting adding new protocol commands?

If Redis uses TCP it will bottleneck long before 0MQ does, because 0MQ uses a reliable-multicast protocol for communication over the LAN. As discussed in the linked article, in the comments.

Also, Redis is a centralized server and a single point of failure - 0MQ doesn't force you into that topology.

I'm curious, what do you feel is overengineered about AMQP?

I've been using it for a while, and I find it to be quite simple. I send messages based on a routing key. They go to the right place, and reasonably fast (though not as fast as zeromq). What's overengineered about it?

Well, for starters try to find a good AMQP tutorial. There's Rabbits & Warrens (which is good but still quite rough) and that's about it. Large parts of the semantics (exchange types, interactions between the various flags) are needlessly complex. Most of the same could have been achieved with much simpler primitives (see other protocols like spread, stomp etc.).

Even one of the authors in this article admits dumb decisions like making it a binary protocol and generally making it extra-ordinarily difficult to create a working protocol endpoint. What's up with this whole XML descriptor thing...

In summary AMQP has been the most painful protocol I had the misfortune to work with since leaving java-land. (Admittedly partly because the python drivers are especially messy)

That's interesting - my experience has been the opposite (that is, AMQP has been quite easy to work with), but I wonder if it's because I primarily use the Ruby clients (http://github.com/tmm1/amqp for asynchronous, http://github.com/celldee/bunny for synchronous)? I've also used the Erlang client a bit, which was somewhat trickier. Maybe this should inspire me to start a blog and write up a tutorial.

It isn't just ruby clients, I've found the python client to be similarly simple. I get the impression that there can be a lot of complexity to AMQP - but so far the c and python client libraries have hidden nearly all of it from me.

Look into the DDS protocol -- various implementations exist, and many of the features you want are available. I believe the open source implementation is called OpenSplice

Reply on news.ycombinator.com