Hacker News Reader

Comment by jbaiter | Hacker News Reader

Are there any non-distributed task queues for Python? I need something like this for a tiny web application that just needs a queue for background tasks that is persisted, so tasks can resume in case the application crashes/restarts. Installing Redis or even ZeroMQ seems kind of excessive to me, given that the application runs on a Raspberry Pi and serves maximum 5 users at a time.

View on HN

My suggestion is to just use RabbitMQ. It's written in Erlang, it uses a BerkeleyDB-like backend for message storage. It's non-distributed, "durable", and optionally with persistent and non-persistent messages. It has a web interface to examine messages. Second suggestion is to use JSON for your messaging format, although it's possible for basic tasks to put all the info you need in the headers.

I have personally found RabbitMQ one of the most bonkers, overengineered and painful to manage services I've ever dealt with.

I wrote https://pypi.python.org/pypi/pq, a transactional task queue for PostgreSQL, depends only on psycopg2.

It does about 1000 ops per second, depending on your database configuration, network etc.

That's pretty cool. What are the benefits of using this over celery?

Celery is typically used with RabbitMQ, which is an advanced message broker.

If you don't need the complexity (most don't), need persistence, transactional guarantees and already have PostgreSQL in your stack, then the pq-library is probably a good option.

It's not really for a distributed setup, but meant to operate within a cluster (local area network, same data center). Most people don't have distributed systems ;-).

I wouldn't really classify this example as a "task queue", it resembles more the RPC pattern really (and I would guess that this example does not persist the jobs in any way). Celery does have a very experimental filesystem based transport that could be used for your use case. I don't know if it fits on a Raspberry PI but celery does not have very high memory/space requirements (c.f. other Python libraries).

ZeroMQ takes care of the queueing. Though I didn't delve into it in depth in this example, you can create quite sophisticated broker-less distributed systems pretty easily with ZeroMQ.

Right, I understand that ZeroMQ is used to send and receive messages, what I mean is that there's no persistency involved so the tasks will not survive a system restart.

No, I mean ZeroMQ takes care of the underlying queue for free (there is an underlying set of queues, which can be persisted if necessary)

No, it doesn't, although it might be possible to sort of emulate it by setting ZMQ_HWM to 1 and enabling ZMQ_SWAP, but I wouldn't bet on it.

The best you can hope is to use the Titanic Service Protocol and just throw data into some sort of disk store. I've looked into doing this, but I settled on using RabbitMQ instead for persistence. Unless you're dealing with more than 10k messages a second, it's just as easy as ZeroMQ.

After a few months of experimenting, I've come to the conclusion that some combination of ZeroMQ and RabbitMQ is likely the easiest solution currently for a combination of low-overhead distributed messaging and broker-assisted persistent messaging.

That must be a very recent development for 0mq then, do you have any source for this new feature?

Are you thinking of RabbigMQ perhaps?

In a similar situation, I've used inotify to watch things just get dumped on the disk. It's a hack, because race conditions, but sometimes that's OK.

You could use mutex locking on the file system to avoid the race conditions.

maybe use sqlite?

http://flask.pocoo.org/snippets/88/

With PostgreSQL, you can use advisory locks and other little tricks to get better performance, e.g.

https://github.com/malthe/pq/blob/master/pq/__init__.py#L224

Reply on news.ycombinator.com