Hacker News Reader

Comment by zrail | Hacker News Reader

If you're working with Ruby I have had good experiences with Que[1], which implements a pattern similar to the OP using advisory locks.

[1]: https://github.com/que-rb/que

View on HN

https://github.com/timgit/pg-boss for node, https://github.com/mbuhot/ecto_job for elixir.

I wish all these libraries agreed on a common schema >_<

Is there a similar library in Python?

I really like the look of this approach, but don't want to build it myself if I can avoid it.

https://pypi.org/project/pq/ looks similar, although it looks like they use `SKIP LOCKED` instead of advisory locks. I'm not sure what the tradeoff is.

SKIP LOCKED is much simpler and doesn't really have any gotchas.

We had an internal library that used advisory locks which had all sorts of strange behavior we couldn't figure out until we just moved to SKIP LOCKED

I was looking into python task queuing recently, here's some other choices from my notes:

https://github.com/malthe/pq - postgres queue based on rq and ruby queue_classic

https://github.com/coleifer/huey - sqlite, redis and in memory by coleifer (peewee creator). Possible to implement postgres storage layer simply.

https://github.com/closeio/tasktiger - flexible redis-based python task queue alternative to celery

https://dramatiq.io/ - actor based python job queue on redis/rabbitMQ

https://github.com/GoogleCloudPlatform/psq - gcp pub/sub based task queue

I just remembered that Celery has a DB backend too, but I've not looked deeply; I can't see it doing anything particularly fancy in the implementation:

https://github.com/celery/celery/blob/master/celery/backends...

I'll second this recommendation. It's great because it just snaps onto your backups for your regular database and you can query on it if you need to.

Anyone know of a good Java library for PostgreSQL-backed queues using the "SKIP LOCKED" technique? I've scoured GitHub but can't seem to find anything.

Reply on news.ycombinator.com