Introducing Resque | Hacker News Reader

Introducing Resque

github.com

135 pointsmrduncan15 comments

I wrote DJ. This looks amazing for high volume sites. At Shopify DJ is barely able to keep up these days.

My biggest regret with DJ is actually that I added numerical priorities instead of choosing the same tag based approach that Resque implemented. I'll likely release a DJ2 at some point which backports some of the ideas.

FWIW the author of DJ posted in more detail on this subject in this Google Group:

http://groups.google.com/group/delayed_job/msg/9e2987caa177b...

FWIW, xal (who you replied to) == the author of DJ.

http://github.com/tobi/delayed_job

The Resque code aside --- and I want to switch to Resque, like, tomorrow --- this is the best article on Ruby job queues I've ever read. It's Github's review of pretty much every mainstream job queue out there. Excellent.

Redis is almost perfect for a queue because of it's lists and atomic operations. And, most of all, because it's persistent. It's amazing how many queues aren't persistent.

I'm glad someone finally did this, it could be a great solution. I'm looking forward to digging into this.

And, most of all, because it's persistent.

Just a slight caution, since Redis stores it's data in memory and asynchronously writes out to disk it is possible to lose small amounts of data in a crash. In most cases this is an acceptable trade off for the speed of an in-memory database though.

There is also plenty of good info on the project page - http://code.google.com/p/redis/

Since Redis 1.1 (currently in beta - don't use it in production for now) there is a new append-only journal storage system alternative to the main one (that is, snapshotting), so starting from 1.1 Resque will be more reliable as well if needed, because in this new (optional) configuration Redis can't lose data. It's also possible to configure if it should fsync() never, everysec, or after every write. It's up to you to select the tradeoff between speed and safety :)

Fantastic. Redis can go in both my ephemeral and critical systems.

You're a whirlwind of activity.

Exciting stuff maybe we can drop JMS and use this instead

Nice. We built a mysql-backed queue with nearly identical properties but it's not especially efficient. It does 10+ jobs per second per worker, but with locking this starts to cause InnoDB concurrency problems. Not insurmountable, but moving to a Redis-backed approach seems like a great option when our current queue can no longer keep up. Really glad to have this approach proven by GitHub.

Really looking forward to trying this - I've given backgroundrb, BJ, DJ, and workling a go and there's always been something that's frustrated me.

It's amazing the number of different things I'm considering using Redis for as well.

I've also been through a lot of different background processing libraries for Rails, and so far, if you have simple needs, I'm finding that Spawn is really awesome. http://github.com/tra/spawn/

I think something like Resque will be very helpful to us over the long term, but if you're looking for something quick that just works to send a big batch of emails, import stuff via a web service, whatever, check out Spawn. Not at all appropriate for a site like Github, but works well for our vastly smaller websites.

Also see VertexDB (http://github.com/stevedekorte/vertexdb).

It is a high performance persistent graph database with filesystem like semantics (will soon support FUSED). It presents an API for an atomic queue pop operation, which allows for synchronization of jobs.

Slightly off-topic, but is there any stable background job queue available for PHP (and perhaps MySQL)? I have been using custom hacked solution on top of tokyo tyrant but I will prefer if someone has a mature library.

Jeeze, I just started migrating from BJ to DJ ... what should I do now?

Reply on news.ycombinator.com