Hacker News Reader

Comment by chadcf | Hacker News Reader

Sidekiq's retry and robust handling of jobs (at least with reliable queuing in pro) was the big win indeed. I had a app I tried switching over to it though and it did not go well. In my case I'm running background jobs that take 4-5 minutes to complete and are pretty cpu intensive, and I suspect because sidekiq is threaded it's only using one CPU core and completely thrashing the machine. I can run about twice as many workers with resque as I can with sidekiq, so I had to switch back. Sadly now I've lost the reliable queuing and automatic retries (the latter at least is not too hard to implement in an ensure block).

I do wonder how this one would work out though, will have to give it a try at some point.

View on HN

More or less the same. Plus it will use all cores on MRI, so you can keep using gems with C-Extensions (i.e. you don't have to run rbx or JRuby to max all cores).

Discussing reliability - this is something that sadly Sidekiq will never give you, by virtue of the fact that it uses Redis. RabbitMQ can be clustered in active-active mode, which means you have won over reliability here by just using a cluster.

When comparing queue systems, comparing Sidekiq+Redis to RabbitMQ is a bit unfair - because RabbitMQ was born to do this. And that's why if you're doing proper background jobs and messaging it's better to pick the right tool.

That being said, I do keep using Sidekiq for small Rails apps for the typical background emailers, denormalizers, etc. But I keep an eye open for when I realize that I'm doing proper messaging - in which case I'll switch over to something like Sneakers.

I've got to say that the more I hear "Redis can never be reliable" the more I cringe. It just seems like one of those things that's been said and repeated without people stopping to fact-check along the way.

Redis Clustering tutorial: http://redis.io/topics/cluster-tutorial

Redis persistence (using AOF or RDB or both): http://redis.io/topics/persistence

Still in beta/alpha as far as I last checked (we've been waiting for it). Additionally as far as clustering is concerned there still seem to be a lot of 'unknowns' for me with clustering Redis. Using a cluster in RabbitMQ is dead simple and it just works.

Right now Redis cannot be clustered production-ready. I wish. As I stated in the Wiki, you'll have to pry Redis from my dead body, I am very happy with it, and for me its a true swiss army knife and I've used it as such.

Even though it doesn't have clustering - it's rock solid in production and I haven't experienced a drop in one of my Redis servers in around 3 years.

That being said, if you are building a system where reliability is an explicit requirement you can't take those risks.

you might want to do some fact checking yourself before you cringe.

i would start here: http://aphyr.com/tags/Redis

On a long-running job, I can see that being an issue.

However, what was your concurrency level set at? I know when I've set it to one, I've had good success with the most demanding tasks. You essentially lose the threaded benefit, but keep the other benefits.

What Ruby version/impl are you using? Native threads in a non-GIL implementation should use all available cores.

Resque maintainer here: in recent versions we do the RPOPLPUSH stuff, so 'reliable queueing' should be there.

At least, as reliable as you can get with Redis...

Reply on news.ycombinator.com