Not that Celery is the only way to go. This is good work - I'm just interested in the why.
Not that Celery is the only way to go. This is good work - I'm just interested in the why.
At the time (it was a couple of years back) there was no way to stop it from taking 2 tasks at a time from the queue. We have a smaller number of long running tasks and it made it very difficult to scale out the work. Also, the monitoring story was pretty rubbish and debugging was hard.
Now we use RQ, which is mostly pretty good but we've been slowly adjusting parts of it's behaviour to make it more suitable for us. Recently, we did some work around how it does it's TTL so we could more quickly see where workers had terminated abnormally.
We've also changed it so a given worker instance will become increasingly unlikely to take on new work as the machine it's running on (AWS) nears the end of the hour since it was started. As the work load lowers, we can remove workers from the system in the most cost effective way.
I'm sure they exist in theory, but it's overkill in every situation that I've seen it deployed.
also, rabbitmq has an amazing admin interface, monitoring is really easy (most things you use for monitoring support it) and the whole thing has really sane defaults (imho).
all in all, it's one of my favorites pieces of software ever!
(btw, i remember people having problems with celery because of they way they imported the tasks. if you always do full imports (as in, from a.b.c import tasks instead of from . import tasks) you shouldn't have any issues).
[1] https://www.rabbitmq.com/getstarted.html
PS. I'll stop evangelising soon, I promise.