Celery in production: Three more years of fixing bugs
ayushshanker.com
ayushshanker.com
Much of what's written comes back to using old python2 libraries or just complaining that celery doesn't have defaults that better suit you. The very first issue mentioned, about not manually setting the concurrency level...this is basic stuff. It always boggles my mind when people deploy something to prod with the copy/paste "get started" instructions from github. Why do people deploy critical infra layers without spending a lot of time to understand the thing they're depending on?
There are genuinely bugs in celery, and task queues are particularly hard to get right, especially with pluggable backends. But if you RTFD and follow good devops, you will have a perfectly fine time.
Use systemd instead of supervisord for misbehaving forking daemons like this. If you set "KillMode=mixed" you get a nice compromise between "signal the parent and give things a configurable grace period to shut themselves down" and "zero risk of orphaned child (or stuck parent) processes being left around after service stop/restart".
Without weighing in on the systemd-vs-other-init debate in general, I do strongly believe that systemd fully obsoletes supervisord. The (slightly) tighter coupling supervisord offers with Python isn't worth the deficient feature set.
Does systemd have anything like supervisor events?
In most linux environments, this is a non-issue. And some will argue you are using containers wrong if you are running multiple services in a single container, but containers as a reproducible environment and isolation unit is a valid use case (for test/dev environments especially).
Using something Python-based for this sort of thing makes me slightly nervous.
I did have to build my own solutions for things like priority queues, but it was much clearer and testable.
https://dramatiq.io/motivation.html
If you're backing onto something like SQS or Rabbit (Redis is a bit trickier), it's not the worst thing to hand-roll either. Then you understand exactly what trade-offs you're making.
It has its own set of problems (mostly from starting at the set of "we will remove all of Celery features" and then going from there), in particular its logging policies are extremely frustrating and we have had to work around them. But the code is so simple in general that we get good compensation for that.
I personally would have liked to use dq instead but that's mostly an aesthetics thing.
These comments say better than I can why I like it
And yes, celery also supports beanstalk, so it's clearly not an alternative to it
http://docs.celeryproject.org/en/latest/getting-started/brok...
In one case of using it I am routinely throwing work at it. Its main job is to contact third party APIs and parse up to 600MB of XML responses. It's not high traffic but it's active enough to be running jobs often.
This box has been really stable, often times going 6 months without having to restart Flask or Celery because it's humming along without issues and the service itself is fairly stable so it's not getting new versions rolled out (just security patches as needed). It's not a legacy system either, it's just feature complete.
In the past I've had other sites where Celery was doing around a million jobs a month to massage a small amount of data and write it to a Postgres database, it also had no issues and was super stable.
I mainly use standard Celery features backed by Redis on a low'ish end VPS. I would like to know more details about what folks are doing where they're encountering all of these problems.
However if you are doin mission critical work in your queue that cannot be replayed Celery is still the best python flavored option IMO
Redis can be tuned to be safer (such as persisting on every key change) but this all comes with tradeoffs.
Redis such a great piece of tech I use it first as often as I can.
Clustering is another topic entirely.
At risk of being overly negative, my experience with celery has been that it is complex and hard to configure and debug.
I chose a simple queueing mechanism instead of using Celery, and for sending emails I was so frustrated by Celery that I wrote the Arnie SMTP buffer server (in 100 lines of Python): https://github.com/bootrino/arniesmtpbufferserver
Sorry to hear others don't feel the same way. I find it easy to use and maintain.
Fwiw though, the alternatives (huey, dramatiq, rq) don't even seem to offer this feature.
We got a lot of mileage out of it (in a past life) before finally moving our job scheduling to a custom solution built on RQ. Celery caused more than a few headaches -- which is why we ditched it -- but its flexibility probably helped us scale up the service to begin with...
Celery is hard to maintain, but the benefits are definitely worth it, and it’s the most mature cross language stack. I can have a celery job sent or received from any major language.
I think the only reason people pick another option is that that start of by only using python in the first place.
Does anyone have any experience with it? I like the pessimistic execution concept..
Ran beanstalkd in small deployments, rq in others, and nsq for real queueing but at a small scale.
Consider where you see physical queues of people in real world: e.g. people lining up to be served at supermarket, to board aircraft, to enter a concert. The purpose of the queue is an orderly way to manage situations where there is more demand for a service than can be immediately processed. If the system tries to serve all the demand simultaneously, perhaps that leads to chaos and no one getting served.
Similar situations happen in computer systems, in some of those cases queues can be used to manage load.
E.g. When you make a purchase to order something online, there is a good chance there is a task queue or something very similar as part of the machinery helping that to happen. When you click "purchase" your order or request is persisted in some queue or DB, before any attempt is made to fulfill it. The UI can give you immediate feedback telling you that your order was received, here's your order ID, sit back and relax, we'll send you an update. When some machine (or person) is available with free capacity, and no higher priority orders to process, they will grab your order and start processing it.
Some worker queue implementations offer additional benefits in terms of increasing robustness of system. e.g. if worker node A begins processing task T, then worker node A explodes before T is complete, the task T may be returned to the queue after some timeout, and surviving worker B may be able to grab T and process it successfully. This can be very valuable in situations where you want to guarantee that task T is done correctly and not half-done or forgotten -- e.g. processing and provisioning customer orders. This gets considerably more challenging to implement correctly if task T involves performing side effects and modifying the state of the real world (transferring money, dispatching vehicles) and the failure occurs after some but not all of the side effects have already happened.
Queues introduce new failure modes: if something stops workers from processing tasks, and queue gets longer and longer, how do you find out and fix it? Queue needs to be stored somewhere, there is some max capacity, what happens when that is exceeded -- does the queue give up and drop all the tasks on the floor, or explode, or so on.
For example imagine you want to receive a file from the user and extract data from it.
Using such system also allowed us to horizontally scale by adding more worker machines.
All this, of course, depends on the tasks you want to schedule.