Hacker News Reader

Comment by bsaul | Hacker News Reader

Yeap, celery is really when i started to feel the pain of using python rather than a more enterprise-ready language&framework. Although in my case it wasn't using it with django but with flask ( but i don't think it makes a big difference).

The main issues when you start to have a stack built of big independant systems ( such as a db, a message queue, worker processes, and web server) are more related to deployment, monitoring and failover management. But they are really the big issues that justify the choice of a technology instead of another.

View on HN

I suspect people jump to Celery when something much simpler would probably be enough. I've seen people use Celery with all the associated deployment and monitoring complexity when all they wanted to do was run a slow task without blocking a request.

You could simply use this: if all you need is to run a slow task and not block a request there's this: https://github.com/defrex/django-after-response

I've usually used the system crontab (sometimes via https://github.com/andybak/django-cron ).

If you really really need a proper task queue then Huey is much lighter than Celery: https://huey.readthedocs.org/en/latest/

If you do need something more than

django-after-response looks fantastic, it'll make 50% of my apps much easier. Does anyone know what the lifecycle of the spawned threads is like? I'm guessing it depends on the server, but it looks like request + tasks need to take less than 30 seconds total in order to not time out.

Ya, it depends on the upstream timeout for the process. I'm pretty sure to default for gunicorn is 30 seconds. After that the process will be killed, which kills the thread.

"Yeap, celery is really when i started to feel the pain of using python rather than a more enterprise-ready language&framework"

Say what?

The suckiness of celery doesn't come from its use of python. I'd be interested to know how you consider python not to be "enterprise ready".

I'm not sure what the suckiness of celery is.

We've been running it 4 years in production with no problems. It's really a nicely performant option with nice fail over options, and great debugging.

I understand the complexity in python of tying together Flask + Celery + DB + app server etc, but what would you use instead (your comment seems to imply there's a language/framework that has the aforementioned things bundles/more integrated)?

In terms of languages that facilitate doing work concurrently or in parallel with the HTTP request/response cycle:

  - Java / Scala / Clojure
  - Go
  - Erlang (add Elixir if you are comfortable with its maturity)

Not meant to be an exhaustive list of languages. Celery is great but there are plenty of situations where chucking units of work into a job queue or running a cron job feels like an incomplete solution.

These for the most part don't help you persist/offload/schedule asynchronous jobs off the main process. If your process dies or gets restarted, the async job dies with it. Celery persists, queues, schedules and distributes jobs for you, it's different.

For something more similar to Go/Java/Scala/Clojure/whatever, you can use Python's gevent or asyncio.

Implementing a task queue is not that hard in Django's management commands. See https://docs.djangoproject.com/en/1.8/howto/custom-managemen... - it covers 90% of the what you'll need. When you start to scale out, you might need a different solution, but this will work and be easy to deploy for the mostly everything.

As an example - I use this to create contracts with LaTeX asynchronously. The contract is created and then sent to the user by email. A management command runs every 5 minutes to check if there are any contracts that need to be generated.

the .net ecosystem is the kind of thing i was thinking about, but i'm pretty sure java has the same kind of solution.

as for the other persons answering me: i'm not only talking about long request. think about something like scheduling emailings or reports or ios notification , periodically or based on some db state change.

not something you'd want to handle with just an async framework.

the downvote made me look for a more precise example , so look at quartz for a robust enterprise grade lib for jobs and task management ( and its .net alternative).

i suppose celery is, on the paper, comparable in terms of fearures, yet the one time i had to use it made me feel like playing with a brittle toy, and in the end, i had to use a "recycle on every jobs" setting to prevent memory leaks from crashing everything. not that i believe celery is leaking memory, but that tying all the pieces together is a real pita.

> celery is really when i started to feel the pain of using python rather than a more enterprise-ready

The hendrix project - and its crosstown_traffic API specifically, is designed to alleviate this pain and remind that python (well, Twisted) is absolutely an enterprise-ready asynchronous network solution.

http://hendrix.readthedocs.org/

Reply on news.ycombinator.com