Comment by adamcharnock | Hacker News Reader

Comment by adamcharnock | Hacker News Reader

Oh wow, I was so close to embarking on a Python task queue the other day because Celery was driving me nuts. I just wanted various apps/processes to be able to fire and listen for events, something that AMQP with a topic exchange seemed great for[1]. The thing that really tripped me up was the overlap in terminology between the AMQP protocol and Celery. Sometimes a queue means a queue, but sometimes it means an exchange. How do task names and routing keys interact? Why does the celery worker die when you listen to more than 3 queues on python 3.6 [2]?

Don't get me wrong, for basic stuff celery is great, albeit with a bit of a learning curve for the beginner. (And who knows, maybe I'm just missing something when it comes to more complex applications.) I also think the devs have done a great job and are super responsive. I think they also made a wise choice to chop features a little while back.

What I would really love to see (create?) is a cut-down python task queue which deals solely and specifically with some kind of AMQP (or similar) broker. Also with a task scheduler with a UI, that is something I'm loathed to lose.

Maybe this seems ridiculous, I get the impression that everyone is talking about microservices, ZeroMQ, service discovery, billions of messages per picosecond etc. I'm talking about medium size projects where a maybe a dozen apps just need to coordinate a bit. I feel like this middle ground lacks solutions.

Sorry for the ranty comment. I suspect I'm just trying to talk myself into creating this at some point.

[1] https://www.rabbitmq.com/tutorials/tutorial-five-python.html

[2] https://github.com/celery/kombu/issues/675

Couldn't agree more.

I wanted to create a task for my web app to run every 15 minutes and I didn't realize how complicated it was going to be. I'm using AWS beanstalk and I wanted the task to run on 0,15,30,45. The "simplest" way to do that was to add a worker box and create a cron job (it uses AWS SQS as an intermediary). I want to make sending email tasks as well but that means I'm going to have to move deeper into SQS integration and I don't want to be stuck with AWS. My only other option seems to be setting up and maintaining rabbitmq/celery and that looks like a nightmare. :/

RabbitMQ is super simple to setup and use (from my experience). I've been quite pleased with it.

That has been my experience too. I've setup a small RabbitMQ cluster on kubernetes and it was pretty straightforward. It also comes with a handy web UI which was more than I expected.

The hardest part seems to be answering the "Why isn't X receiving messages from Y?" question. Although it didn't help that I got '*' and '#' switched around in my head (seriously, '#' is the wildcard?!)

The other "simplest" way is to run the cron job on the web workers, and put something in the job to abort unless this host is the first web-worker instance. (Otherwise your jobs would run multiply on each instance.) That's what we're doing to kick off scheduled tasks for Django and it works well. It'd be very messy if the scheduled tasks created load because backend load in the web layer would really confuse autoscaling, but ours don't.

I'm also running little celery instances on each web worker, using Django as the backend. No SQS/Redis/RabbitMQ, no huge AWS lock in, no having to set up a backend worker layer. In general it's been fine for a year. You don't get good visibility into processing queue length etc but we usually have four bone-idle web workers so there is no queue length. If that might be a fit for your situation I could bang together a blog post or something.

Oh interesting. I also recently came across Huey and TaskTiger that seem to be doing similar things. Have you looked into those?

You could create a stepped lambda function that emits topic every minute with the date and time in it, something like YY.MM.DD.h.m and then register one function handler for multiple topics, I.e. .0, .15,.30, .45. I use this concept extensively in a few edge devices for which I don't want to build dedicated/centralized infrastructure. It works rather well. There seems to be some drift and the timing is not 100% accurate but is ok in my setup

I can't, or rather, I don't want to use lambda because my functions need access to both the DB and the open internet. The only way I can get that working is to add a NAT Gateway to my VPC which has a relatively high fixed monthly cost. :/

You emit a topic from the lambda function, your handlers can be hosted anywhere and can access anything they need. I think you misunderstood

>I wanted to create a task for my web app to run every 15 minutes

Cron would probably be a better bet then.

I love celery but I'm not a fan of celery beat.

Reply on news.ycombinator.com