denibertovic

Could be, wavelengths thing I mean. :)

I'm not aware of how ruby does it other then hearing about 2 most popular solutions that are used with rails.

As for Postgres's PUB/SUB I've replied to the reply that mentioned that and you can't really use that. Would be great if we cold update the broker driver and see if we could get support though.

No, you are not storing the job itself, just a flag that indicates that the job was not done/or even stored. What's more, there really are various ways to handle failure of storing a job on a queue, but in most cases it's just the absence of the desired effect that can be a giveaway. Ie. you know that the job either didn't finish or wasn't even sent to the queue if it's effect (ie. updating the user's friend list from facebook) was not done.

Also, take a look at #5 for a great monitoring stats solution.

Sorry to disappoint. Flower isn't about a real flower either. :P

Precisely. Tnx jaegerpicker.

I think you're missing the point. The Celery (or any task queue really) particulars are very important here, cause you don't want background workers hammering your database if they don't need to. Cause the workers wan't work with a AMQP implementation, which the database is not. It's like using a fork instead of a hammer, sure you might get a few nails but it's not the right tool for the job.

The systems that use these kinds of tools are usually not structured in a way that they need to wait for something in the database to be stored. By nature they are async tasks and they should be able to run whenever and return sometime in the future, and they will most likely produce some kind of result in the database, so there is no reason to store the job information itself in the database.

Jobs are usually not created as hooks after a database commit, so jobs being persisted with database transactions is not quite relevant and Celery has failure mechanism and ways to recover if it was not able to send a task to the broker (ie. RabbitMQ was down).

Redis and RabbitMQ do have a mechanism of persisting jobs onto disk as well so they don't get lost when the process is restarted. So there is no way that a job get's lost forever as you say, if you handle all these cases correctly.

One more thing, Python's database drivers don't work quite as you've described. Namely they don't (by design) make use of the autocommit feature of the database engine, rather they wrap every sql statement in a transaction, so either way each statement get's executed separately in it's own transaction. This would not guarantee, let's say a db record being added and the job being saved as well. You would have to use explicit atomic blocks (something a kin to what Django >= 1.6 has) to get both things or none to be persisted.

Doing non-blocking is tricky, and I'm not convinced that Python's solution are where I'd like them to be on this topic. Also keep in mind that a number of open TCP connections is also a finite number, so you can't really scale well with websockets that way, IMO. But again, it depends on your use case.

Anyone that's interested in Postgres's pub/sub might find this useful: https://denibertovic.com/talks/real-time-notifications/#/

Just slides though. Haven't gotten around to writing a post about it yet.

It is true that Postgres supports Pub/Sub but unfortunately the Celery broker driver does not take advantage of this. It would be great if we could get support for it. Nevertheless, just because it has pub/sub doesn't mean it's a full AMQP implementation. Also, there's the fact that most amqp solutions are in memory, wheres a database is on disk... also has it's costs.

Hmm, seems we're talking about 2 things here.

If your ajax request requires long task processing and requires you to wait for it than this is not a background task any more, it's done in one of the web server threads, and even if the thread outsources the task to another process it's still waiting on that proces to finish before returning the ajax response. This is bad.

I'm not entirely convinced about websocket solutions in Python yet, but I've been told flask-websockets is awesome. Nevertheless this doesn't solve the problem for you. Cause the request is just keeping an open line and waiting for a respone....blocking is bad.

The most simplest advise I would have is to have the ajax request trigger a background task and return immediately. The background task will then have some kind of side effect (ie. write some result to a database somewhere) which the ajax request can the look for with some kind of polling mechanism (on some other endpoint). Of course you can complicate this a lot, depending on your needs, but this seemed like the most straightforward solution.

More awesome tips. Thank you.

Noted, the wording is a bit contrived i give you that. I like Redis as well, it was mentioned in the comments here a few times. Good thinking about pointing out reloading btw. Tnx.

Ah no, you got that one perfectly I just didn't understand what was meant at first. Fixed. Thank you.

It's somewhat akin to a Django vs. Flask discussion really. But yes for more light weight stuff I too would recommend rq.

In general using an AMQP for the result storage is somewhat of a bad idea i think. But yes I agree about the ignoring results part seeing as most tasks I've seen in the wild don't return anything at all. Hence #6 in the post.

With container solutions like Docker and prebuilt images the setup part is kinda eliminated. Although I don't remember having any special configuration issues with RabbitMQ as well, it just works TM. That's always nice right? :)

Redis is still not an AMQP, but yes Redis's Pub/Sub works quite nicely. Out of all the brokers celery supports I'd recommend only RabbitMQ and Redis to people.

Awesome stuff, tnx for these. :)

Not really, that's just the name of the actual task itself ie. "def my_taskA(a, b, c)".

Ruby doesn't have functions as first class citizens. Python does.

I'm using docopt....It's really nice with it's automagicness. But for some stuff it's just not enough, too unflexible. I'm really excited about click.

The part that bothers me is that is's all shell scripts... Why not use a modern provisioning tool now-a-days?

This looks way more interesting to me: https://github.com/al3x/sovereign

This article really hits the nail on the head. In the comments, many people confuse layers of indirection with dependency injection. Those are 2 separate things, you can have dependency injection and not have 100 layers of indirection that confuse the hell out of people.

And while we're at it. Why the is there a term "Dependecy injection"... it's freaking argument passing to functions, and it's been around a really really long time. I'd expect this kind of crap from the JVM and other OO only worlds, but JS is more functional then it is OO, you have higher order functions...why the hell would you want to think in OO terms?

We have a SGI machine (wights like ton) in our shop. Can't remember the exact model but I remember how someone mentioned it's the exact model used for animation in the movie....we use it mostly as a chair XD

Why does every definition of new future super cool messaging system imply "immediate". I quite don't like that. Not everything needs to be a chat and immediate.

This is true, the article does not use python-oauth2 though. It shows how to do the flow manually...well using requests. :)

a link to an article that requires you to login to read it. Wonderful, just wonderful.

Actually encrypted does mean that. What you're thinking about is hashing. Hash functions are a one-way trip, and what is most commonly used for password storing.

Working from 9-5 includes lunch as well. The fact that 9-6 includes lunch, the fact that it needs to be mentioned, is ridiculous, you still spend 9 hours in or around the office.

How is 9-6 8 hours a day? They work 9 hours a day. As stated in the article.

I can relate to this. Very well put.