This bit of code in worker.py makes me uncomfortable though:
# Deserialize the task
d_fun, d_args = dill.loads(data)
# Run the task
d_fun(*d_args)
Basically the function (as in the Python object that holds the function's code, etc) to run gets pulled out of redis along with its arguments, then it gets executed.If the entire system is trusted, this is a really cool way to let workers run code without needing to deploy it to them. Likewise, you wouldn't even need to restart the workers for a new version of the function to run - just start feeding the new code into the queue and the workers will run it. The more I think about this, the cooler it seems.
On the other hand, from the worker's perspective, this is basically a wide-open "feed me any code and I'll run it" thing.