Hacker News Reader

Comment by vtuulos | Hacker News Reader

We have been using this architecture for containerized batch processing for more than a year:

http://tech.adroll.com/blog/data/2015/09/22/data-pipelines-d...

It is very convenient to be able to define any jobs as Docker containers, using the best language for the task, and define dependencies between their inputs and output explicitly, not being tied to any particular paradigm like MapReduce.

I am more than happy to let AWS handle plumbing for this instead of having to maintain and operate our custom solution.

View on HN

AdRoll has done some very cool stuff using containers for data processing. Afaik they were some of the first people to use this approach, and I think it's worked out quite well for them and given them quite an edge over their competitors.

If you're interested in building this type of infrastructure for yourself, I'd recommend taking a look at http://github.com/pachyderm/pachyderm. I'm one of the founders so this is a somewhat shameless plug, but Pachyderm is designed to get people most of the way to what AdRoll has built without having to build as much of the "plumbing" from scratch. AdRoll was some of our earliest inspiration for the design of Pachyderm.

Thanks! Pachyderm's version control over datasets is a killer feature.

By chance, have you tried https://gc3-uzh-ch.github.io/elasticluster/ ?

Also, have you looked at Airflow?

Not comparing them, just curious.

First time I hear about Elasticluster. Regarding cluster provisioning, auto-scaling is critical for cost and convenience. We managed to do this with a simple job queue that outputs a CloudWatch metric that in turn controls an auto-scaling group of spot instances. Works like a charm but AWS Batch should make this redundant.

We took a look at Airflow, which looks nice, but we haven't had a compelling reason not to use Luigi.

We've looked at Airflow. Two main things why we went with Luigi are

* time-based and dependency-based scheduling do not mix well in our experience, if you try to do both at the same time you end up with a very complicated execution model. Luigi is much simpler, it is basically a makefile. To handle time one can just add dummy dependencies where "exists()" function returns true if now()>X, and/or just run the whole thing by cron.

* We heavily use dynamic dependencies in Luigi, where one job decides how many childs to spawn and what parameters to use at run time. Makes it easy to do e.g. mapreduce style jobs purely in Luigi.

Reply on news.ycombinator.com