Hacker News Reader

I like Argo and have used both solutions, but I'm not yet convinced that YAML based workflows are superior to Airflow's Python Code as workflows.

Agree that YAML is not really ideal for this case.

For other readers Argo does support 'Scripts' which means you can implement complex logic similar to Airflow if required but can be a bit clumsy: https://argoproj.github.io/argo/examples/#scripts-results

Additionally, I think we may see other layers on top of Argo in future that may abstract from the YAML similar to how Kubeflow uses Argo for the Kubeflow Pipelines capability: https://www.kubeflow.org/docs/components/pipelines/pipelines...

Have you tried Flyte - flyte.org?

Running something on Kubernetes does not make this something highly available. It makes this something randomly restarting.

Maybe.

In this case Argo uses the Kubernetes distributed state store (etcd or equivalent) and is a stateless service so can deal with failures quite well.

Agreed. We've had good luck running all sorts of jobs on Argo and wrote up some of our experiences at https://www.interline.io/blog/scaling-openstreetmap-data-wor...