Hacker News Reader

(Sorry if it's a little off topic) We were close to adopt Airflow in our company but we were let down by a detail: the scheduler isn't natively in a high availability mode. There was an article from Clairvoyant about how to make it HA but it didn't look safe at all. That was a serious issue for us, and at the end we went for NiFi. Have anyone had this problem?

View on HN

Airflow maintainer here

If you have idempotent tasks, which is a best practice, it is possible to use Airflow in HA even in active/active. It might occasionally schedule a task twice, which should be caught but in any case mitigated by idempotency.

If you are looking for more Enterprise support you can reach out to Astronomer (disclaimer: I'm a advisor to them) or use a Google cloud hosted version (Cloud Composer). Both are great products.

Use K8s to schedule the work.

And make sure you hire two engineers to keep it working.

Reply on news.ycombinator.com