Hacker News Reader

GKE is amazing too, none of the other major clouds even come close.

I found Airflow for a small to medium project to be a nightmare on its own; I'm sure its a powerful framework, but plain Python ended up meeting our needs more than adequately. I was self-hosting it while kicking the tires and found it to be a lot to admin. I always thought I would look at Cloud Composer if I wanted to revisit Airflow. Were your issues with Airflow or Cloud Composer? Agreed BQ IS great.

My issues have been with cloud-composer. I like Airflow.

Would love to know the pain points so we can make it better - Composer, Airflow, or both.

Triggering a DAG externally fails consistently for 1% of requests, and 100% of the time when the web server API goes down, which is too often (502). I understand that in the realm of Airflow, triggering a DAG externally is perhaps a bit of an anti-pattern, but it should not have been documented then:

https://cloud.google.com/composer/docs/how-to/using/triggeri...

I have cases open with the composer team, and I am upgrading the cluster again this weekend, but I have lost hope at this point; I am designing a solution to stop using external triggers.