Understanding Airflow DAG and Task Concurrency on Google Cloud Composer

At scale it's probably to ensure your environment is capable of handling as many DAGs as possible. also as a data engineer, not knowing why a task isn't running can be frustrating. so this might help those understand and resolve

This article is blog post describing concurrency on Google's managed service version of Airflow, Compose, on Google's blog.

While I don't have hands on with that, I have some hands on experience with the AWS managed service equivalent, Managed Workflows for Apache Airflow (MWAA) and its slightly similar documentation and/or challenges. We migrated 300+ jobs from a different but roughly similar orchestration tool to Airflow DAGs (and some jobs had as many as 50 parallel tasks for extracting data).

If my experience with MWAA is any guide, there are enough minor differences between a managed-service Airflow vs self-managed Airflow that -- particularly for someone new to Airflow and concerned with scaling or performance issues for which SaaS platforms give you a narrower set of lower-effort knobs -- its probably worth having a tutorial to help people out so they don't have to mentally understand the full Airflow documentation and which bits of it do or don't, might or might not be able to be applied to them. (Managed Airflow does lock down some things which it has to for security/multi-tenancy reasons, and then has some differences to support "autoscaling" beyond what you get with self-hosted single-server Airflow. I didn't really see any aspects of the differences that I considered to be vendor attempts at vendor lockin, just needed differences for a SaaS product.) That sort of coherent-just-for-Google-Compose viewpoint seems to be what this blog post seems to focus on.

I feel like we didn't get that coherent documentation from AWS's Airflow SaaS. It was mediocre (but passable) for a team new to Airflow. We had to learn when to read one set of documentation, when the other, and how to infer and map across the two to get a wholistic view of Airflow sufficient to solve our problems. It actually is nice under such situations if the cloud vendor provides coherent complete documentation even if it recapitulates the core project documentation or is somewhat redundant.