airflow

Developed originally at Airbnb in 2014 and donated to the Apache Software Foundation, Airflow has become the de facto standard for data pipeline orchestration in the data engineering ecosystemused by thousands of organizations to manage ETL workflows, ML training pipelines, data quality checks, and any complex multi-step process that needs scheduling, dependency management, and failure handling.

Airflow's DAG-based model lets engineers define pipeline logic in Python, specifying task dependencies through graph relationships rather than imperative scheduling code.

The platform handles the operational concerns: scheduling DAG runs on cron-like schedules or event triggers, retrying failed tasks with configurable policies, alerting on failures, and providing a web UI where operators can inspect task logs, re-run failed pipeline segments, and monitor queue depth across the scheduler.

Operators (Airflow's abstraction for task types) cover integrations with every major data platformBigQuery, Snowflake, Spark, AWS S3, Kubernetes, dbt, and hundreds more.

Data engineering teams building ETL pipelines, ML platforms teams orchestrating model training and deployment workflows, and analytics engineers scheduling data transformation runs use Airflow as their pipeline orchestration layer.

The Python-native DAG definition makes it approachable for teams already working in Python without requiring a separate DSL or configuration language.

While newer orchestration tools (Prefect, Dagster, Temporal) have addressed some of Airflow's limitations around dynamic workflows and developer experience, Airflow's maturity, massive ecosystem of operators, and widespread adoption make it the default choice for organizations standardizing on an orchestration platform.