What does an Apache Airflow consultant do?
An Apache Airflow consultant diagnoses and fixes the Airflow setup your pipelines run on: stuck or slow tasks, failing DAGs, dependency conflicts and an overloaded scheduler or metadata database. They also plan and carry out upgrades, such as Airflow 2 to Airflow 3, build new DAGs, and add tests, CI and monitoring. DharmOps also stays on to support the setup after the work is done.
Is Airflow 2 still supported?
No. According to the Apache Airflow project, Airflow 2 entered limited maintenance on 22 October 2025 and reached end of life on 22 April 2026. End-of-life versions get no bug or security fixes. The last Airflow 2 release is 2.11.2. Airflow 3 is the supported major version.
How hard is it to upgrade from Airflow 2 to Airflow 3?
It depends on how your DAGs are written. The Airflow project calls Airflow 3 a major release with breaking changes. You need to be on Airflow 2.7 or later first. SubDAGs, SLAs, the v1 REST API and context keys such as execution_date are removed, and task code can no longer access the metadata database directly. The project's Ruff rules (AIR301 and AIR302) find most breaking changes in DAG code. Back up the metadata database and clean it with airflow db clean before migrating.
Can Amazon MWAA or Cloud Composer upgrade to Airflow 3 in place?
Not today. Amazon MWAA supports minor version upgrades only, not a new major version, so moving from Airflow 2 to Airflow 3 means creating a new environment. Google Cloud Composer 3 lists in-place upgrades from Airflow 2 to Airflow 3 as not yet supported and documents a side-by-side migration. In both cases the old and new environments run in parallel while DAGs move over.
Why are my Airflow tasks stuck in queued?
The common causes are: no spare worker capacity, pool or concurrency limits reached, a scheduler slowed down by DAG files that take too long to parse, zombie tasks holding slots, or a slow metadata database. Each has a different fix, such as more workers, adjusted pools, moving heavy code out of the DAG file's top level, or metadata database cleanup and tuning. The diagnostic finds which one applies to you.
Which Airflow executor should we use?
LocalExecutor runs tasks on the scheduler's machine and suits small setups. CeleryExecutor keeps a pool of workers running, which suits steady, high-volume workloads. KubernetesExecutor starts a pod for each task, which isolates tasks and their dependencies but adds startup time. Airflow can also run more than one executor at once. In the 2025 Airflow survey, 47.7% of teams used KubernetesExecutor and 37.3% used CeleryExecutor. The choice follows your workload, not a default.
Should we self-host Airflow or use MWAA, Cloud Composer or Astronomer?
Managed services remove the work of running the scheduler, database and upgrades, but limit configuration and can lag behind Airflow releases. Self-hosting on VMs or Kubernetes gives full control and can cost less at scale, but someone has to run it. In the 2025 Airflow survey, 15.5% of teams used a managed service, and the rest ran Airflow on cloud VMs, on-premises or on Kubernetes. We recommend one after the diagnostic, based on your team, workload and cost.
Should we move from Airflow to Dagster or another orchestrator?
Usually not as a first step. Many Airflow problems come from how it is set up and run, not from the tool. If the diagnostic shows a different orchestrator fits better, we say so and explain the cost of moving. Our Airflow vs. Dagster comparison covers the main differences.
How long does an Airflow 3 migration take?
It depends on the number of DAGs, how many use removed features, custom operators and plugins, whether tests exist, and whether you're on MWAA or Cloud Composer, which need a new environment. The diagnostic gives you a scoped plan with a timeline before you commit. Moving DAGs in batches means the most critical ones can move first.
How is Apache Airflow consulting priced?
Scope is quoted after the diagnostic, once we have looked at your setup, and before you commit. Cost depends on the number of DAGs, the version you're on, the hosting model, how much custom code there is, compliance requirements and whether support continues afterward.
What happens after the work is done?
The same engineers stay on. They fix breaks, handle planned upgrades, plan capacity as DAG count grows, and handle small changes as requests. The runbooks and documentation stay current, so your Airflow doesn't depend on one person.