Apache Airflow Consulting Services

    For teams whose reports depend on Airflow, and nobody has time to keep it healthy. Tasks sit in queued, DAGs break after an upgrade, and Airflow 2 stopped getting fixes on 22 April 2026. DharmOps finds the cause, fixes it or moves you to Airflow 3, and stays on to run and support it, self-hosted, on Kubernetes, MWAA, Cloud Composer or Astronomer.

    An engineer replies within 24 hours. Or describe the problem first, and see our Airflow case studies.

    Common Apache Airflow Problems We Fix

    Most teams don't call it an Airflow problem at first. They notice a late report or a failed job. Survey figures are from the Apache Airflow Survey 2025.

    Tasks sit in queued and nothing runs

    The DAG is scheduled, the tasks show as queued, and the report everyone waits on doesn't arrive. The cause is usually worker capacity, pool and concurrency limits, a scheduler slowed down by DAG files that take too long to parse, or zombie tasks holding slots.

    Still on Airflow 2, which no longer gets fixes

    Airflow 2 reached end of life on 22 April 2026, so it gets no bug or security fixes. In the 2025 Airflow survey, about 74% of 5,527 respondents were not yet on Airflow 3, and about a third were on 2.7 or older, below the minimum version for upgrading straight to 3.

    The upgrade breaks DAGs that worked yesterday

    Airflow 3 removes SubDAGs, SLAs, the v1 REST API and context keys such as execution_date and next_ds. Catchup is now off by default, and xcom_pull without task_ids only reads the current task. DAGs that used any of these fail or quietly behave differently.

    One new package breaks the whole environment

    Someone adds a Python library for one DAG, a dependency conflicts with an Airflow provider, and every DAG on that environment fails to import. Without pinned images, constraint files and a staging environment, production is where it gets found.

    The task succeeded, but the data is wrong

    Airflow reports whether a task ran, not whether the data it produced is complete or correct. In the 2025 survey, 25.5% of teams said they test DAGs only by hand or not at all, so wrong data reaches reports before anyone notices.

    Only one person can change Airflow

    The setup runs because one engineer knows its quirks. In the 2025 survey, 38.7% of teams named a lack of people to run Airflow as their biggest challenge, and 24.8% said upgrades and maintenance never get prioritized.

    Apache Airflow Consulting at DharmOps

    Apache Airflow consulting is diagnosing, fixing, upgrading and running the Airflow setup your data pipelines depend on: the scheduler, executor and workers, the metadata database, the DAGs, and the tests and monitoring around them. It is part of our data platform engineering work.

    Airflow Health Check & Diagnostic

    • Scheduler, executor, pool and concurrency review
    • DAG parse times and import errors
    • Queued, stuck and zombie task analysis
    • Metadata database size, load and cleanup
    • DAG code review with the Ruff AIR rules
    • Upgrade readiness for Airflow 3

    Airflow 3 Upgrade & Migration

    • Older 2.x releases to 2.11, then to Airflow 3
    • Removed features replaced: SubDAGs, SLAs, REST API v1
    • Context variables, catchup and timetable defaults updated
    • Direct metadata database access moved to the Task SDK or API
    • Metadata database backup and cleanup before migration
    • Side-by-side migration on Amazon MWAA and Cloud Composer

    Deployment & Infrastructure

    • Self-hosted on VMs, on-premises or Kubernetes with the official Helm chart
    • Amazon MWAA, Google Cloud Composer and Astronomer
    • Executor choice: Local, Celery, Kubernetes or more than one
    • PostgreSQL metadata database setup and tuning
    • Highly available schedulers and workers
    • DAG delivery through git-sync, images or DAG bundles

    DAG Development & Pipeline Engineering

    • New DAGs for ingestion, ETL and ELT
    • Cron jobs and scripts moved into Airflow
    • TaskFlow API, dynamic task mapping and asset-based scheduling
    • Deferrable operators for long waits
    • Custom operators, hooks and providers
    • dbt orchestration with Astronomer Cosmos

    Airflow vs. Dagster

    Testing, CI/CD & Dependencies

    • DAG import and integrity tests
    • Unit tests for tasks and custom operators
    • CI checks on every DAG change
    • Pinned images and Airflow constraint files
    • Development, staging and production environments
    • Infrastructure as code with Terraform or OpenTofu

    dbt for Data Engineering: A Practical Guide

    Monitoring & Data Checks

    • Airflow metrics through StatsD or OpenTelemetry to Prometheus and Grafana
    • Alerts on queue time, parse time, failures and missed runs
    • Freshness, volume and schema checks on the data each DAG produces
    • Lineage with OpenLineage
    • Log storage and retention

    Data observability tools compared

    Security & Access

    • Auth manager setup: Simple Auth, FAB or your identity provider
    • Role-based access to DAGs and connections
    • Secrets backends for connections and variables
    • Private networking for workers and the metadata database

    Operations & Ongoing Support

    • Fixes when a DAG or environment breaks
    • Planned minor and major upgrades
    • Capacity planning as DAG count grows
    • Runbooks for failures, backfills and recovery
    • Ongoing support from the engineers who did the work

    Who It's For

    Companies that already run Airflow in production, whose tasks get stuck, whose DAGs fail after changes, or who are still on Airflow 2 and haven't had the time to upgrade. Also teams moving cron jobs and scripts into Airflow for the first time.

    What It Solves

    Late reports, stuck and failing tasks, an unsupported Airflow version, upgrades that break DAGs, dependency conflicts, and knowledge that lives in one person's head.

    How It Starts

    With a diagnostic, not a tool or a quote. The diagnostic decides whether to fix the current setup, upgrade it, or change how Airflow is hosted.

    What It Isn't

    We don't sell an Airflow hosting platform or training courses. We work in your account. Dashboards and BI reports sit with Data Analytics, streaming pipelines with Real-Time Data Engineering, and spend reduction with Data Cost Optimization.

    Where It Sits in Your Stack

    Airflow decides when each pipeline runs and in what order: extract from source systems, load into the warehouse, run transformations, refresh reports. When a report is late, the cause is often a task that never started, not the report itself.

    What Changes When Airflow Is Fixed

    What DharmOps puts in place in an Apache Airflow consulting engagement and what it means for you
    What we put in placeWhat it means for you
    Scheduler, executor and pools sized to your workloadTasks start when they are scheduled, and reports arrive on time
    A supported Airflow 3 release, upgraded and tested side by sideBug and security fixes again, without a single risky cut over
    Pinned images, constraint files and a staging environmentA new package can't take production down
    Import tests and CI on every DAG changeA broken DAG is caught before it is deployed
    Data checks and alerts on queue time, failures and missed runsLate or wrong data is caught by an alert, not by someone reading a report
    Runbooks and documentation for the people who run itAirflow doesn't depend on one person

    What You Receive and What You Provide

    What You Receive

    Diagnostic output

    A written assessment of your Airflow setup: scheduler and executor health, DAG parse times, metadata database state, DAG code issues and upgrade readiness, with fixes ranked by impact and effort.

    Upgrade plan

    If you're on Airflow 2, a plan for reaching Airflow 3: which DAGs need changes, which features need replacing, and the order DAGs move in, agreed before any change is made.

    Target setup

    The Airflow version, executor, hosting model and environments that fit your workload, whether self-hosted, on Kubernetes, MWAA, Cloud Composer or Astronomer.

    Fixed and updated DAGs

    DAGs fixed, updated for Airflow 3 or newly built, with the reasoning behind each change written down.

    Tests and CI/CD

    Import tests, task tests and CI checks that run on every change, plus pinned images and dependency constraints.

    Monitoring and alerts

    Dashboards and alerts for queue time, parse time, failed tasks, missed runs and late or missing data.

    Runbooks

    Step-by-step procedures for failed tasks, stuck queues, backfills, upgrades and recovery, written so a junior engineer could follow them alone at 2am.

    Documentation and training

    How your Airflow is set up and why, and sessions for the people who write DAGs and run the environment.

    Ownership

    The DAG code, infrastructure definitions, credentials and documentation belong to you. Nothing proprietary sits in the middle.

    What You Provide

    1. Access to the Airflow UI, logs, the DAG repository and the hosting account, approved by you before we use it.
    2. People who own the DAGs: someone who knows what each critical DAG does and who depends on its output.
    3. A list of critical DAGs and the deadlines the business holds them to, such as a finance report by 8am.
    4. Your current setup: Airflow version, executor, where it runs, and any custom operators or plugins.
    5. Infrastructure and security contacts who can approve network, identity and environment changes.
    6. Early notice of regulated data such as health, financial or personal data, so handling is agreed before work starts.

    How an Engagement Works

    Six steps, in order. Each ends with something you can see, and nothing moves on until you've approved it.

    1. Discovery call
    2. Diagnose
    3. Prioritize
    4. Fix
    5. Stay

    Discovery Call

    We talk through what you're seeing in Airflow, what version and hosting you run, and what you need.

    • You receive: a clear answer on whether Airflow consulting is the right next step
    • You provide: a short description of the symptoms, your Airflow version and where it runs
    • Approval: both sides agree there is something to diagnose

    Diagnostic

    The step after the discovery call, and the gate before any scope is set. We review the scheduler, executor and pools, DAG parse times, the metadata database and the DAG code, and check what an Airflow 3 upgrade would touch.

    • You receive: a written plan ranking fixes by impact and effort, an upgrade plan if you're on Airflow 2, and the cost and risk behind each option
    • You provide: read access to Airflow, its logs and the DAG repository, and time with the people who own the DAGs
    • Approval: you approve the plan

    Scope is quoted after this step, not before

    Design

    We design the target setup: Airflow version, executor, hosting, environments, CI and monitoring.

    • You receive: a design document covering the target setup, the DAG changes and the order they move in
    • You provide: sign-off from your infrastructure and security contacts
    • Approval: you sign off on the design before any change starts

    Fix and build

    We fix the causes found in the diagnostic, update DAGs for Airflow 3 where needed, and build the new environment as code with tests and monitoring in from the start.

    • You receive: fixes and changes in your repository, regular demos and documentation written as we go
    • You provide: a named contact who answers questions promptly, and decisions on any DAG we flag
    • Approval: each change is accepted against criteria agreed in the design

    Cut over

    For upgrades, the new environment runs next to the old one. DAGs move in batches, each one is checked against the old run's output, and the old copy is paused, not deleted, until you approve.

    • You receive: matching outputs, resolved issues and a log of what was fixed
    • You provide: someone on your side to confirm the outputs match what the business expects
    • Approval: you approve retiring the old environment

    Ongoing support

    Support starts at cut over, with the same engineers who did the work. The first weeks get closer attention, because that is when production surprises appear.

    • You receive: fixes when a DAG or environment breaks, planned upgrades, capacity planning, small changes handled as requests, and runbooks that stay current
    • You provide: a named contact, and early notice when a source system or DAG is about to change
    • Approval: support scope is agreed in writing before support begins

    Continues after cut over, with the same engineers

    Timeline

    How Timing Works

    The diagnostic produces a timeline with milestones, each tied to something you can verify, not only a date. Dates move when a dependency on either side moves, and we tell you the same day it happens. The dependencies that matter most:

    • DAGs: how many DAGs there are and how many use features Airflow 3 removed.
    • Custom code: custom operators, plugins, and any task code that reads the metadata database directly.
    • Hosting: MWAA and Cloud Composer need a new environment for Airflow 3, while self-hosted setups can be upgraded in place.
    • Tests: whether DAG tests exist today, or need to be written first.
    • Access: how quickly access to Airflow, logs and the hosting account is approved.
    • Review: compliance or security review, if your data is regulated.

    Two Engagements Built on Apache Airflow

    Logistics: manual billing to Airflow-scheduled pipelines

    Situation
    A logistics operator's billing and reporting ran as a manual process.
    What we built
    PostgreSQL pipelines scheduled by Apache Airflow, with Superset dashboards, processing 2M+ records a day. Each part of the pipeline runs on the cadence it needs.
    Result
    Manual effort dropped sharply, and data issues that used to surface after days are caught close to real time.

    Read the logistics data platform case study

    E-commerce: one-off pipelines to reusable templates

    Situation
    Custom, one-off pipelines, with requests queuing at a central data team.
    What we did
    Reusable pipeline templates on a Snowflake platform, orchestrated with Airflow and Dagster, with automated validation and CI/CD.
    Result
    A repeatable platform where problems are caught earlier.

    Read the e-commerce data platform case study

    More on our work in all case studies.

    Why Teams Bring DharmOps In for Airflow

    We diagnose before we recommend.We look at your scheduler, DAGs and metadata database before suggesting an upgrade path, a new executor or a different tool. The written plan comes before any timeline or price.

    We know the database underneath.Airflow keeps its state in a metadata database, PostgreSQL for 77% of teams in the 2025 Airflow survey. Many slow schedulers and failed migrations start there. Our engineers work hands-on across PostgreSQL and MySQL, so those causes get found, not worked around.

    Wherever your Airflow runs.Self-hosted on VMs, in your own data centre, on Kubernetes, on Amazon MWAA, Google Cloud Composer or Astronomer. We work in your account, not on a platform we sell.

    You own it, and we stay.DAG code, infrastructure definitions and documentation are yours. After the work, DharmOps stays on to run and support it, so the next upgrade isn't another emergency.

    How We Handle Your Systems

    Airflow holds connections and credentials to many of your systems. Giving an outside team access to it is a serious decision. These are the rules we work by.

    • NDA before any access. Signed before we see anything.
    • Least access. We ask only for what the current step needs.
    • Approval before access. You approve each access level before we use it.
    • Your data stays where it lives. We work inside your environment, in the cloud or on-premises.
    • Deletion we verify. When work ends, anything we held is deleted and the deletion is confirmed.
    • Written incident procedure. If something goes wrong, there is a documented process, not improvisation.

    Apache Airflow Questions Teams Ask Before Booking

    What does an Apache Airflow consultant do?

    An Apache Airflow consultant diagnoses and fixes the Airflow setup your pipelines run on: stuck or slow tasks, failing DAGs, dependency conflicts and an overloaded scheduler or metadata database. They also plan and carry out upgrades, such as Airflow 2 to Airflow 3, build new DAGs, and add tests, CI and monitoring. DharmOps also stays on to support the setup after the work is done.

    Is Airflow 2 still supported?

    No. According to the Apache Airflow project, Airflow 2 entered limited maintenance on 22 October 2025 and reached end of life on 22 April 2026. End-of-life versions get no bug or security fixes. The last Airflow 2 release is 2.11.2. Airflow 3 is the supported major version.

    How hard is it to upgrade from Airflow 2 to Airflow 3?

    It depends on how your DAGs are written. The Airflow project calls Airflow 3 a major release with breaking changes. You need to be on Airflow 2.7 or later first. SubDAGs, SLAs, the v1 REST API and context keys such as execution_date are removed, and task code can no longer access the metadata database directly. The project's Ruff rules (AIR301 and AIR302) find most breaking changes in DAG code. Back up the metadata database and clean it with airflow db clean before migrating.

    Can Amazon MWAA or Cloud Composer upgrade to Airflow 3 in place?

    Not today. Amazon MWAA supports minor version upgrades only, not a new major version, so moving from Airflow 2 to Airflow 3 means creating a new environment. Google Cloud Composer 3 lists in-place upgrades from Airflow 2 to Airflow 3 as not yet supported and documents a side-by-side migration. In both cases the old and new environments run in parallel while DAGs move over.

    Why are my Airflow tasks stuck in queued?

    The common causes are: no spare worker capacity, pool or concurrency limits reached, a scheduler slowed down by DAG files that take too long to parse, zombie tasks holding slots, or a slow metadata database. Each has a different fix, such as more workers, adjusted pools, moving heavy code out of the DAG file's top level, or metadata database cleanup and tuning. The diagnostic finds which one applies to you.

    Which Airflow executor should we use?

    LocalExecutor runs tasks on the scheduler's machine and suits small setups. CeleryExecutor keeps a pool of workers running, which suits steady, high-volume workloads. KubernetesExecutor starts a pod for each task, which isolates tasks and their dependencies but adds startup time. Airflow can also run more than one executor at once. In the 2025 Airflow survey, 47.7% of teams used KubernetesExecutor and 37.3% used CeleryExecutor. The choice follows your workload, not a default.

    Should we self-host Airflow or use MWAA, Cloud Composer or Astronomer?

    Managed services remove the work of running the scheduler, database and upgrades, but limit configuration and can lag behind Airflow releases. Self-hosting on VMs or Kubernetes gives full control and can cost less at scale, but someone has to run it. In the 2025 Airflow survey, 15.5% of teams used a managed service, and the rest ran Airflow on cloud VMs, on-premises or on Kubernetes. We recommend one after the diagnostic, based on your team, workload and cost.

    Should we move from Airflow to Dagster or another orchestrator?

    Usually not as a first step. Many Airflow problems come from how it is set up and run, not from the tool. If the diagnostic shows a different orchestrator fits better, we say so and explain the cost of moving. Our Airflow vs. Dagster comparison covers the main differences.

    How long does an Airflow 3 migration take?

    It depends on the number of DAGs, how many use removed features, custom operators and plugins, whether tests exist, and whether you're on MWAA or Cloud Composer, which need a new environment. The diagnostic gives you a scoped plan with a timeline before you commit. Moving DAGs in batches means the most critical ones can move first.

    How is Apache Airflow consulting priced?

    Scope is quoted after the diagnostic, once we have looked at your setup, and before you commit. Cost depends on the number of DAGs, the version you're on, the hosting model, how much custom code there is, compliance requirements and whether support continues afterward.

    What happens after the work is done?

    The same engineers stay on. They fix breaks, handle planned upgrades, plan capacity as DAG count grows, and handle small changes as requests. The runbooks and documentation stay current, so your Airflow doesn't depend on one person.

    Version dates and upgrade steps are from the Apache Airflow supported versions and upgrading to Airflow 3 pages.

    Data Platform Engineering Use Cases by Industry

    Every industry has source systems that don't talk to each other, often split across cloud and on-premises environments. Data platform engineering is the platform and pipeline work that turns them into one dataset your teams can query.

    Data Platform Engineering for Retail & eCommerce

    POS, storefront, and warehouse systems each hold part of the inventory picture. We build ELT pipelines into one warehouse so stock, sales, and demand sit in one place.

    Data Platform Engineering for SaaS & Software

    Product usage events have to reach reporting without a ticket to the data team. We build the ingestion, dbt models, and self-service golden paths that let product and finance teams ship their own datasets.

    Data Platform Engineering for Logistics

    Load events, warehouse scans, telematics, and EDI documents arrive in different formats. We land each source raw, normalize it in a silver layer, and publish business-ready views on top.

    Data Platform Engineering for Healthcare

    Claims, EHR extracts, and lab feeds use different codes and identifiers. We build pipelines that standardize them and enforce data contracts, so downstream reporting stops breaking when a feed changes.

    Data Platform Engineering for FinTech

    Ledger and payment data have to reconcile across systems. We build the pipelines with schema checks at each hop, so a changed field is caught at ingestion instead of in a finance report.

    Data Platform Engineering for Manufacturing

    MES, ERP, and sensor data live in separate systems on separate schedules. We integrate them into one platform, so plant and finance teams read the same numbers.

    Start With a Conversation About Your Airflow

    If your tasks get stuck, your DAGs break after changes, or you're still on Airflow 2, the first step is a 30-minute discovery call.

    Apache, Apache Airflow and Airflow are trademarks of The Apache Software Foundation.