Dagster Consulting Services

    For teams whose pipelines run on Dagster, or are moving there from Airflow. Code locations go unreachable, backfills crawl, and the move off Airflow stalls with both tools still running. DharmOps finds the cause, fixes it or finishes the migration, and stays on to run and support it, self-hosted, on Kubernetes or on Dagster+.

    An engineer replies within 24 hours. Or describe the problem first, and see our Dagster case study.

    Common Dagster Problems We Fix

    Most teams don't call it a Dagster problem at first. They notice a stale table or a backfill that never finishes. Limits and defaults are from the Dagster documentation.

    The move off Airflow stalled halfway

    Airflow 2 stopped getting fixes on 22 April 2026, so the team started moving to Dagster. Months later both orchestrators still run, some pipelines live in each, and nobody is sure which one owns a given table. Dagster's Airlift toolkit has four stages, Peer, Observe, Migrate and Decommission, and most stalled moves sit between the second and third.

    Code locations keep going unreachable

    The UI shows DagsterUserCodeUnreachableError or a gRPC UNAVAILABLE error, and schedules and sensors stop until someone restarts the server. Each code location runs a single server, and when Dagster starts it as a subprocess it waits 180 seconds by default. Slow imports, heavy definitions and memory limits push past that.

    Runs and backfills get slower as partitions grow

    A daily partition per customer or region adds up fast. Dagster recommends 100,000 partitions or fewer per asset, and by default a backfill over N partitions launches N separate runs. On Kubernetes, each of those runs is its own job, with its own startup time.

    The instance database keeps growing

    Dagster stores run history, event logs and schedule and sensor ticks in its instance database, usually PostgreSQL. By default it keeps every tick type except skipped sensor ticks indefinitely, and old event logs are only removed if you write the cleanup job yourself. The UI slows down and the database bill climbs.

    Deprecated APIs pile up between upgrades

    Dagster ships new releases often, and the APIs move with them. The original FreshnessPolicy has been deprecated since version 1.6 and is now LegacyFreshnessPolicy, and freshness policies replaced freshness checks in 1.12. Code written two years ago still runs, until the upgrade that removes what it depends on.

    Nobody decided where Dagster should run

    Open-source Dagster needs a webserver, a daemon and a code location server, plus a database you run. Dagster+ Serverless runs your code in Dagster's environment, with a 4,500 step-minute daily limit. Dagster+ Hybrid runs the control plane for you while your code runs in your own cloud. Each choice has a different cost and a different security review.

    Dagster Consulting at DharmOps

    Dagster consulting is diagnosing, fixing, migrating to and running the Dagster setup your data pipelines depend on: code locations, the daemon and run launcher, the instance database, the assets and partitions, and the tests and monitoring around them. It is part of our data platform engineering work.

    Dagster Health Check & Diagnostic

    • Code location load times and gRPC errors
    • Daemon, run queue and run launcher review
    • Instance database size, load and cleanup
    • Partition counts and backfill policies
    • Asset graph and dependency review
    • Deprecated API use before the next upgrade

    Airflow to Dagster Migration

    • Inventory of which DAGs are asset-shaped and which are task sequences
    • Airlift stages: Peer, Observe, Migrate, Decommission
    • Airflow DAGs observed in Dagster before anything moves
    • Single tasks or whole DAGs moved, most critical first
    • Outputs checked against the Airflow run
    • Airflow retired only after you approve

    Airflow vs. Dagster

    Deployment & Infrastructure

    • Self-hosted on Kubernetes with the Dagster Helm chart, or on Docker
    • Dagster+ Serverless or Dagster+ Hybrid
    • Hybrid agents on Kubernetes, Amazon ECS, Docker or Azure
    • PostgreSQL or MySQL instance storage setup and tuning
    • Run launchers: a Docker container or Kubernetes job per run
    • Compute logs stored in S3, GCS or Azure Blob Storage

    Asset & Pipeline Engineering

    • Software-defined assets for ingestion, ETL and ELT
    • Time, static, dynamic and multi-dimensional partitions
    • Declarative Automation, schedules and sensors
    • Resources and I/O managers for each environment
    • Cron jobs and scripts moved into Dagster
    • Projects set up with Components and the dg CLI

    dbt + Dagster

    • dbt models loaded as Dagster assets with dagster-dbt
    • The dbt project component, configured in YAML
    • dbt tests run as Dagster asset checks
    • Subsets of dbt models run by selection
    • dbt runs gated on upstream data being ready

    dbt for Data Engineering: A Practical Guide

    Testing, CI/CD & Branch Deployments

    • Unit tests for assets with mocked resources
    • Definition load checks on every change
    • Branch deployments on Dagster+
    • Pinned images and dependencies
    • Development, staging and production environments
    • Infrastructure as code with Terraform or OpenTofu

    Data Quality & Monitoring

    • Asset checks, including checks that block downstream assets
    • Freshness policies on the assets the business reads
    • Alerts on failed runs, failed checks and late data
    • Daemon, run queue and code location health
    • Log storage and retention

    Data observability tools compared

    Operations & Ongoing Support

    • Fixes when an asset, run or deployment breaks
    • Planned Dagster upgrades and API migrations
    • Capacity planning as assets and partitions grow
    • Runbooks for failures, backfills and recovery
    • Ongoing support from the engineers who did the work

    Who It's For

    Companies that already run Dagster in production and see unreachable code locations, slow backfills or a growing database. Also teams moving from Airflow to Dagster, whether they are planning the move or stuck partway through.

    What It Solves

    Stale tables, stopped schedules and sensors, slow runs and backfills, two orchestrators running at once, deprecated code that blocks upgrades, and knowledge that lives in one person's head.

    How It Starts

    With a diagnostic, not a tool or a quote. The diagnostic decides whether to fix the current setup, finish a migration, keep some pipelines on Airflow, or change where Dagster runs.

    What It Isn't

    We don't resell Dagster+ or sell training courses. We work in your account. Dashboards and BI reports sit with Data Analytics, streaming pipelines with Real-Time Data Engineering, and spend reduction with Data Cost Optimization.

    Where It Sits in Your Stack

    Dagster decides when each asset is built and checks it after: raw data from source systems, tables in the warehouse, dbt models, the datasets reports read. When a report shows stale numbers, the cause is often an asset that never refreshed, not the report itself.

    What Changes When Dagster Is Fixed

    What DharmOps puts in place in a Dagster consulting engagement and what it means for you
    What we put in placeWhat it means for you
    One orchestrator owning each pipeline, with Airflow retired on a planNo more guessing which tool produced a table, and one system to pay for and run
    Code locations that load fast and restart on their ownSchedules and sensors keep running without someone watching the UI
    Partitions and backfill policies sized to your dataBackfills run as fewer, larger runs where that fits, and the UI stays usable
    Instance database cleanup and tuningRun history stays useful instead of slowing everything down
    Asset checks and freshness policies on critical assetsStale or wrong data is caught by an alert, not by someone reading a report
    Current APIs, tests, CI and runbooksUpgrades are planned work, and Dagster doesn't depend on one person

    What You Receive and What You Provide

    What You Receive

    Diagnostic output

    A written assessment of your Dagster setup: code locations, daemon and run launcher, instance database, partitions, asset graph and deprecated API use, with fixes ranked by impact and effort.

    Migration plan

    If you're moving from Airflow, which DAGs to move, which to rebuild as assets and which to leave, the Airlift stage each one is in, and the order they move, agreed before any change is made.

    Target setup

    Where Dagster runs and how: self-hosted on Kubernetes or Docker, Dagster+ Serverless or Dagster+ Hybrid, with the environments and storage that fit your workload.

    Fixed and built assets

    Assets, partitions and automation fixed, updated to current APIs or newly built, with the reasoning behind each change written down.

    Tests and CI/CD

    Asset tests, definition load checks and CI that runs on every change, plus branch deployments where you use Dagster+.

    Monitoring and alerts

    Asset checks, freshness policies and alerts for failed runs, failed checks, unreachable code locations and late data.

    Runbooks

    Step-by-step procedures for failed runs, unreachable code locations, backfills, upgrades and recovery, written so a junior engineer could follow them alone at 2am.

    Documentation and training

    How your Dagster is set up and why, and sessions for the people who build assets and run the deployment.

    Ownership

    The asset code, infrastructure definitions, credentials and documentation belong to you. Nothing proprietary sits in the middle.

    What You Provide

    1. Access to the Dagster UI, logs, the code repository and the hosting account, approved by you before we use it.
    2. People who own the pipelines: someone who knows what each critical asset or DAG produces and who depends on it.
    3. A list of critical assets and the deadlines the business holds them to, such as a finance table ready by 8am.
    4. Your current setup: Dagster version, OSS or Dagster+, where it runs, and, if you're migrating, the Airflow version and DAG count.
    5. Infrastructure and security contacts who can approve network, identity and environment changes.
    6. Early notice of regulated data such as health, financial or personal data, so handling is agreed before work starts.

    How an Engagement Works

    Six steps, in order. Each ends with something you can see, and nothing moves on until you've approved it.

    1. Discovery call
    2. Diagnose
    3. Prioritize
    4. Fix
    5. Stay

    Discovery Call

    We talk through what you're seeing in Dagster or in the move from Airflow, where it runs, and what you need.

    • You receive: a clear answer on whether Dagster consulting is the right next step
    • You provide: a short description of the symptoms, your Dagster version and where it runs
    • Approval: both sides agree there is something to diagnose

    Diagnostic

    The step after the discovery call, and the gate before any scope is set. We review code locations, the daemon and run launcher, the instance database, partitions and the asset graph, and, if you're migrating, which Airflow DAGs should move and how.

    • You receive: a written plan ranking fixes by impact and effort, a migration plan if you're moving from Airflow, and the cost and risk behind each option
    • You provide: read access to Dagster, its logs and the code repository, and time with the people who own the pipelines
    • Approval: you approve the plan

    Scope is quoted after this step, not before

    Design

    We design the target setup: where Dagster runs, code locations, storage, environments, CI and monitoring.

    • You receive: a design document covering the target setup, the asset changes and the order pipelines move in
    • You provide: sign-off from your infrastructure and security contacts
    • Approval: you sign off on the design before any change starts

    Fix and build

    We fix the causes found in the diagnostic, rebuild DAGs as assets where that fits, and set up the deployment as code with tests and monitoring in from the start.

    • You receive: fixes and changes in your repository, regular demos and documentation written as we go
    • You provide: a named contact who answers questions promptly, and decisions on any pipeline we flag
    • Approval: each change is accepted against criteria agreed in the design

    Cut over

    For migrations, Dagster observes the live Airflow DAGs first. Pipelines then move in batches, each one is checked against the Airflow run's output, and the Airflow copy is paused, not deleted, until you approve.

    • You receive: matching outputs, resolved issues and a log of what was fixed
    • You provide: someone on your side to confirm the outputs match what the business expects
    • Approval: you approve retiring Airflow or the old deployment

    Ongoing support

    Support starts at cut over, with the same engineers who did the work. The first weeks get closer attention, because that is when production surprises appear.

    • You receive: fixes when an asset or deployment breaks, planned upgrades, capacity planning, small changes handled as requests, and runbooks that stay current
    • You provide: a named contact, and early notice when a source system or pipeline is about to change
    • Approval: support scope is agreed in writing before support begins

    Continues after cut over, with the same engineers

    Timeline

    How Timing Works

    The diagnostic produces a timeline with milestones, each tied to something you can verify, not only a date. Dates move when a dependency on either side moves, and we tell you the same day it happens. The dependencies that matter most:

    • Pipelines: how many assets or Airflow DAGs there are, and how many need rebuilding as assets rather than moving as they are.
    • Custom code: custom resources, I/O managers, sensors and any code that relies on deprecated Dagster APIs.
    • Hosting: open-source Dagster on your own infrastructure, Dagster+ Serverless or Dagster+ Hybrid, and whether that changes as part of the work.
    • Tests: whether asset tests exist today, or need to be written first.
    • Access: how quickly access to Dagster, Airflow, logs and the hosting account is approved.
    • Review: compliance or security review, if your data is regulated or Dagster+ is a new data processor for you.

    An Engagement Built on Dagster

    E-commerce: one-off pipelines to reusable templates

    Situation
    Custom, one-off pipelines, with requests queuing at a central data team.
    What we did
    Reusable pipeline templates on a Snowflake platform, orchestrated with Dagster and Airflow, with automated validation and CI/CD.
    Result
    A repeatable platform where problems are caught earlier.

    Read the e-commerce data platform case study

    More on our work in all case studies.

    Why Teams Bring DharmOps In for Dagster

    We diagnose before we recommend.We look at your code locations, instance database and asset graph before suggesting a migration, a new deployment model or a different tool. The written plan comes before any timeline or price.

    We run Airflow too.Most Dagster projects start from Airflow. We also upgrade and support Airflow, so we can tell you which pipelines are worth moving and which are cheaper to fix where they are.

    We know the database underneath.Dagster keeps run history, event logs and ticks in its instance database, usually PostgreSQL or MySQL. Many slow UIs and stalled daemons start there. Our engineers work hands-on across both, so those causes get found, not worked around.

    You own it, and we stay.Asset code, infrastructure definitions and documentation are yours, whether you run open-source Dagster or Dagster+. After the work, DharmOps stays on to run and support it.

    How We Handle Your Systems

    Dagster holds resources and credentials for many of your systems. Giving an outside team access to it is a serious decision. These are the rules we work by.

    • NDA before any access. Signed before we see anything.
    • Least access. We ask only for what the current step needs.
    • Approval before access. You approve each access level before we use it.
    • Your data stays where it lives. We work inside your environment, in the cloud or on-premises.
    • Deletion we verify. When work ends, anything we held is deleted and the deletion is confirmed.
    • Written incident procedure. If something goes wrong, there is a documented process, not improvisation.

    Dagster Questions Teams Ask Before Booking

    What does a Dagster consultant do?

    A Dagster consultant diagnoses and fixes the Dagster setup your data pipelines run on: unreachable code locations, slow runs and backfills, a growing instance database and deprecated APIs. They also move pipelines from Airflow to Dagster, choose and set up where Dagster runs, build assets, and add tests, CI and monitoring. DharmOps also stays on to support the setup after the work is done.

    Is Dagster free for commercial use?

    Yes. Open-source Dagster is released under the Apache License 2.0, which allows commercial use. You pay for the infrastructure you run it on. Dagster+ is the paid, managed version from Dagster Labs, available as Serverless or Hybrid.

    Why use Dagster over Airflow?

    Airflow schedules tasks and reports whether they ran. Dagster is built around assets, the tables and files your pipelines produce, and reports whether each one is up to date. That gives Dagster built-in asset checks, lineage and per-model dbt integration. If your Airflow setup works and the problems are in how it is run, fixing it is often cheaper than moving.

    How do we migrate from Airflow to Dagster?

    Dagster's Airlift toolkit splits a migration into four stages: Peer, which shows your Airflow instance inside Dagster; Observe, which maps Airflow DAGs to assets; Migrate, which moves individual tasks or whole DAGs to Dagster; and Decommission, which removes the Airflow code. You don't have to finish every stage, and Dagster connects to live Airflow through its REST API, so both run while pipelines move.

    Why is Dagster so slow?

    The common causes are: code locations that take too long to load, assets with more partitions than Dagster recommends (100,000 or fewer per asset), backfills that launch one run per partition, startup time for a new container or Kubernetes job per run, and an instance database that has never been cleaned up. Each has a different fix, and the diagnostic finds which one applies to you.

    What is the difference between Dagster+ and open-source Dagster?

    With open-source Dagster you run everything yourself: the webserver, the daemon, code location servers and the instance database. Dagster+ runs the control plane for you. In Dagster+ Serverless your code also runs in Dagster's environment, with limits such as 4 vCPU and 16 GB of RAM per run. In Dagster+ Hybrid your code runs in your own environment through an agent on Kubernetes, Amazon ECS, Docker or Azure.

    Should we use a Dagster sensor or a schedule?

    A schedule runs on a fixed time, such as every day at 6am. A sensor checks for an event, such as a new file landing, and starts a run when it finds one. Dagster also has Declarative Automation, which starts runs based on the status of assets and their dependencies, with conditions such as on_cron, eager and on_missing. For asset-based pipelines it often replaces hand-written sensors.

    Does the Prefect acquisition change anything for Dagster users?

    Prefect announced on 13 July 2026 that it is acquiring Dagster Labs. Dagster's customer FAQ says the announcement doesn't change how your Dagster deployment runs, and Dagster and Dagster+ keep their names. Open-source Dagster remains under the Apache 2.0 license. For a team already running Dagster, it doesn't force a decision.

    Is Dagster an ETL tool?

    Not on its own. Dagster is an orchestrator: it decides when your ETL and ELT code runs, tracks the assets it produces, and checks them. The extraction and transformation are done by your Python code and tools such as dbt, which Dagster runs and tracks.

    How long does an Airflow to Dagster migration take?

    It depends on the number of DAGs, how many are asset-shaped and how many are plain task sequences, custom operators, whether tests exist, and where Dagster will run. A DAG can't be converted mechanically, so the diagnostic produces an inventory and a scoped plan before you commit. Moving pipelines in batches means the most critical ones can move first.

    How is Dagster consulting priced?

    Scope is quoted after the diagnostic, once we have looked at your setup, and before you commit. Cost depends on the number of assets or DAGs, whether you're migrating from Airflow, the hosting model, how much custom code there is, compliance requirements and whether support continues afterward.

    What happens after the work is done?

    The same engineers stay on. They fix breaks, handle planned upgrades and API changes, plan capacity as assets and partitions grow, and handle small changes as requests. The runbooks and documentation stay current, so your Dagster doesn't depend on one person.

    Migration stages, limits and deployment details are from Dagster's Airlift, partitions and Dagster+ documentation and its customer FAQ.

    Data Platform Engineering Use Cases by Industry

    Every industry has source systems that don't talk to each other, often split across cloud and on-premises environments. Data platform engineering is the platform and pipeline work that turns them into one dataset your teams can query.

    Data Platform Engineering for Retail & eCommerce

    POS, storefront, and warehouse systems each hold part of the inventory picture. We build ELT pipelines into one warehouse so stock, sales, and demand sit in one place.

    Data Platform Engineering for SaaS & Software

    Product usage events have to reach reporting without a ticket to the data team. We build the ingestion, dbt models, and self-service golden paths that let product and finance teams ship their own datasets.

    Data Platform Engineering for Logistics

    Load events, warehouse scans, telematics, and EDI documents arrive in different formats. We land each source raw, normalize it in a silver layer, and publish business-ready views on top.

    Data Platform Engineering for Healthcare

    Claims, EHR extracts, and lab feeds use different codes and identifiers. We build pipelines that standardize them and enforce data contracts, so downstream reporting stops breaking when a feed changes.

    Data Platform Engineering for FinTech

    Ledger and payment data have to reconcile across systems. We build the pipelines with schema checks at each hop, so a changed field is caught at ingestion instead of in a finance report.

    Data Platform Engineering for Manufacturing

    MES, ERP, and sensor data live in separate systems on separate schedules. We integrate them into one platform, so plant and finance teams read the same numbers.

    Start With a Conversation About Your Dagster

    If your code locations go unreachable, your backfills crawl, or your move off Airflow has stalled, the first step is a 30-minute discovery call.

    Dagster and Dagster+ are trademarks of Dagster Labs, Inc.