Airflow 2.x to Airflow 3: Migrate DAGs, Metadata, Providers, and Operations Safely

An Airflow upgrade can look green and still be wrong. The new user interface loads, every Directed Acyclic Graph (DAG) parses, and a manually triggered finance workflow finishes successfully. Yet the output lands under the wrong business date because the task kept treating data_interval_start as the date supplied by the operator. Nothing crashed; the control failure appeared only when downstream reconciliation found a duplicate partition and a missing one.

An Airflow 2 to 3 migration is not a package replacement. Verified September 3, 2026, the Apache Airflow 3.3.1 release notes date 3.3.1 to August 12, 2026 and list 2.11.2 as the final 2.x patch. The version-specific 2.11.2 notes use March 14 rather than the stable index’s March 11; the staging version is unchanged. The supported-versions policy records Airflow 2 as end of life on April 22, 2026. Use 2.11.2 only as a temporary migration baseline. Airflow requires 2.7 or later and recommends the latest 2.x first.

A defensible migration proves four surfaces independently: DAG code compatibility; provider, plugin, and executor packaging; metadata and deployment architecture; and schedule-to-business semantics. This Airflow 3 migration runbook treats each surface as a testable control. It stabilizes the 2.x estate, inventories dependencies, rehearses database changes, migrates public interfaces, compares old and new outcomes, and keeps a restore-capable rollback path until the final evidence is signed. Import success is an input, never the acceptance decision.

Understand Why Airflow 3 Is an Architecture Migration

The official Airflow 3 upgrade guidance makes the architectural boundary explicit: tasks and workers no longer receive unrestricted direct access to the metadata database. Runtime interactions such as state transitions, heartbeats, XCom operations, and resource retrieval move through the Task Execution API, with the API server as the controlled path. DAG authors are expected to use the stable Task Software Development Kit (Task SDK) and the documented public interface instead of importing internal models simply because those imports worked in Airflow 2.

This changes threat exposure and failure behavior. Removing database credentials and drivers from ordinary worker images reduces the blast radius of task code, but it also makes API-server availability, authentication, network policy, and latency part of task execution. The public interface for Airflow 3 therefore becomes a production contract. A custom operator that reads TaskInstance rows through an Object-Relational Mapper (ORM) is not merely deprecated; it violates the new isolation model and can fail even when the DAG file imports.

Architecture decision matrix

The platform owner should record one architecture decision for each runtime dependency: who operates it, how it is authenticated, which network path it uses, what saturation looks like, and what happens when it is unavailable. Promotion is blocked if workers can bypass the intended API boundary, if API-server failure is absent from resilience testing, or if deployment manifests still assume that airflow webserver is the production entry point. Rollback at this stage means reverting design and code changes on the 2.x baseline; no database restore should yet be required.

Inventory DAG Ownership, Providers, Plugins, and Deployment Paths

An estate inventory is the migration control plane. Package manifests miss dependencies baked into different component images or supplied by a managed service. A low-frequency DAG can also be financially critical. Inventory each workflow, runtime, and business effect as one record.

Refonte Learning separately introduces Kafka, Spark, Airflow, and other data-engineering tools; this runbook asks a narrower question: which estate dependencies must remain compatible while the orchestrator changes?

Map DAG Criticality, Schedules, and Business Consumers

Minimum DAG inventory record

Classify criticality by business impact, not frequency. Record the latest safe completion time, recovery objective, and whether replay is idempotent. Do not approve a DAG with no business owner or no independent output-reconciliation method.

Build the Provider, Plugin, and Executor Matrix

The Airflow provider packages index documents separately released packages, so core compatibility cannot be inferred from one apache-airflow version. Inventory every scheduler, API-server, DAG-processor, triggerer, worker, CLI, migration, and authoring image, including internal wheels and editable installs.

Dependency matrix

Use the broader data-engineering tools and technologies landscape to locate adjacent systems. The matrix is complete when every DAG maps to a tested image and every package has an owner, exact version, compatibility result, and rollback artifact. Unknown or “latest” versions stop promotion.

Stabilize the Latest Airflow 2.x Staging Baseline

Do not combine an unresolved 2.x upgrade with Airflow 3 remediation. The floor is 2.7, but the project recommends the latest 2.x first. Use 2.11.2 as the upstream staging baseline unless a managed-service vendor imposes a lower ceiling. Because Airflow 2 is end of life, use the hold period to remove variables and collect evidence, not delay Airflow 3. Record any distribution divergence.

The general Airflow upgrade procedure recommends incremental upgrades and treats database migration as a distinct operation. Move through required intermediate releases, fix deprecations, then hold a stability period on 2.11.2. Freeze the Python version, constraints or lock files, operating-system base, provider set, and internal wheels. Rebuild all component images from the same bill of materials rather than patching live containers.

Baseline verification sequence

# Reference baseline commands; run inside the immutable Airflow image
python -m pip install --require-hashes -r requirements-airflow-2.11.2.lock
airflow version
airflow config lint
airflow dags reserialize

2.x stabilization gate

This baseline supplies the control group for every later comparison. Set measurable tolerances now: for example, zero missed Tier 0 runs, zero duplicate external writes, no unexplained increase in scheduler delay, and task-duration variance bounded by a workload-specific threshold. Rollback is the known 2.11.2 image and configuration plus the pre-change provider lock. If 2.x cannot remain healthy under that frozen set, the Airflow 3 cutover clock has not started.

Clean, Back Up, and Rehearse the Metadata Migration

Treat the metadata upgrade as a production database change. Its size affects duration, locks, transaction logs, storage headroom, and restore time. The database owner, platform owner, and incident commander must agree on the sequence and the point where restore becomes mandatory.

Official guidance recommends cleanup and warns about half-migrated states. The Airflow command-line reference documents airflow db clean, including cutoff, dry-run, table, archive, and batch options. Obtain retention approval; never delete history only to shorten the window.

Metadata cleanup rehearsal

# Illustrative cutoff only; obtain retention approval before execution
airflow db clean \
  --clean-before-timestamp "2025-09-03T00:00:00+00:00" \
  --dry-run

# Execute only after reviewing the dry-run and backup/restore evidence
airflow db clean \
  --clean-before-timestamp "2025-09-03T00:00:00+00:00" \
  --batch-size 5000 \
  --yes

1. Quiesce writers. Pause Airflow components and automation unless a validated consistent hot-backup method exists.

2. Capture a restorable backup. Record engine version, backup ID, key dependency, recovery position, checksum, location, and expiry.

3. Restore in isolation. Validate critical row counts, connectivity, authentication, and representative historical records.

4. Measure recovery. Time restore, startup, and reconciliation against the rollback window.

Metadata safety gates

Reserialize DAGs and Time the Database Changes

Before leaving 2.x, run airflow dags reserialize after all fixes are processed. Retain output, import errors, serialized-DAG counts, and parser duration. Duplicate IDs or unexplained missing DAGs block migration.

Rehearse airflow db migrate on a production-sized restored clone. The database migration reference lists revisions and warns that the schema is internal. Review generated SQL where supported, then time the real command while recording locks, CPU, I/O, space, log growth, and schema revisions.

Schema review and rehearsal

# Run with the Airflow 3.3.1 migration image against an isolated clone
airflow db migrate --show-sql-only \
  --from-version "2.11.2" \
  --to-version "3.3.1" > migration-plan.sql

airflow db migrate

Set the window from the slowest credible rehearsal plus restore and validation. Do not deploy the old image over a failed new schema. Unless an exact supported downgrade exists, restore the pre-migration database and frozen 2.11.2 stack. Protect that restore point through business reconciliation.

Scan DAGs and Custom Code for Airflow 3 Incompatibilities

Compatibility scanning must cover DAG repositories, plugin packages, shared Python libraries, deployment scripts, Helm values, command wrappers, test fixtures, and notebooks copied into production images. Start with automated discovery, then assign every result. A search hit is not automatically a defect, and a clean linter is not proof that runtime semantics are preserved.

Repository discovery searches

rg -n "airflow\.(models|settings|utils\.session)|create_session|provide_session" dags plugins src
rg -n "SubDagOperator|sla=|execution_date|/api/v1|airflow webserver" dags plugins deploy tests
rg -n "data_interval_(start|end)|xcom_pull\(|TriggerDagRunOperator" dags plugins src
rg -n "AIRFLOW__[A-Z0-9_]+__|webserver:" deploy charts config

Compatibility finding register

Assign each finding an owner, criticality, remediation pull request, test case, target image, and disposition date. For custom operators, inspect both constructor-time code and execute behavior; top-level imports can pass while runtime access fails only on workers. For callbacks and listeners, verify the context keys actually consumed, not just the callback signature. For XCom calls, identify pulls that omit task_ids, because Airflow 3 narrows that default behavior.

Promotion requires zero unowned findings and zero open breaking-change findings for the cutover cohort. A temporary exception must state why the path remains supported, how it is monitored, how long it is allowed, and which rollback trigger applies. “DAG imports on my laptop” is not evidence because it bypasses deployment packaging, secrets, executor, API, and business-side-effect conditions.

Migrate Imports, Operators, and the Standard Provider

Separate Airflow core, the standard provider, third-party providers, and custom packages. Airflow 3 moves authoring interfaces toward airflow.sdk, while common operators and sensors move to apache-airflow-providers-standard. Stage compatible import changes on 2.x to shrink the cutover.

Use the standard provider documentation and each provider’s release notes. Verified September 3, 2026, stable documentation listed apache-airflow-providers-standard 1.18.0 with minimum Airflow 2.11.0, allowing many import changes to be staged on 2.11.2. Pin the approved provider and hashes; a solvable environment can still have wrong extras, libraries, authentication, or behavior.

Representative import migration map

Build clean 2.11.2 and 3.3.1 images from the same source. Install internal release wheels, then test imports, provider discovery, serialization, hooks, templates, connections, logs, and executor tasks. Compare installed packages with the approved bill of materials.

Use Compatibility Rules and Ruff AIR Checks Deliberately

Use Ruff 0.13.1 or later. AIR301/AIR302 identify breaking changes; AIR311/AIR312 flag recommended updates. Scan DAGs, plugins, and shared code by commit, and preview fixes before applying them.

Compatibility lint workflow

python -m ruff check dags plugins src --select AIR301,AIR302,AIR311,AIR312
python -m ruff check dags plugins src --select AIR301 --show-fixes

# Apply only reviewed fixes on a branch; do not auto-merge unsafe changes
python -m ruff check dags plugins src --select AIR301 --fix

Treat automated changes as behavior-affecting. New packages can change optional dependencies, and rewritten context can alter dates. Require a reproducible install plus semantic tests, not only zero Ruff findings. Preserve the previous lock, commit, and artifact tag for rollback.

Remove Direct Metadata-Database Access from Task Code

Direct metadata access is the compatibility issue most likely to be hidden inside shared helpers. Search for Airflow sessions, ORM models, SQL against metadata tables, and utilities that accept a session argument. Then identify the actual intent: retrieve a variable, inspect a connection, pull an XCom, learn the current task identity, query run state, or maintain application data. The replacement should match that intent rather than reproduce an internal table query through another credential.

The stable Task SDK API exposes task context and supported resource access. Use context, variables, connections, XComs, hooks, or the public REST/Python client where the project provides the operation. Move business state to a business-owned database or object store. Airflow metadata is an orchestration record, not a general-purpose application database.

Metadata-access replacement decisions

Supported task-context access

from airflow.sdk import get_current_context, task

@task
def publish_checkpoint() -> dict[str, str]:
    context = get_current_context()
    ti = context["ti"]
    return {
        "dag_id": ti.dag_id,
        "task_id": ti.task_id,
        "run_id": ti.run_id,
        "try_number": str(ti.try_number),
    }

Account for the new failure path. API access adds token acquisition, authorization, network, timeout, and API-capacity dependencies. Tests must distinguish retryable transport errors from authorization or invalid-request failures; uncontrolled retries can amplify an API-server incident. A database hook aimed at Airflow metadata may exist as a documented workaround, but the project warns that it breaks task isolation and is not a durable public contract. Treat it as a time-bounded exception with read-only temporary credentials, not as the target design.

Adapt to the API Server, Task SDK, and Process Layout

Airflow 3 separates responsibilities often bundled in 2.x: API service, scheduling, DAG parsing, triggering, task execution, and administration. Give each component an image, command, identity, network policy, resource budget, logging path, and health model.

Airflow 3 process contract

The Airflow production deployment guidance covers external databases, remote logging, controlled configuration, and executor choice. Map every component to the API server, database, broker, logs, secrets, and business endpoints. Deny worker-to-metadata-database traffic after compatibility proof.

Replace Webserver and Startup Assumptions Safely

The webserver command becomes airflow api-server, and the DAG processor starts independently. Search units, entrypoints, Deployments, probes, services, ingress, dashboards, and runbooks for webserver. A renamed process without network and probe changes can look running while unusable.

Startup contract

# Airflow 3.3.1 component commands
airflow api-server
airflow scheduler
airflow dag-processor
airflow triggerer

# Verify the deployed version from the same image used by each process
airflow version

The Airflow security model documents trust boundaries and /api/v2/monitor/health. Also test authenticated routes, scheduler and worker heartbeats, DAG processing, database access, and a synthetic DAG. Tune startup probes for initialization and readiness for traffic removal.

Scale the resource that saturates: API concurrency, scheduler delay, parse queue, triggers, or workers. Cap scaling at database and broker capacity. Block promotion if ownership is unclear, workers retain unnecessary DB credentials, or service-name rollback is untested.

Qualify Authentication, Plugins, Executors, and Helm Packaging

Authentication is not a post-upgrade login check. Airflow 3 changes defaults, routes, compatibility providers, and plugin interfaces. Auth-manager routes use /auth; update redirect URIs and contract-test browser and automated clients. Inventory the auth manager, Single Sign-On (SSO) protocol, role mapping, token flow, service accounts, API clients, and emergency access. Test authorization boundaries, not only successful login.

Core Airflow 3 defaults to Simple auth, while the official Helm chart’s upgrade path defaults to FabAuthManager for continuity. Set the auth manager explicitly. Airflow’s Simple auth manager documentation says it is for development and testing; production requires other access controls. Teams retaining Flask-AppBuilder must pin the FAB provider and verify routes, permissions, callbacks, and failure isolation. Move unsupported Flask blueprints to external views, FastAPI applications, or middleware.

Platform qualification matrix

For Kubernetes, follow the official Airflow 3 Helm upgrade guide and compare rendered manifests. Verified September 3, 2026, stable chart documentation was 1.22.0; the guide requires at least 1.19.0, sets both defaultAirflowTag and airflowVersion, and moves webserver values under apiServer. Chart and core versions are independent, and chart 1.22.0 still defaulted to Airflow 3.2.2, so pin 3.3.1. Also move [webserver] secret_key to [api] secret_key and provision the JWT secret before components start.

Minimal version-explicit Helm values fragment

airflowVersion: "3.3.1"
defaultAirflowTag: "3.3.1"

apiServer:
  replicas: 2
dagProcessor:
  replicas: 2

The chart 1.22.0 parameters reference exposes separate apiServer and dagProcessor controls. The guide also requires Kubernetes 1.30 or later for chart 1.19.0. Validate rendered manifests, permissions, services, storage, logs, migration jobs, network policies, and secrets against the process map. Preserve the previous chart, values, image digests, and manifests.

Test Scheduling, Logical Dates, Backfills, and Manual Runs

Scheduling is where green upgrades become business incidents. For each critical DAG, record the expected business key and source window for scheduled, manual, backfill, catchup, asset, downstream-trigger, retry, clear, rerun, and timezone cases. Compare outputs, not only UI timestamps.

Airflow 3 defaults catchup_by_default and create_cron_data_intervals to false; bare cron schedules therefore use trigger-oriented behavior unless configured otherwise. The timetable documentation explains logical dates and intervals. Decide and test the intended contract before creating Airflow 3 runs.

Scheduling semantics test matrix

The Airflow DAG run guidance says to use logical_date when a manual task needs the supplied date. The following 3.3.1 DAG imports and runs, yet can select the wrong finance date.

Worked example: code-compatible but business-wrong

import pendulum
from airflow.sdk import DAG, get_current_context, task

with DAG(
    dag_id="finance_daily_close_wrong",
    schedule="0 2 * * *",
    start_date=pendulum.datetime(2026, 1, 1, tz="UTC"),
    catchup=False,
):
    @task
    def close_ledger() -> str:
        context = get_current_context()
        # WRONG for a manual run when the operator-supplied date is required.
        business_date = context["data_interval_start"].date().isoformat()
        return f"closing ledger for {business_date}"

    close_ledger()

An operator supplies logical date 2026-08-31T00:00:00Z for the August 31 close. The resolved interval is not guaranteed to start there, so the task can succeed against another partition. Imports, task state, and row count may all pass.

Corrected manual-run business-date logic

import pendulum
from airflow.sdk import DAG, get_current_context, task

with DAG(
    dag_id="finance_daily_close",
    schedule="0 2 * * *",
    start_date=pendulum.datetime(2026, 1, 1, tz="UTC"),
    catchup=False,
):
    @task
    def close_ledger() -> str:
        context = get_current_context()
        # CORRECT when the contract is the operator-supplied manual date.
        business_date = context["logical_date"].date().isoformat()
        return f"closing ledger for {business_date}"

    close_ledger()

Do not replace every interval field mechanically: scheduled interval processing often needs both boundaries. Test fixed context, every production trigger path, output partition, and downstream ledger. Any unexplained date, timezone, run-count, or partition difference blocks promotion.

Run Shadow Workloads and Compare Business Outcomes

Run Airflow 3 against controlled inputs without production side effects. Use immutable fixtures, cloned sources, isolated outputs, stubbed notifications, and synthetic APIs. Permit production reads only with read-only credentials; capture unsafe writes as comparison events.

Pair every 2.11.2 and 3.3.1 run by DAG, case, logical date, interval, fixture, commit, package lock, and configuration. Compare graph, mapped tasks, states, retries, callbacks, XComs, checksums, business totals, and downstream events per run.

Shadow-run comparison controls

The discipline of building reliable data pipelines combines correctness, recoverability, observability, and controlled change. Here, evidence must also be Airflow- and run-specific; a green dashboard cannot waive failed reconciliation.

Prove Interval, Retry, XCom, and Callback Equivalence

Test the drift points: logical date and intervals per trigger path, retry after an external action, task clears, worker and triggerer restarts, mapped-task cardinalities, callback order, permissions, payload, and duplicate suppression.

In Airflow 3, xcom_pull() without task_ids checks the current task. Use an explicit producer and key, then test missing values and reruns.

Business-equivalence evidence

Require signed evidence per critical pattern. Shared operators may reuse a pattern test, but distinct trigger paths need separate cases. Exceptions need residual risk and reconciliation; the shadow stage ends only when every material difference is explained.

Cut Over with Observability and Promotion Gates

Cutover observability must show platform health and business completion in the same timeline. Establish dashboards and alerts before the change, then annotate the exact times of traffic shifts, scheduler pauses, database migration, component starts, DAG enablement, and first runs. Retain raw logs and metrics long enough to investigate a defect discovered after a monthly or quarter-end workflow.

Cutover control board

Use DataOps workflow automation principles to automate evidence collection, but keep promotion authority explicit. Automation can compare run manifests, query health endpoints, fetch metrics, and reconcile outputs; it should not silently promote a cohort whose business owner has not accepted the result. The incident commander owns the stop/continue decision, while platform, database, security, and business owners sign their gates.

1. Canary enablement. Start with non-destructive representative DAGs, then one critical pattern with strong reconciliation and an open rollback window.

2. Cohort promotion. Enable DAGs by dependency and side-effect group, not arbitrary count; watch at least one full relevant schedule cycle.

3. Hold points. Pause after metadata migration, component health, first scheduled run, first manual run, and each critical cohort.

4. Closure. Preserve the 2.11.2 artifacts and backup until the agreed observation period, backlog, and business reconciliation are complete.

Use Airflow audit logging guidance to retain who triggered, cleared, edited, or changed state during the migration. Stop promotion on any unexplained missed run, duplicate side effect, wrong business date, privilege regression, rising database saturation, or API failure that threatens task progress. A stop is not automatically a rollback; first determine whether freezing further enablement preserves a safer recovery path.

Rehearse Rollback from Every Migration Stage

Rollback differs across package, configuration, metadata, and business-state boundaries. Rehearse each stage with production images, backups, manifests, credentials, and roles, recording when each path expires.

Stage-specific rollback matrix

Name who stops components, identifies active jobs, retrieves backup keys, switches endpoints, verifies schema, authenticates old images, and calculates the resumption boundary. Test under pressure; an inaccessible key or deleted image invalidates rollback.

• Rollback triggers: failed schema change, corruption, unrecoverable auth/API outage, wrong dates, duplicate Tier 0 effects, or an expiring recovery window.

• Recovery conditions: known schema, healthy 2.11.2 control plane, no ambiguous run, reconciled effects, and approval to resume.

• Post-rollback: preserve logs, migration output, database diagnostics, manifests, and events before another attempt.

Practice a forward stop too. A semantically wrong cohort may be paused and fixed while validated cohorts remain. A half-migrated database is platform-wide: stop writers and restore.

Execute the 60-Day Airflow 3 Migration Plan

A 60-day plan sequences risk retirement. Smaller estates may compress elapsed time, not evidence gates. Managed services can control versions, schema, images, or rollback; document that constraint and use the supported path.

60-day migration timeline

Maintain one board for surface, component/DAG, owner, evidence, gate, blocker, rollback dependency, due date, and exception expiry. Review risk changes daily; status colors without evidence are not progress.

For adjacent skills, see must-have tools for data engineers and the data engineering career roadmap. Keep learning goals outside the release-critical path: approval depends on ownership and evidence.

• Day-one actions: freeze the estate, name critical-DAG owners, and create the two-version build matrix.

• Weekly evidence: risk register, package results, migration timing, semantic mismatches, and rollback readiness.

• Exception rule: require an owner, control, test, expiry, and rollback consequence.

• Completion rule: retire Airflow 2 artifacts only after observation, reconciliation, and recovery approval.

Reserve parallel compute, isolated outputs, database headroom, test executor capacity, and reconciliation staff. They buy evidence before rollback narrows. Close by operationalizing dashboards, retiring exception credentials, deleting shadow data under policy, and transferring runbooks to on-call.

A successful Airflow 2 to 3 migration proves four things at once: DAG code uses supported interfaces; the metadata database can be migrated and recovered within an accepted window; the Airflow 3 process and security architecture remains healthy under real workload; and every critical workflow preserves its intended business date, interval, retries, XCom behavior, callbacks, and downstream effects. Any one of those can fail while the user interface remains green.

Start with three actions: freeze and inventory the current estate, stabilize the 2.11.2 staging baseline, and run a production-sized backup/restore plus metadata-migration rehearsal. Then move code and packages through reproducible images, test schedule semantics explicitly, compare shadow outcomes, and promote only at signed gates. Readers considering the wider profession can separately explore whether data engineering is future-proof, but this cutover should remain an engineering-control decision.

For structured practice in the broader foundations behind this work, the Refonte Learning Data Engineering Program describes a three-month commitment of 12–14 hours per week and covers data warehousing and ETL, scalable data pipelines, batch and streaming processing, data governance, Hadoop and Spark, and practical projects. Its page also advertises a Training Certificate and Certificate of Internship upon successful completion. The page does not explicitly name Airflow 3, the Task SDK, provider packages, Helm, or metadata migration, so the connection is to verified pipeline and governance foundations rather than an asserted product-specific module.