aidp-migrator-overview

aidp-migrator-overview — router

Migrating a Databricks workload onto AIDP is a multi-phase operation. This skill picks the right next skill based on what the user asks.

When to use

• The user mentions "migrate Databricks", "port from Databricks to AIDP", "Unity Catalog migration", "DBX → AIDP", "lift-and-shift Databricks job", or similar.
• The user asks "where do I start" with the migrator toolkit.
• The user asks for an overview of the migrator architecture.

The two-pass architecture (mental model)

┌────────────────────┐   ┌──────────────────────┐   ┌─────────────────────┐
│ Pass-0: Plan       │ → │ Pass-1: Dep code     │ → │ Pass-2: Execute     │
│  build_dag.py      │   │  ensure_migrated()   │   │  job_migrate.py     │
│  check_data_       │   │  walks %run tree,    │   │  cell-by-cell on a  │
│  availability.py   │   │  rewrites Databricks │   │  live AIDP cluster, │
│                    │   │  APIs in each dep    │   │  4-way verify, up   │
│  (read-only)       │   │  notebook (no run)   │   │  to 10 fix attempts │
└────────────────────┘   └──────────────────────┘   └─────────────────────┘

The catalog migration is a SEPARATE flow, run BEFORE Pass-2 (so the
schemas + table locations exist when migrated notebooks try to read them):

┌──────────────────────────────────┐   ┌──────────────────────────────────┐
│ extract_catalog_databricks.py    │ → │ migrate_catalog.py               │
│  REST against Unity Catalog API  │   │  18 DDL rewrite rules → batched  │
│  → reports/catalog_pack.json     │   │  CREATE SCHEMA / CREATE TABLE    │
│                                  │   │  on AIDP in a single WS execute  │
└──────────────────────────────────┘   └──────────────────────────────────┘

Pick the right skill for the user's ask