aidp-resume-migration

aidp-resume-migration — pick up where you left off

The migrator is designed to be resumable. Each successful Pass-1 dep migration is cached in two places:

• Module-level _migration_cache dict in job_migrate.py (in-memory, per-run).
• On-cluster os.path.exists() probe against the output path (persistent across runs).

Pass-2 task notebooks are similarly skipped if their final .ipynb already exists at <output-base>/<job>/notebooks/....

When to use

• A prior aidp-migrate-job run was killed (Ctrl-C, cluster restart, network drop, timeout).
• The user manually edited a specific dep / task notebook and wants to resume without re-doing the rest.
• The user added a new task to the manifest and wants to migrate only the new task.

Default behavior (resume is automatic)

python3 ${CLAUDE_PLUGIN_ROOT}/engine/scripts/job_migrate.py \
  --manifest reports/<MyJob>_manifest.json \
  --cluster <CLUSTER_ID> \
  --aidp-base <AIDP_BASE> \
  --datalake-ocid <DATALAKE_OCID> \
  --workspace-id <WORKSPACE_UUID> \
  --output-base <output-workspace-path> \
  --oci-profile <profile>

The default flag is --skip-migrated (ON). Every notebook already at the output path is silently skipped. So a plain re-invoke after a crash IS the resume — no special flag needed.

Resuming from a specific task

If the failure was task-specific and you want to start there (skipping all earlier tasks):

python3 ${CLAUDE_PLUGIN_ROOT}/engine/scripts/job_migrate.py \
  --manifest reports/<MyJob>_manifest.json \
  --start-task <substring_of_task_key>
  # + the rest of the standard args

--start-task is a substring match on task_key. Skip every task whose key sorts before the matched task.

Resuming a single task (skip everything else)

python3 ${CLAUDE_PLUGIN_ROOT}/engine/scripts/job_migrate.py \
  --manifest reports/<MyJob>_manifest.json \
  --only-tasks "<task_key_1>,<task_key_2>"
  # + the rest

Useful when you've manually fixed one dep and want to re-run only the tasks that consume it.