etl-stabilization

ETL Orchestrator

Fix SnowConvert ETL conversion gaps through phased execution with upfront unit analysis, agent-determined test strategies, and cross-phase learning.

Prerequisites

• Converted ETL unit folder (orchestration .sql + optional dbt subfolders)
• Original source definition file (e.g., .dtsx for SSIS, .xml for Informatica)
• Active Snowflake connection with a warehouse
• DATABASE + SCHEMA with write privileges (CREATE TABLE, CREATE FUNCTION, CREATE PROCEDURE)

When the converted-output folder is not isolated

The default contract above assumes one folder per unit. If instead you're handed a flat, whole-repository conversion output where a dbt project is referenced by more than one sibling unit's orchestration file (e.g. Informatica SHORTCUT mappings reused across workflows), do not edit the shared project in place:

1. Stage a real copy of the orchestration file and every dbt project it references under {PACKAGE_FOLDER}/Output/ETL/{unit}/. Treat the original location read-only until sync-back.
2. If a staged dbt project's packages.yml has a local path: dependency, do not edit the path to account for the extra staging depth — that breaks canonical when synced back. Instead, symlink the shared-assets directory into the unit folder at the depth packages.yml already expects.
3. Before syncing any fix back, run the leak gate on every touched file, then diff against the original — test-environment values (schema/database names, credentials) must never reach the canonical copy:

   uv run --project {SKILL_DIR} python {SKILL_DIR}/scripts/check_sync_leaks.py {SESSION_JSON} <file1> [<file2> ...]
   

   Do not copy until this exits 0.

Persistent Files

All files stored in {UNIT}/stabilization/:

File	Purpose
planning/scan.json	Unit scan output (elements, EWIs, dbt projects)
planning/orchestration-context.md	Orchestration structural understanding (statements, element relationships, containers)
planning/dbt-context.md	dbt project analysis (model inventory, health, source mapping, bootstrap blockers). Only present when unit has dbt projects.
planning/source-excerpts.md	Source-definition excerpts for traceability (conditional — only if source file is large)
planning/ROADMAP.md	Phase plan (agent-authored from template; updated in-place during execution). Never delete content.
tracking/STATE.md	Resumption bookmark (GENERATED — use track_status.py commands, do not edit directly)
tracking/progress.json	Element-level tracking (machine-readable)
tracking/fix-log.md	Append-only record of every fix applied. Absent until Phase 1 completes.
phases/phase-{N}/	Per-phase artifacts: baselines, batch reports, learnings, infrastructure SQL
report.html	Self-contained HTML report aggregating all artifacts. Generated during Final Validation by generate_report.py.

Progressive disclosure: Reference files are loaded on-demand — only read a reference file when its content is needed for the current step.