data-infrastructure-setup

Data Infrastructure Setup

Overview — Tell the User

Before starting any configuration, tell the user verbatim:

Snowflake's Data Migration and Validation Framework is a fault-tolerant, scalable system for Snowflake Migrations. It uses a two-component architecture: an Orchestrator that runs inside your Snowflake account (on Snowpark Container Services) and breaks migration work into parallel tasks, and one or more Workers that run in an environment of your choice, connect and read data from your source system, and upload it to your Snowflake account.

You make two independent placement choices in Step 0: where the orchestrator runs — local (on your machine, no compute pool — simplest) or SPCS (on a Snowpark Container Services compute pool — scalable) — and where each worker runs (usually your machine, since it reads your source; optionally other VMs, SPCS, or Kubernetes). The two don't have to match; nothing about compute pools matters until you put the orchestrator on SPCS.

You are in full control of what gets migrated and how. You decide which tables to include, how they are partitioned for extraction, whether to use incremental synchronization or full loads, and how many workers run in parallel. You can filter rows, remap column names and types, and stop or resume the process at any point without losing progress. Nothing runs until you review and approve the configuration.

The same infrastructure is used for validation. You choose what level of validation to run: schema validation, metrics validation, and row-level validation.

Metadata storage mode (trial accounts): On Snowflake accounts without Hybrid Table support, the orchestrator falls back to standard/TRANSIENT metadata tables automatically. That path preserves correctness but reduces task-queue throughput under contention and effectively limits how many workers you should run in parallel — start with fewer workers and scale up carefully. This is not the same as L3 “hybrid” row validation. See Metadata storage mode reference.