worker-distributed-setup

Worker Distributed Setup

Use this path when you will run more than one Data Exchange Worker on separate hosts — for example to parallelize a large migration, or to migrate multiple source databases/servers concurrently, one worker per source.

The per-host install/start mechanics are identical to a single worker — this skill covers only what is different across multiple hosts: affinity (so workers don't steal each other's tasks) and per-host config.

Read first: Affinity Reference — what affinity is, the orchestrator claim rule, and why the orchestrator must stay null-affinity.

When you need affinity here

• Multiple workers, one source, sharing load — leave the explicit override unset. scai stamps the same stable project-default affinity on the workflow and every generated worker config, and the orchestrator hands each matching free worker the next task. Follow worker-local-setup/SKILL.md on each host and stop.
• Multiple sources (or you want strict isolation) — you must tag each source's workflow and its worker with a matching, non-null label, or a worker will claim another source's tasks and write to the wrong place. This is the rest of this skill.

Step 1 — Choose one affinity label per source

Pick a short, stable label per source database/server — a per-database label reads well, e.g. mdx-salesnorth, mdx-finance. Confirm the set of labels with the user (one per worker/source).

Step 2 — Tag the workflow and the worker together

For each source, set the label with the plugin's single source of truth:

configure(affinity="<label>")

configure(affinity=…) persists the override. migrate_data/validate_data setup forwards it to workflow generation, and data_infrastructure(mode="up") forwards it to local-worker generation, so the two agree by construction (see the Affinity Reference for the two-carrier rule). Then generate that source's workflow and worker config as usual.