version-dataset

Version Dataset Skill

An analysis must run on the data it claims to, with a fixed seed. This skill makes drift between runs loud instead of silent: it records a deterministic fingerprint (file SHA-256 plus, for tabular files, schema and per-column value hashes; no timestamp unless explicitly passed) so a later run can prove the inputs are unchanged. It never alters data, and manifests hold hashes, not the data itself. Provenance notes are in English.

Deterministic Script

# Build a manifest (record the analysis seed + provenance)
python "${CLAUDE_SKILL_DIR}/scripts/version_dataset.py" manifest data.csv \
  --out manifest.json --seed 42 --provenance "KNHANES 2018 extract v1"

# Verify a later copy against it (CI / pre-analysis gate)
python "${CLAUDE_SKILL_DIR}/scripts/version_dataset.py" verify --manifest manifest.json --strict

# Compare two manifests (what changed between versions)
python "${CLAUDE_SKILL_DIR}/scripts/version_dataset.py" diff --old v1.json --new v2.json

File hashing is stdlib-only; tabular schema/column hashing uses pandas when present. --ignore-cols excludes volatile columns; --base makes manifest keys relative.

Workflow

Step 1: Lock the version (gate)

Build the manifest at the moment the dataset is frozen for analysis. Gate: confirm with the user the seed and provenance note are correct before locking — the manifest is the record they will cite as "this is the data the results came from." The provenance note is the user's text; never invent it.

Step 2: Verify before each run (gate)

Before re-running an analysis (or in CI), verify --strict. Never say a dataset is unchanged without running verify. Gate: if drift is reported (MANIFEST_DRIFT, exit 1), stop and show the user the drift report exactly as computed — never downplay a changed column hash. Do not proceed on changed data without their explicit acknowledgement and a re-lock. Silent re-run on drifted data is the failure this skill exists to prevent.