snapshot-data

Skill: Snapshot Data

Produce a manifest YAML for the dataset at $ARGUMENTS so any future re-run of an analysis can verify that the input data is unchanged. The manifest captures both byte-level identity (file hash) and semantic shape (schema fingerprint).

Principle

Two hashes are not redundant — they answer different questions:

• sha256 of the file detects any byte-level change, including a benign re-export with different encoding.
• schema fingerprint detects semantic changes. If the file is re-saved with a different encoding the SHA changes but the fingerprint stays — that tells you the data is the same even if the bytes are not.

Step 1 — Locate and validate the dataset

Resolve $ARGUMENTS to an absolute path. If the file does not exist, fail with a clear error message. Support these formats by extension: .csv, .parquet, .json, .jsonl, .xlsx, .xls. If the format is unknown, fail with a clear error message.

Step 2 — Compute the manifest

Run a Python one-shot via Bash (use python3 -c "..." or write a temp script at /tmp/snapshot.py) that computes and prints YAML to stdout. The script must:

1. Compute sha256 of the file by reading it in chunks (do not load the whole file in memory for large datasets).
2. Capture os.stat: size_bytes, modified_at (ISO-8601, UTC).
3. Load the dataset with the appropriate reader (pd.read_csv etc.) and capture:
   • rows, cols
   • columns (list, in original order)
   • dtypes per column (as strings, e.g. "int64", "object")
   • null_counts per column (only columns with at least 1 null)
4. Compute dtypes_fingerprint: SHA-256 (first 16 hex chars) of the canonical string <rows>|<sorted_columns>|<sorted_dtypes_tuples> where each pair is name:dtype. Sorting is deliberate so column order does not affect the fingerprint — only schema does.
5. Capture generation metadata: by: snapshot-data, at: <ISO-8601 UTC now>, format: <inferred from extension>.

The manifest structure must follow this exact YAML layout: