data-workflows: Skill for Claude Code

Dataset Recipes

Copy the closest-shaped recipe rather than deriving a pipeline from scratch.

💡 Read one small table out of a HUGE remote zip — range reads, no localize (#518)

To inspect a schema, a lookup/domain table, or one layer's coverage inside a multi-GB zipped GDB, do not localize the archive (a PVC + 30 GB download for a 126-row table). GDAL's /vsizip//vsicurl/ reads the zip central directory plus only the bytes it needs over HTTP range requests — a small cluster job, seconds to a couple of minutes:

# authoritative coded domain out of the archived 30 GB national GDB (internal endpoint)
SRC="/vsizip//vsicurl/http://rook-ceph-rgw-nautiluss3.rook/public-usgs-nhd/raw/NHD_H_National_GDB.zip/NHD_H_National_GDB.gdb"
ogr2ogr -f CSV /vsistdout/ "$SRC" NHDFCode          # 126 rows, ~25 s, no PVC
ogrinfo -ro -q "$SRC" -dialect SQLITE \
  -sql "SELECT COUNT(*), SUM(StreamOrde > 0) FROM NHDPlusFlowlineVAA"

• Works on a public source URL too (/vsizip//vsicurl/https://prd-tnm.s3.amazonaws.com/...) — ideal for pre-flighting a candidate import before committing to a build.
• Use -dialect SQLITE: OGR SQL has no CASE, and keep the SQL on one line (a folded YAML block mangles multi-line SQL). SUM(cond) works in the SQLITE dialect.
• Full-table COUNT(*) over range reads is slow (minutes) because it decodes every feature; schema reads and small tables are fast. Aggregate on the small table, not the geometry layer.
• Working manifests: catalog/usgs-nhd/k8s/extract-fcode-domain.yaml, catalog/usgs-nhd/k8s/preflight-nhdplus-hr-vaa.yaml.
• ⛔ Never hand-write a coded domain from memory (#294) — this is how you get the real one.

💡 Check coverage against the source before you publish (#615)

A source file can be valid, readable, internally consistent — and still be a fraction of the dataset. Nothing downstream notices: convert, pmtiles, hex and verify-stac.py all pass on a short read, because none of them knows what the source was supposed to contain. fmmp-2022 shipped and was consumed for two months holding 45,285 of 127,133 features — 13 of 38 counties, missing Riverside, Monterey, Stanislaus, Sacramento and Imperial.