setup-data-workspace

Setup Data Workspace

Turn a folder of loose data files into a single queryable DuckDB database, and teach future Claude sessions (via CLAUDE.md) how to use it.

DuckDB is the right default here: zero-server, single-file .duckdb database, reads CSV/Parquet/JSON/Excel natively, and every skill in this plugin already assumes duckdb is on PATH.

Inputs

• Target repo (default: current working directory).
• Optional: explicit data directory (default: scan the repo for data files).
• Optional: database filename (default: data.duckdb at repo root).

Procedure

Step 1 — Discover data

Scan the repo for data files, ignoring .git, node_modules, .venv, __pycache__, dist, build:

find . -type f \( -iname '*.csv' -o -iname '*.tsv' -o -iname '*.parquet' \
  -o -iname '*.json' -o -iname '*.jsonl' -o -iname '*.ndjson' \
  -o -iname '*.xlsx' -o -iname '*.xls' \) \
  -not -path '*/\.*' -not -path '*/node_modules/*' -not -path '*/\.venv/*'

Report what was found: file count, total size, formats, any obvious groupings (e.g. all CSVs in data/raw/). If nothing is found, ask the user where the data lives.

Step 2 — Confirm scope

Before loading, show the user the file list and confirm:

• Which files to load (default: all discovered).
• Table naming — default: sanitised filename stem (e.g. sales_2024.csv → sales_2024). Flag collisions.
• Whether to load as views (re-read source on every query, always fresh) or tables (materialised, faster but frozen). Default: views for CSV/JSON, tables for Excel (xlsx reading is slow).

Step 3 — Create the database

Generate a loader SQL script at scripts/load_data.sql (create scripts/ if needed). Example shape:

-- Auto-generated by setup-data-workspace
-- Re-run with: duckdb data.duckdb < scripts/load_data.sql

INSTALL excel; LOAD excel;  -- only if xlsx present

CREATE OR REPLACE VIEW sales_2024 AS SELECT * FROM read_csv_auto('data/raw/sales_2024.csv');
CREATE OR REPLACE VIEW customers AS SELECT * FROM read_csv_auto('data/raw/customers.csv');
CREATE OR REPLACE TABLE budget AS SELECT * FROM read_xlsx('data/raw/budget.xlsx');
-- ...