data-sources

Data Sources Skill

Pick the right data source in the data module, respect the schema contract that keeps forward-only sources away from schema-dependent callers, and run the uniqueness checker to find whether a set of columns can serve as a key.

In-memory vs iterative — pick first

Every format ships in two variants:

• InMemory… — loads all rows once (readAll()), then runs multiple recursive checks cheaply. Use when the data fits in heap.
• Iterable… — holds one row at a time (tiny heap), but re-reads the source for each recursive pass. Use for large data or streaming.

The schema contract (don't fight it)

• IterableDataSource — the base forward-only contract: open(), nextRow(), hasMoreData(), reset(), nextRows(int batchSize).
• ColumnarDataSource extends IterableDataSource — adds getColumnNames() for sources whose columns are known up front.
• Forward-only sources that discover keys as they stream — iterable JSON, YAML, TOON — deliberately do not implement ColumnarDataSource. Callers that need the schema (e.g. the uniqueness check) depend on the narrower ColumnarDataSource, so a schema-less source can never be handed in and answer with null. Don't widen IterableDataSource to "fix" a compile error — reach for a columnar source instead.
• getColumnNames() never returns null: a columnar source that cannot name its columns throws IllegalStateException saying why. IterableSQLDataSource does it for a source read before open(); CSVDataSource does it for one built without a columnsRow (new CSVDataSource(fileName) defaults it to -1), which otherwise surfaced two frames away as execForAllColumns() failing with Wrong input: null. A CSV header survives close() and is only dropped — and reloaded — by reset().

Source picker

Format	In-memory	Iterable	Notes
CSV	InMemoryCSVDataSource	CSVDataSource	file path or InputStream
JDBC	InMemorySQLDataSource	IterableSQLDataSource	batchSize sets JDBC fetch size
Parquet	InMemoryParquetDataSource	IterableParquetDataSource	read via in-process DuckDB, JDBC-like
JSON	InMemoryJSONDataSource	IterableJSONDataSource	nested flattened to dotted paths
YAML	InMemoryYAMLDataSource	IterableYAMLDataSource	same flattening; no size limit
TOON	InMemoryTOONDataSource	IterableTOONDataSource	compact, LLM-friendly