storage-format

Storage Formats

The format decides what is cheap later. Almost every slow read is a write-time decision showing up months afterwards.

Which format

Parquet — tables. Rows of records with named, typed columns: measurements, catalogues, parameter sweeps, logs. Columnar, so reading two columns of a hundred reads roughly two columns' worth. Compresses well because a column holds one kind of value. Not for multidimensional arrays: a 3-D field flattened into a table loses the very structure that makes it addressable.

HDF5 — multidimensional arrays in a hierarchy, written once and read many times. Slicing along any axis, attributes attached to any node, and a directory tree inside one file. The default for a simulation result someone will analyse later.

ADIOS BP5 — arrays produced during a run, especially in parallel and especially as a time series. Its model has steps built in, and it is designed for many ranks writing concurrently without fighting over one file. Where HDF5 is a result, BP5 is often a stream of them.

Rough test: table → Parquet. Array someone will analyse → HDF5. Array being produced by a running parallel job → BP5.

Chunking is the decision that matters

A chunked HDF5 dataset is stored in fixed blocks. Any read of compressed data fetches and decompresses intersecting chunks, even for one element, so the chunk shape decides which reads are fast and which are disastrous.

Chunk along the axis that will be read. A (time, lat, lon) field chunked as (1, lat, lon) gives cheap whole-maps-per-timestep and expensive time series at one point. Chunked (time, 1, 1), exactly the reverse. Intermediate chunk shapes can balance both access patterns. Benchmark the actual workload before choosing one compromise or maintaining two layouts.

Size the chunk in the region of 100 KB to a few MB. Too small and per-chunk overhead and metadata dominate; too large and every small read drags a large block off disk.

clio-hdf5:get_chunks reports what an existing file chose, which is the first thing to check when reads are slow. A contiguous dataset — no chunking at all — cannot be compressed and cannot be extended.