h5py

h5py - Hierarchical Data Storage

h5py provides a seamless bridge between NumPy and HDF5. It allows you to organize data into groups (like folders) and datasets (like NumPy arrays), with rich metadata (attributes) attached to every object.

When to Use

• Storing datasets that are much larger than your computer's RAM.
• Organizing complex scientific data into a hierarchical "folder-like" structure.
• Storing numerical arrays (NumPy) with high-speed random access.
• Keeping metadata (units, experiment dates, parameters) attached directly to the data.
• Sharing data between different languages (C, C++, Fortran, Java, MATLAB), as HDF5 is a cross-platform standard.
• Reading/writing large datasets in chunks to optimize I/O performance.

Reference Documentation

Official docs: https://docs.h5py.org/
HDF Group: https://www.hdfgroup.org/
Search patterns: h5py.File, create_dataset, h5py.Group, chunks=True, compression="gzip"

Core Principles

The Hierarchy

HDF5 files contain two main types of objects:

• Datasets: Multidimensional arrays of data (NumPy-like).
• Groups: Container structures that can hold datasets or other groups (like directories).

Slicing

h5py datasets support standard NumPy slicing. When you slice a dataset, only that specific slice is read from the disk, keeping memory usage low.

Attributes

Every group and dataset can have attributes (key-value pairs) for metadata.

Quick Reference

Installation

pip install h5py

Standard Imports

import h5py
import numpy as np

Basic Pattern - Writing and Reading

import h5py
import numpy as np

# Writing data
with h5py.File('data.h5', 'w') as f:
    dset = f.create_dataset('main_data', data=np.random.rand(100, 100))
    dset.attrs['units'] = 'meters'
    grp = f.create_group('subgroup')
    grp.create_dataset('results', data=[1, 2, 3])

# Reading data
with h5py.File('data.h5', 'r') as f:
    data_slice = f['main_data'][0:10, 0:10] # Only read 100 elements
    units = f['main_data'].attrs['units']
    print(f"Group content: {list(f['subgroup'].keys())}")