reproducibility-workflow

Reproducibility Workflow

Description

This skill covers the systematic provenance tracking, data versioning, and environment capture required to make computational materials science campaigns reproducible: DFT dataset provenance, MLP training experiment tracking, active learning iteration versioning, environment snapshots, model registry management, and data integrity verification. Invoke this skill when designing a new simulation or ML campaign from scratch, when auditing an inherited dataset or model, or when preparing data and results for publication or long-term archiving.

Domain Context

Reproducibility in computational materials science and scientific ML has two distinct failure modes that are easy to confuse. The first is bit-exact reproducibility: given identical inputs, code, and hardware, the same numbers appear. The second is scientific reproducibility: an independent researcher using the same protocol and nominally identical inputs arrives at the same scientific conclusions. The first is often achievable for DFT and classical MD; it becomes harder for GPU-accelerated ML training where floating-point nondeterminism is inherent. The second requires that the protocol itself — inputs, settings, preprocessing steps, and analysis code — is fully recorded.

In computational materials science, the most common reproducibility failures are not deliberate falsification but silent data corruption: a POTCAR file quietly updated mid-campaign, a preprocessing script run without recording its parameters, a train/test split generated with no fixed random seed, a model checkpoint overwritten by a re-run. Each of these produces results that look correct locally but cannot be reproduced six months later, or produce subtly different results when the dataset is extended.

The provenance chain in a typical MLP campaign spans at least four levels:

1. DFT calculations: Code version, pseudopotentials (checksums), functional, ENCUT/ecutwfc, k-mesh density, smearing, convergence thresholds, cell, PBC, MAGMOM. A change at this level invalidates all downstream training.
2. Dataset assembly: Which raw calculations were included, which were filtered (and why), the extxyz conversion script, the train/val/test split seed, and the resulting dataset file checksums.
3. Model training: Architecture hyperparameters, loss weights, random seed, training data file checksum, optimizer settings, number of epochs, the exact training command, and the validation metrics at the best checkpoint.
4. Deployment and production: Model file checksum, LAMMPS version and plugin hash, simulation input script, initial structure, random seed for velocity initialization, ensemble and thermostat settings.