analyze-model-data

Analyze Model Data

For publication figures, preserve complete plotting arrays, units, group order, independent statistical units, denominators and missing-state rules in the existing data handoff. Record zero-order/zero-transport semantics explicitly; do not treat an unobserved rate as an observed zero.

For calendar, composition or matrix figures, read references/figure-data-handoff.md during preparation. Save dates, grouping/order, units, denominators and missingness with the complete plotted data; perform explicit aggregation here, with prediction or decision fitting delegated to solve-model.

Produce a reproducible evidence trail from raw data to model-ready outputs.

Workflow

1. Preserve the raw files and identify schema, units, provenance, missingness, duplicates, sampling coverage, and target leakage risks. Decide whether the observed data can identify the requested quantity before filling gaps or fitting a model. A missing forecast column is not by itself proof that a forecasting task is impossible: distinguish an externally supplied forecast from a forecast that the problem expects the modeler to generate from prior observations.
2. Run scripts/profile_csv.py for CSV inputs, then inspect domain-specific anomalies manually.
3. Define the analysis question and evaluation metric before choosing transformations or models. For entity-by-time planning data, call algorithms.data_diagnostics.panel_diagnostics (or an equivalent registered implementation) to quantify temporal dependence, cross-entity correlation, support/zero inflation, and a time-ordered holdout before choosing an uncertainty model.
4. Split train/test data before fitting imputers, scalers, encoders, or feature selectors. Keep a model-selection validation partition separate from the final test; when batches or operating conditions differ, report a held-out-condition check and a paired baseline/ablation on the same evaluation units.
5. Compare against a simple baseline and report uncertainty, not only point metrics. Define the independent sampling unit separately from rows, repeated measurements, folds, seeds and parameter-grid points. Label the type of uncertainty; do not infer significance or equivalence from interval overlap alone.
6. Save cleaned data, analysis code, configuration, figures, and a machine-readable result summary.
7. Document every exclusion, imputation, transformation, random seed, and the evidence for or against independent sampling. Preserve a time-aware holdout or stress slice when the task contains future planning; never replace it with a random split merely because the random split scores better.