policyengine-data

PolicyEngine data

How the microdata behind PolicyEngine population runs is built, calibrated, versioned, and named. For using datasets in a simulation, see the policyengine skill (this skill is about where the data comes from). For diagnosing why one score disagrees with a benchmark, see policyengine-calibration-diagnostics.

The current data stack is Microcosm (repo PolicyEngine/microcosm, local mirror ~/PolicyEngine/microcosm; read its README.md + DESIGN.md). It replaced the technique-named packages of the previous stack (microdf / microimpute / microcalibrate / L0 / policyengine-us-data), which shared no datatype and had their worst bugs at the seams between flat DataFrames.

The Microcosm architecture

Microcosm is one kernel datatype — the Frame — with packages as operators on it. It is a PEP 420 namespace (microcosm.*) shipped as independently-installable shard distributions, so an analyst doing imputation never has to install torch and vice versa. Microcosm releases pin the shards as a constellation.

Package	Import	Role	Succeeds
microcosm-frame	microcosm.frame	the kernel: Frame, typed weights, strata, links, weighted accounting, unit structure, the RulesEngine protocol	microdf, microunit
microcosm-fit	microcosm.fit	conditional models (weight-aware by construction)	ad-hoc imputation scripts
microcosm-calibrate	microcosm.calibrate	targets → calibrated weights (APG / L0)	microcalibrate
microcosm-build	microcosm.build	build plans, donor graphs, release gates, country build stages	one-off build drivers
microcosm-data	microcosm.data	published population registry + lazy engine loaders	country-specific data packages

Key design facts (from DESIGN.md) that change how you reason about the data:

• The Frame is a weighted sampling frame of entity tables. Person + group-entity tables with explicit person_<group>_id linkage established once at assembly — no operator re-derives person↔unit attachment from a flat frame. It carries typed weights (design | importance | calibrated, one vector per weighted entity) with conservation invariants the kernel enforces (strata mass sums; no silent zeroing; no NaN/negative), and strata giving every record explicit provenance (cps_passthrough, synthetic_conditional, tail_verbatim, ...). Generation owns support (oversample where it is scarce); calibration owns representation.
• The rules engine is an adapter, not a dependency. microcosm.frame.rules.RulesEngine is a Protocol (variable_entity, variable_dtype, entity_schema, materialize, export_contract, write_dataset). Today's adapter is policyengine_us; the Axiom rulespec-us adapter is written against the same protocol so the swap is a new adapter, not a migration.
• microcosm-fit is weight-aware by construction — fits read the frame's typed weights; there is no unweighted default. Canonical model: regime-gated, chained quantile forests with weights materialized by weighted bootstrap.
• microcosm-calibrate is the only place calibrated weights are produced. Sparse target-matrix compilation + APG / L0 pruning is the core, not an option — "generate big then prune" is the intended design (300k → 3M → 30M candidate pools pruned to a compact frame). Its longitudinal rule: one weight per trajectory (multi-period targets stack as (target, period) constraint rows over one weight vector).
• Process rules are as binding as the architecture: behavioral contract tests in CI from day one (weighted fits shift draws toward the weighted truth; calibration conserves declared mass; unit assignment partitions exactly); constellation versioning (consumers pin the constellation, not git SHAs); artifacts embed a certificate of the rules-engine + package versions that produced them; stage manifests are versioned artifacts with invariant checks.