metamorphic-evaluation

Note: the crate paths, documents, and measured numbers below are illustrations from the aurora-agent workspace where these methods were developed and tested. The methods themselves apply to any test generation or data augmentation effort.

Metamorphic evaluation

When ground truth is scarce, transformations with known consequences are the next best oracle: if renaming every subject should not change the verdict, and it does, something is wrong — in the system or in the mutation. The discipline is in never letting a transformation into the suite on its own say-so.

Postconditions are executable, and they gate admission

In the workspace's mutation engine (crates/mutation), a mutation declares what the oracle should do as a result of the transformation, and that declaration is checked by running the oracle. The crate's own header states the admission rule: "a mutation is admitted only when the oracle confirms its declared relation." Postconditions are typed, not prose — the verdict is preserved, or a named witness is removed, or a named witness is added — and the relation module draws the line that matters:

A mutation whose postcondition fails is a defect in the mutation, not a new benchmark instance, and is rejected. [...] This is the difference between a mutation engine and a paraphrase generator.

The rejected instance is retained with what the oracle actually did, because a failing postcondition is diagnostic evidence about your transformation code.

Declare both relation kinds where they exist. The metamorphic-response audit (docs/BIOEVAL_METAMORPHIC_AUDIT.md) distinguishes invariance ("this change should not move the response") from directional change ("this change should move it, in this named direction"), and its finding buckets are deliberately not synonyms: a directional trial that stayed put is false sensitivity (a missed response — not proof of invariance); an invariant trial that moved is false invariance (a shortcut); a move against the declaration is wrong direction. An incomparable observation stays undetermined — it is never coerced to "unchanged" and never counted as a pass or a fail. Keep those four apart in your own reports; collapsing them into one pass rate destroys exactly the information the method exists to produce.