production-autopsy

Production Autopsy

A system that scores 99.75% on evaluation can still be wrong 10% of the time in production. That is not a contradiction. It means the evaluation measured something other than what deployment requires. This skill exists to find what the model or agent actually learned, why it fails where it fails, and to prove both with numbers that anyone can reproduce.

The output is a diagnostic, not a fix. The goal is a defensible root cause, backed by measurement, that a team can act on. Resist the pull to jump to remediation before the failure is reproduced and explained.

How to run it

This is the start-here skill. Point your coding agent at the system and let it work the procedure:

Run production-autopsy on this system. Treat the eval set and the production behavior as separate distributions. Reproduce the failure on production-realistic inputs, quantify the gap by slice, test confidence calibration under the shift, and isolate root cause by ablation. Verify every number with runnable code, separate measured from inferred, and produce the prioritized report. Where a finding needs depth, use the matching skill in this plugin.

If you are new to this or inheriting a system you did not build, run the full audit. If you already know which failure you are chasing, skip to the matching deep skill listed under "Triage and hand-off" below.

Core commitments

These are non-negotiable and define the methodology:

• Every number is verified, never asserted. Each quantitative claim in the report must trace to a specific measurement produced by runnable code. If a figure cannot be reproduced, it does not go in the report. No estimated, remembered, or "approximately" numbers.
• Measure the gap, not the headline. A single aggregate score (accuracy, exact match, win rate, task success) is the thing that hid the failure in the first place. Always decompose: by input slice, by distribution, by confidence band, and for agents, by step in the trajectory.
• Separate measured from inferred. State plainly which conclusions are directly measured and which are hypotheses still to be tested. Calibrate the confidence of the diagnosis itself.
• Reproducibility is part of the deliverable. The diagnosis is only as good as the script that regenerates it. Seed everything. Report the compute envelope.
• Never regenerate committed fixture data. If the target already ships a fixture (committed data/, results/, or equivalent), treat it as immutable ground truth and measure against it as-is. Synthesize new data only when none exists, and say so explicitly. Re-deriving committed numbers is the job; overwriting the inputs that produced them is never permitted.