review-accuracy-calibration

Review Accuracy and Calibration

Overview

The accuracy problem in code review has two faces: over-flagging (false positives that waste reviewer and author time) and under-flagging (missing real defects). AI-assisted review tools generate false positives that waste 2-5 hours per developer per week, and 25% of AI suggestions contain errors. The fix is not reviewing less — it is calibrating more precisely.

This skill provides the meta-layer that makes every other review skill more effective: a confidence model, heuristics to suppress false positives, a severity calibration table, and an escalation decision guide. Load when filtering comments, assigning severity, or deciding whether to block a PR.

---

Quick Reference — Confidence Levels

Level	Label	Post?	Severity floor
C4 — Certain	You have evidence: test failure, spec violation, data loss	Yes	HIGH or CRITICAL
C3 — High	Strong reasoning: well-known anti-pattern, measurable impact	Yes	MEDIUM or higher
C2 — Medium	Plausible concern but depends on context you lack	Conditional	LOW or NIT
C1 — Low	Speculative; could be intentional or context-dependent	No (investigate first)	—

---

Confidence Scoring Model

Assign a confidence level to every finding before posting it.

C4 — Certain

You have direct evidence the code is wrong:

• A test fails or would fail if run
• The code violates a documented spec, contract, or schema
• Data loss or security exposure is provable (e.g., missing WHERE clause on DELETE, secret in source)
• The behavior contradicts the PR description

Action: Always post. Set severity to HIGH or CRITICAL. No hedge language needed.

// C4 example — provably wrong
DELETE FROM users   -- Missing WHERE clause: deletes ALL rows
// Post as CRITICAL. No ambiguity.

C3 — High

Strong reasoning based on established patterns:

• A well-documented anti-pattern (N+1 query, mutable default argument, race condition on shared state)
• Clear performance or reliability impact measurable from the code
• Inconsistency with the existing codebase pattern (all other handlers do X; this one does Y)