analysis-review

Analysis Review

Overview

Code review asks "is this code correct?" Analysis review asks a harder question: "is this conclusion correct?" — and the two come apart completely. Analytics code can be clean, well-tested, and pass any software review while still delivering a confidently wrong number, because the bugs that matter here don't live in the syntax. They live in a join that fanned out, a metric nobody defined, a feature that leaked the target, a specification that was fished. This skill is a review lens aimed at exactly those.

Core principle: Review the path from data to claim, not just the code. The question is never "does it run?" — it's "would I bet the decision on this number?"

"Review it" re-fires this skill every time, including mid-session — a design reviewed last week, or earlier in this conversation, doesn't stay reviewed. A new cut, a re-run, or a fresh "does this look right?" is a new artifact to review from scratch. Don't answer from loaded context ("I already looked at this") — re-run the checklist against this result.

Reviewing an analysis — the checklist

Start from result-verification's verification checklist — reconcile to source, reproduce from a clean state with a fixed seed, joins/cardinality, missingness, units/grain, artifacts tied to prose — run it, don't re-derive it here. Review adds the adversarial lens (below), plus what that checklist doesn't cover:

The claim

• Is the metric/estimand defined precisely enough that you could recompute it the same way? (If "active users" or "the effect" is undefined, stop here — see question-framing.)
• Does the conclusion actually follow from the number, or is it a causal claim resting on a descriptive estimate?

Models & causal claims

• Leakage: any feature that encodes the target, any train/test overlap, any future information in a predictor? Leakage is the most common reason a model metric is "too good."
• Identification: for any causal claim, is the design named and are its assumptions stated and tested? (Hand off to causal-identification — parallel trends, first-stage F, manipulation test, balance.)
• Specification search: were the reported specs chosen before or after seeing results? Are the robustness checks the complete set, or a flattering subset? (See pre-analysis-plan.)
• Structural models: is each parameter's identification stated — what variation or moment moves it? Was the estimator shown to recover known parameters (a Monte-Carlo recovery test), or is a converged optimizer being taken as proof of identification? Is any counterfactual computed by re-solving equilibrium rather than holding prices fixed? Is the conduct/distribution assumption flagged as load-bearing and untestable? (Hand off to structural-estimation.)