analysis-checkpoints

Analysis Checkpoints

Overview

Autonomy is the point of a good analysis loop — and also its biggest hazard. The same momentum that makes you productive fast lets you redefine the goal mid-flight without noticing: a debugging session quietly becomes a redesign, an outlier "obviously" gets dropped, a near-vs-far DiD silently becomes a triple-difference. Each step felt like progress; collectively, the user got an analysis they never agreed to.

Core principle: Loop autonomously toward the agreed goal. Never redefine the goal — the design, the sample, the spec, the estimand, the metric — behind the user's back. When execution wants to change any of those, that is a checkpoint, not a task: stop, surface it, let the user decide.

This is the execution-time form of "Think Before Coding": don't decide silently, surface the tradeoff. question-framing / pre-analysis-plan establish the agreed goal up front; this skill protects it while the work runs.

The line: your call vs. the user's call

The test is simple — does this change what is being estimated, on what data, or a number the user has already seen? If yes, it's the user's call. Run this one question on every decision you're about to make; the two lists below are just worked examples of "yes" (STOP) and "no" (proceed and report). Two sanctioned stops sit outside the test: the execution-mode choice (inline vs subagents) and discretionary-robustness selection — they stop because the plan assigns them to the user, not because they change what's estimated.

Decisions that REQUIRE a checkpoint — STOP and ask

• Design / identification strategy. Switching estimators or designs (near-vs-far DiD → triple-difference, OLS → IV, adding/removing a fixed effect that changes identification, changing the comparison group). This is the most commonly smuggled-in change.
• The structural model itself. For structural work: the utility/payoff form, the random-coefficient distribution, the conduct/equilibrium assumption, what's treated as a primitive vs. held fixed or calibrated, and the counterfactual design. These decide what is even being estimated and what the counterfactual means; they belong in the approved model card, so changing one mid-estimation — switching Nash–Bertrand to collusion, adding a random coefficient to make estimates behave — is a deviation, not a fix (structural-estimation).
• Any deviation from the framed question or the pre-analysis plan. The PAP exists precisely so these stops happen. A deviation is allowed — but disclosed and approved, never hidden.
• The estimand. ATE vs. ATT vs. LATE, the population, the time window.
• The sample. Dropping rows, filtering, winsorizing, trimming, excluding outliers, changing inclusion/exclusion rules, restricting to a subsample — and the additive direction too: adding to, re-pulling, or substituting the data source/vintage, and selecting among optimizer runs / seeds / starting values for the reported estimate.
• Materially different specifications or models where there's a real tradeoff (functional form, control set, clustering level, missing-data handling, imputation).
• Metric definition / units / grain. Redefining the numerator or denominator, changing the unit of observation.
• The scope of the robustness suite. Don't fan out an exhaustive menu of checks. Propose the ~3 that probe the main threat, with rationales, and get approval before running — robustness is an argument, not an inventory (executing-analysis-plans).
• Any reported or actionable number the user has seen — a result, headline total, or anything in a deliverable — that your change would move. (Echoed intermediates — row counts, quick chat diagnostics — don't stop the work: apply the change and report the old → new delta inline.)