research-playbook

Research Playbook

This is the literal procedure a frontier model follows to evaluate a technical choice and write a recommendation. This team writes no code. The order is the point: defining criteria AFTER seeing candidates is how motivated reasoning sneaks in, and it is the single most expensive mistake in research.

Operating loop

1. Read the PROJECT first — the recommendation lives in its constraints. A generic "A vs B" comparison is worthless; the same question has opposite answers in different projects. Before searching the web, read the repo:
   • Stack, language, framework, and their VERSIONS (from lockfiles / dependency files, not memory — a project on framework v2 can't use a library that needs v5).
   • Deployment target (serverless, container, edge, on-prem), team size, and what is ALREADY installed (an incumbent that half-solves the problem changes the math entirely).
   • What the user actually needs the tech to DO — restate it in one sentence, including any constraint they didn't say out loud (license, bundle size, sync/async, data residency).
2. Define 3–6 weighted decision criteria FROM those constraints, BEFORE looking at candidates. Write them down first. Weight them (e.g., "fit for async: high; maintenance health: high; migration cost: medium; license: gate"). Fixing criteria before candidates is what prevents you from reverse-engineering criteria to justify a favorite.
3. Enumerate candidates, INCLUDING "do nothing / use what's already installed." 2–4 real candidates plus the null option. The incumbent is always a candidate; skipping it biases toward change. If the question names candidates, start there and add any obvious omission.
4. Triangulate each candidate across independent source types, and record the DATE of every source:
   • Official docs — for the feature set and the CURRENT major version.
   • Changelog / release history — release cadence, is it still shipping?
   • GitHub issues/PRs — maintainer responsiveness, date of last release, open critical-bug count, "is this abandoned" signal.
   • Independent posts/benchmarks — for real-world failure modes docs omit. A claim from one source type is a lead, not a fact; confirm load-bearing claims from a second, independent source.
5. Build a tradeoff matrix where every cell cites evidence. Rows = candidates (including "do nothing"), columns = your criteria. No cell may be a vibe: each is a fact + source + date, or an explicit "unknown — could not verify." An empty or unsourced cell is a hole to fill, not a cell to guess.
6. Lead the report with the recommendation. First line names ONE winner. Then: the reversal condition ("choose B instead if <specific, checkable condition>"), your confidence (high/medium/low + why), and the migration cost FROM THE CURRENT STATE (not from zero). Rationale and matrix follow; they justify the call, they don't bury it.