research-experiments

Phase 3 — Experiments & ablations

Where the claims get tested and 30% of effort (and most variable cost) lands. The deliverable is evidence: a clean ablation matrix, honest baselines, and results that survive scrutiny — produced on local hardware under real rate limits and a finalized budget. Apply the full experimental-rigor protocol before the full run and again during reporting.

Step 1 — Pre-flight smoke tests = boundary calibration (do this BEFORE any full run)

A smoke test here is not "does it run once" — it discovers the operating envelope and proves robustness, especially for cost-triggered services (LLM APIs, data-vendor APIs). Run it small and cheap, then read off:

• Boundary discovery per service. Measure the real input/output token-length distribution for the actual call scenarios — set max_tokens from measurement, not a guess. Distinguish reasoning models (much larger output budgets, longer/variable latency — verify against the provider's current official model/API documentation) from standard ones. Record per-call latency and timeout tolerances for the specific models/providers in this project.
• Meaningfulness checks. Confirm function-calling / prompt framings return useful, parseable data on the large majority of calls. Measure and drive down parse-failure and excessive-retry rates; fix schemas/framings until results are reliable. A run that mostly retries or returns garbage is wasted budget.
• Config robustness. Validate configs across the scenario spread — long vs short inputs, rate-limit bursts, partial failures, empty/degenerate responses — with sane backoff and retry caps.
• Outputs feed two consumers. The measured I/O sizes + latencies + failure rates refine (a) the throttled/resumable run plan below and (b) the cost estimate that research-tracking uses for the budget-finalization gate. Record a go/no-go summary before the full run is authorized.
• A mock/fault-injection battery is necessary, not sufficient. Run it first (free: every role × every injected failure mode → a recovery matrix with valid fallbacks), then run a live battery of real calls per (model × role) with realistic contexts under a hard USD cap — only real calls reveal provider contracts. Earned: an all-green mock matrix was followed by three defects in the first 30 real calls: (i) reasoning-capable models spend the output budget on hidden reasoning and return an empty body with finish_reason=length (classify as truncation → grow the budget; never "fix" it by disabling reasoning — see research-code-review); (ii) structured-output schemas must sit in the intersection of provider constraints (strict modes want every property declared and required with nullable unions for "optional"; some providers reject an array-of-objects nested in an array-of-objects — keep depth-2 free-form data as strings) and be A/B-verified per provider; (iii) per-cell breaker resets in a battery so one dead role does not hide the others. Start output budgets generous and unified (largest observed need + headroom) and narrow from measured maxima; if the provider does not itemise reasoning tokens, budget and price on total output.
• Validity is not quality — plant ground truth before pinning any capability knob. A live battery's first-try-valid rate measures engineering health, not decision quality (with reasoning effectively off, every role still returned schema-valid answers), so it cannot choose an effort level, a model or a prompt version. Build a small planted-truth case bank per role (cases with a known cause, a transparent oracle for allocations, an ambiguous case whose right answer is abstain), score QUALITY against it, and run every knob that was pinned by argument (reasoning effort, context size, memory arm) as a sweep with the selection rule stated before the data — and its bias: with ~10 calls per cell a "within one SE of the best" rule defaults to the cheap level on ties, so print the argmax and the full table too, and treat a monotone gradient as the trigger for a confirmatory run on that role only. Agent-authored code is scored by running it in a subprocess sandbox on synthetic worlds (discrimination + robustness to empty/NaN input), never by static checks alone; keep the prompt's worked examples out of the case bank — a case that mirrors the example measures copying, not the capability. Earned: effort pins were "high by argument"; the first sweep showed flat quality on allocation roles at 3× the cost and latency, a gradient only on diagnosis, and one probe case contaminated by the prompt's own example. Three follow-ups the first sweep needed: (i) a flat curve on an easy case bank is not evidence — add a difficulty ladder (graded signal, fewer post-event points, an unsynchronised distractor cue) so accuracy can leave the ceiling; (ii) run a manipulation check first — a task with exact answers where the knob must show (puzzles for reasoning effort): if tokens and accuracy do not move, the setting is not reaching the model (one vendor's levels were inert through the aggregator) and no sweep on it means anything; (iii) the unit of analysis is model × setting, not role × setting — the same knob bent one model's curve and left another's byte-identical, so pin per role only after checking per model, and record a per-vendor table as the alternative.