wb-test-and-iterate

WB-Test & Iterate: How Will We Measure Success?

Overview

Test & Iterate defines HOW you'll know whether the product delivers on the promise made in the PR/FAQ. It establishes metrics, targets, and experiments BEFORE building — not after. This prevents the failure mode where teams build something, find metrics that make it look good in retrospect, and declare victory regardless of actual customer impact.

The key distinction in this stage: input metrics (what you control and do) vs output metrics (what customers experience and value). Input metrics are leading indicators — they move first. Output metrics are lagging indicators — they move later as a consequence of input metrics moving.

Your job is to identify the input metrics you can drive, predict which output metrics should respond, set specific numeric targets for both, and design experiments to validate the causal chain.

When to Use

• After PR/FAQ is approved (Stage 4) and before engineering begins
• When defining success criteria for a product or feature launch
• During annual planning to set measurable goals for initiatives
• When re-evaluating an existing product that isn't meeting expectations
• When leadership asks "how will you know this is working?"

Amazon Context

At Amazon, every significant initiative has a metrics deck that is reviewed weekly. Metrics are not chosen after launch to make the team look good — they are committed to before building, published in the PR/FAQ's Internal FAQ ("How will we measure success?"), and reviewed rigorously after launch.

Amazon's metrics philosophy:

• Input metrics over output metrics for operational control. You can't directly control "customer satisfaction" but you can control "page load time" which drives satisfaction.
• Controllable metrics over vanity metrics. "Total signups" is vanity if most users sign up and never return. "Weekly active users doing core action" is controllable.
• Leading indicators over lagging indicators. By the time revenue moves, it's too late to course-correct. Watch the metrics that predict revenue movement.