ab-testing

A/B Testing

When to Propose an A/B Test

A/B testing is for controlled experiments where you have two plausible approaches and want empirical evidence for which performs better. Use it when:

• Optimizing prompts -- "Does system prompt A produce better results than system prompt B?"
• Comparing approaches -- "Is caching strategy X faster than strategy Y?"
• Parameter tuning -- "Does temperature=0.3 produce more consistent output than temperature=0.7?"
• Validating improvements -- "Does the refactored code actually perform better than the original?"

Do NOT use for: safety-critical parameters, authentication settings, cost/rate limits, or anything that could compromise system integrity. These are enforced as forbidden parameters (see Safety Constraints below).

The Full Lifecycle

An A/B test moves through these states:

PROPOSED -> GRACE_PERIOD -> ACTIVE -> MONITORING -> CONCLUDED
                |                                       |
                v                                       v
             VETOED                                 REVERTED
                                                (auto: cost/time exceeded)

Step 1: Propose

3s ab-propose "temperature" "0.3" "0.7" "Lower temperature produces more consistent classification output"

Or via MCP:

ab_propose(
    param="temperature",
    variant_a="0.3",
    variant_b="0.7",
    hypothesis="Lower temperature produces more consistent classification output"
)

This creates the test in PROPOSED status and returns a test ID (UUID). The test is not yet running.

Step 2: Grace Period (Human Veto Window)

ab_start(test_id="<uuid>")

Moves the test to GRACE_PERIOD. This is a deliberate pause (default 30 minutes) that allows a human to review the proposal and veto it before any resources are spent. The grace period duration is informational -- enforcement is the caller's responsibility.

Why a grace period? Autonomous A/B testing without human oversight can waste budget on ill-conceived experiments. The grace period is a safety valve.