controlled-submission-experiment

Controlled Submission Experiment

Problem

When multiple changes are bundled into a single submission (new features + new model + new post-processing), it's impossible to tell which change helped or hurt. A submission that scores worse than baseline might actually contain valuable model improvements hidden by a post-processing regression.

Symptoms

• New submission scores worse than baseline despite "better" CV
• Multiple changes made simultaneously (features + model + post-processing)
• Zero ratio in submission changes significantly (>2%) between versions
• Can't explain WHY the score changed in a specific direction

Solution

The Controlled Experiment Pattern

Isolate one variable at a time by creating multiple submissions from the same model output:

Baseline:  Model_A + Postproc_A  → Score_A
New:       Model_B + Postproc_B  → Score_B
Control:   Model_B + Postproc_A  → Score_C  (KEY!)

Interpretation matrix:

Pattern	Model B	Postproc B	Conclusion
Score_C > Score_A, Score_B ≈ Score_C	Better	Neutral	Model B is better, postproc doesn't matter
Score_C > Score_A, Score_C >> Score_B	Better	Worse	Model B is better BUT postproc B regresses
Score_C < Score_A, Score_B ≈ Score_C	Worse	Neutral	Model B is genuinely worse
Score_C ≈ Score_A, Score_B >> Score_A	Neutral	Better	Postproc B carries the improvement
Score_C > Score_A, Score_B > Score_C	Better	Better	Both improve, model B slightly more

Implementation

import pandas as pd
import numpy as np

# Step 1: Load both submission predictions (before post-processing)
r10_raw = pd.read_csv("submission_r10_raw.csv")  # baseline raw predictions
r11b_raw = pd.read_csv("submission_r11b_raw.csv")  # new raw predictions

# Step 2: Identify disputed predictions
merged = r10_raw.merge(r11b_raw, on="id", suffixes=("_r10", "_r11b"))
disputed = merged[
    (merged["sales_r11b"] == 0) & (merged["sales_r10"] > 0)
]
print(f"Disputed rows: {len(disputed)}")
print(f"R10 values in disputed: mean={disputed['sales_r10'].mean():.2f}")

# Step 3: Create controlled submission
# Use Model_B predictions but with Postproc_A logic
controlled = r11b_raw.copy()
# Apply only the baseline post-processing
controlled.loc[controlled["sales"] < 0.1, "sales"] = 0
controlled.to_csv("submission_r11c_controlled.csv", index=False)