kaggle-experiment-sop

Kaggle Experiment Standard Operating Procedure

Purpose: Every Kaggle experiment — whether a new competition, a kernel fork, or a parameter sweep — follows this SOP. It prevents wasted quota, repeated mistakes, and lost knowledge.

---

Phase 0: Pre-Experiment (10 minutes)

0.1 Recall Relevant Experience

□ AutoMem recall: query for competition name, task type, metric
□ File memory: grep ~/.claude/projects/*/memory/ for keywords
□ Skills: check skills/examples/ for matching trigger conditions
□ Dead ends: read feedback_no_recheck_confirmed_dead.md (if exists)

Pass criterion: Can name ≥1 past competition or skill relevant to current task. Fail action: Proceed with caution — first time on this task type.

0.2 Identify Competition Type

□ Read competition Overview + Evaluation + Rules
□ Classify into one of 6 types:

  Type 1 — Standard Tabular (CSV upload, deterministic score)
  Type 2 — Code Competition (notebook required, hidden test)
  Type 3 — Simulation (TrueSkill Bayesian, agent vs agent)
  Type 4 — Research (custom metric, domain-specific)
  Type 5 — Playground (synthetic data, swag prize)
  Type 6 — LLM Benchmark (open-source model, prompt engineering)

Pass criterion: Type identified. Strategy selected (see kaggle-competition-type-strategy). Why it matters: Simulation → don't re-submit. Code → find artifacts first.

0.3 Evaluate Public Kernel Landscape

□ kaggle kernels list --competition <name> --sort-by scoreDescending
□ Read top 5 kernel titles + first cell
□ Estimate: best_public_score × 0.6 = my_expected_baseline
□ Check: does best public use external data? artifacts? special tricks?
□ Record: best_public_score, best_public_approach

Pass criterion: Know the best public score and approach. Decision point:

• If best public > 0.95 × top LB → fork is the right strategy
• If best public < 0.80 × top LB → significant custom work needed
• If <5 public kernels → less proven, higher custom value