per-category-modeling-backfire

Per-Category Modeling Backfire: When Splitting Data Hurts

Problem

A common "advanced" technique in Kaggle competitions is to train separate models for each category (e.g., per product family, per store, per region). While this seems intuitively correct—different categories have different patterns—it can backfire spectacularly when individual categories lack sufficient data volume.

Context / Trigger Conditions

Use when considering:

• Training separate GBDT models per product family, store, or region
• Per-category CV improves but leaderboard score degrades
• Each category has <100K rows but the full dataset has >1M rows
• Categories have highly imbalanced data volumes (some huge, some tiny)
• The global model already handles category differences via categorical features

Specific symptoms:

• Per-category CV: 0.367 (better) vs Global CV: 0.375 → Per-category LB: 2.10 vs Global LB: 1.86
• Small categories show high CV variance (>0.05 std across folds)
• Test predictions from per-category models have shifted mean (higher or lower than expected)

Solution

The Data Volume Threshold

Per-category modeling only works when each category has enough data to train a robust model.

Rows per category	Per-category modeling	Recommendation
<50K	Harmful — overfitting, high variance	Use global model
50K-200K	Risky — depends on problem complexity	Test both, compare LB
200K-1M	Usually beneficial	Per-category if CV confirms
>1M	Almost always beneficial	Per-category recommended

Why It Fails With Small Categories

1. Overfitting: 70K rows × 5-fold CV = 14K validation rows. Easy to overfit.
2. High variance: Small categories (BOOKS, HARDWARE) have CV std >0.05, meaning the model is unstable across folds.
3. Feature instability: Lag/rolling features with limited history don't generalize.
4. Loss of cross-category signal: Global model learns that "weekend → more sales" applies across ALL categories, amplifying this signal. Per-category models each must re-learn this from limited data.