tabular-polynomial-features-breakthrough

Tabular Polynomial Features Breakthrough

Problem

After exhausting standard optimization techniques (hyperparameter tuning, ensembling), small tabular datasets often hit a performance plateau. Conventional wisdom suggests tree-based models like XGBoost should capture non-linear patterns natively, but this isn't always true for feature interactions.

Context / Trigger Conditions

Use this skill when:

• Small dataset: <10K samples, <50 original features
• Baseline plateau: After hyperparameter tuning, improvements are <0.005
• Tree models underperform: XGBoost/LightGBM scores lower than expected
• Feature interactions suspected: Domain knowledge suggests features combine non-linearly
• Model optimization exhausted: Grid search, Optuna yield minimal gains

Red flags that polynomial features may help:

Baseline XGBoost:           F1 = 0.687, LB = 0.805
Grid Search (depth=8):      F1 = 0.686, LB = 0.810  (+0.005) ← Diminishing returns
Ensemble (3 models):        F1 = 0.688, LB = 0.815  (+0.010) ← Minimal gain
Feature Selection (Top 10): F1 = 0.692, LB = 0.819  (+0.014) ← But high gap!

What this solves:

• Captures explicit feature interactions (x₁×x₂, x₁×x₃, etc.)
• Provides non-linear decision boundaries without complex model architectures
• Outperforms model tuning on small datasets where data > model complexity

Solution

Step 1: Standardize Features First

Critical: Polynomial features are scale-sensitive. Always standardize before expansion.

from sklearn.preprocessing import StandardScaler, PolynomialFeatures
import pandas as pd

# Load data
X_train = train_df.drop(columns=['target', 'id'])
X_test = test_df.drop(columns=['id'])
y_train = train_df['target']

print(f"Original features: {X_train.shape[1]}")  # e.g., 16

# Step 1: Standardize
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

Step 2: Generate Degree-2 Polynomial Features