adversarial-validation-kaggle

Adversarial Validation Implementation

Problem

Adversarial validation is a powerful technique for improving model generalization, but it's frequently implemented incorrectly. The wrong approach can lead to worse performance instead of better.

Real Case: Initial implementation used "train vs real data" (wrong) and "percentile-based filtering" (wrong), resulting in 0.90457 AUC vs baseline 0.95513. Corrected implementation (train vs test, sort-based) achieved 0.97058 AUC.

Performance Impact:

• Wrong method (train vs UCI): 0.90457 AUC (-5% vs baseline)
• Correct method (train vs test): 0.97058 AUC (+1.5% vs baseline)

Context / Trigger Conditions

Use adversarial validation when:

• Working with Kaggle Playground Series (synthetic data)
• Training and test distributions may differ
• Need to improve model generalization
• OOF score is good but LB score drops

Symptoms you need adversarial validation:

• High OOF AUC (>0.95) but much lower LB AUC
• Model overfits training distribution
• Synthetic data has GAN artifacts
• Large dataset (>100K) with potential low-quality samples

Common misconceptions:

• ❌ "Adversarial validation distinguishes real vs synthetic data"
• ✅ "Adversarial validation distinguishes training vs test distribution"

Solution

Core Principle

Goal: Identify and keep training samples that are most similar to the test set. This ensures your model trains on data that matches the evaluation distribution.

Step 1: Prepare Adversarial Dataset

# Combine train and test sets
adv_train = train[features].copy()
adv_train['is_test'] = 0  # Label: training set

adv_test = test[features].copy()
adv_test['is_test'] = 1  # Label: test set

adv_combined = pd.concat([adv_train, adv_test], axis=0, ignore_index=True)

Critical: Distinguish train vs test, NOT train vs real/external data!

Step 2: Train Adversarial Classifier

from sklearn.model_selection import StratifiedKFold
import lightgbm as lgb

lgb_params = {
    'objective': 'binary',
    'metric': 'auc',
    'num_leaves': 31,
    'max_depth': 6,
    'learning_rate': 0.05,
    'n_estimators': 500,
    'verbosity': -1,
    'n_jobs': 10
}

skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
oof_pred = np.zeros(len(adv_combined))

for tr, val in skf.split(adv_combined, adv_combined['is_test']):
    train_data = lgb.Dataset(adv_combined.iloc[tr][features],
                              label=adv_combined.iloc[tr]['is_test'])
    val_data = lgb.Dataset(adv_combined.iloc[val][features],
                            label=adv_combined.iloc[val]['is_test'])

    model = lgb.train(lgb_params, train_data, num_boost_round=500,
                      valid_sets=[val_data],
                      callbacks=[lgb.early_stopping(stopping_rounds=50)])

    oof_pred[val] = model.predict(adv_combined.iloc[val][features])

auc_score = roc_auc_score(adv_combined['is_test'], oof_pred)
print(f"Adversarial AUC: {auc_score:.5f}")