domain-knowledge-constraints-trap

Domain Knowledge Constraints Trap

Problem

Applying domain knowledge "constraints" or "rules" to filter training data can seem like a good idea for improving model quality, but often destroys model performance by changing the data distribution and removing valuable information.

Real Case: Adding 4 medical constraints to heart disease prediction:

• Expected: Improve model with medical domain knowledge
• Actual: CV AUC dropped from 0.970 to 0.943 (-2.8%)
• Root cause: Adversarial AUC changed from 0.501 → 0.661 (distribution shift)

Context / Trigger Conditions

Use this skill when:

• Working on ML projects with domain experts providing "rules"
• Considering filtering data based on physical/medical/logical constraints
• Adversarial validation shows train/test distribution change after filtering
• CV score decreases after adding "reasonable" constraints

Symptoms:

• Domain experts say "this data point is impossible"
• Filtering "anomalies" or "outliers" based on domain rules
• CV score drops after adding constraints
• Train/test distributions become different (adversarial AUC ≠ 0.5)

Common Trap Examples:

• Medical: "Heart rate can't exceed 220 - age" → Removes valid extreme cases
• Physical: "Temperature can't be negative" → Removes sensor errors AND valid extremes
• Business: "Customer can't spend >$10K/month" → Removes high-value outliers
• Temporal: "Events can't happen in the future" → Removes data entry errors AND valid edge cases

Solution

Step 1: Quantify Distribution Impact

Before applying constraints, check adversarial validation:

from sklearn.model_selection import StratifiedKFold
import lightgbm as lgb
from sklearn.metrics import roc_auc_score

def check_adversarial_auc(train_df, test_df, features):
    """Check if train/test distributions are similar"""
    adv_train = train_df[features].copy()
    adv_train['is_test'] = 0

    adv_test = test_df[features].copy()
    adv_test['is_test'] = 1

    adv_combined = pd.concat([adv_train, adv_test], axis=0)

    # Train classifier to distinguish train vs test
    skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
    oof_pred = np.zeros(len(adv_combined))

    for tr, val in skf.split(adv_combined, adv_combined['is_test']):
        train_data = lgb.Dataset(adv_combined.iloc[tr][features],
                                  label=adv_combined.iloc[tr]['is_test'])
        val_data = lgb.Dataset(adv_combined.iloc[val][features],
                                label=adv_combined.iloc[val]['is_test'])

        model = lgb.train({'objective': 'binary', 'verbosity': -1},
                          train_data, num_boost_round=100,
                          valid_sets=[val_data],
                          callbacks=[lgb.early_stopping(stopping_rounds=10)])

        oof_pred[val] = model.predict(adv_combined.iloc[val][features])

    auc = roc_auc_score(adv_combined['is_test'], oof_pred)
    return auc

# Before constraints
auc_before = check_adversarial_auc(train, test, features)
print(f"Adversarial AUC (before): {auc_before:.5f}")
# Output: 0.50111 ≈ 0.5 (distributions are similar) ✅