model-training-and-evaluation

Model Training and Evaluation

Overview

A model's score is an estimate, and its quality depends on how it was estimated: the split that matches the data's structure, the baseline it is compared with, the tuning kept away from the final test, and the spread around the number. Gradient-boosted trees on tabular data are the strong default; anything more complex has to beat them on the same folds.

Core principle: every number you report answers "compared with what, measured how, and how sure" — baseline, CV scheme, spread.

1. Pick the CV scheme from the data

Data	Splitter
Independent rows, classification	StratifiedKFold(n_splits=5, shuffle=True, random_state=seed)
Independent rows, regression	KFold(n_splits=5, shuffle=True, random_state=seed)
Repeated entities (users, patients, stores)	GroupKFold / StratifiedGroupKFold with groups=
Time-ordered, forecasting or scoring the future	TimeSeriesSplit(n_splits=5, gap=horizon) or a rolling-origin split by date
Small data (< a few thousand rows)	RepeatedStratifiedKFold for a stabler mean and spread

The scheme mirrors production: a model that scores next month is validated on a later month than it trained on. Leakage rules apply throughout (leakage-safe-feature-engineering).

2. Baselines on the same folds

scoring = "average_precision"
baseline = cross_validate(DummyClassifier(strategy="prior"), X_train, y_train, cv=cv, scoring=scoring)
linear = cross_validate(make_linear(seed), X_train, y_train, cv=cv, scoring=scoring)
gbm = cross_validate(make_gbm(seed), X_train, y_train, cv=cv, scoring=scoring)

Always include the dummy and the simple rule from ml-problem-framing. A complex model that beats the linear one by less than the fold-to-fold spread is not better.

3. Tune inside cross-validation

Hyperparameter search runs on the training data with the same CV; the test set is not involved (rule test-set-discipline).

def objective(trial: optuna.Trial) -> float:
    params = {
        "learning_rate": trial.suggest_float("learning_rate", 1e-3, 0.3, log=True),
        "max_leaf_nodes": trial.suggest_int("max_leaf_nodes", 15, 255, log=True),
        "l2_regularization": trial.suggest_float("l2_regularization", 1e-8, 10.0, log=True),
    }
    return cross_val_score(make_gbm(seed, **params), X_train, y_train, cv=cv, scoring=scoring).mean()

study = optuna.create_study(direction="maximize", sampler=optuna.samplers.TPESampler(seed=seed))
study.optimize(objective, n_trials=50)