kaggle-submission-format-by-metric

Match Submission Format to Evaluation Metric

Problem

A perfect model can score near-random on the public leaderboard if the submission file format doesn't match the metric's expectations. The most common mistake: submitting 0/1 thresholded labels for a metric that needs continuous probabilities.

Real failure case (2026-06-14, S6E2 Heart Disease):

• Model: AutoGluon best_quality ensemble, OOF AUC 0.95554
• Submission A: submission_autogluon.csv (0/1 thresholded via predictor.predict())
  • Public LB: 0.88403 ❌ (looked like complete model failure)
  • Private LB: 0.88643
• Submission B: submission_autogluon_proba.csv (continuous via predictor.predict_proba())
  • Public LB: 0.95357 ✅
  • Private LB: 0.95510
• Same model, same OOF — only the submission format differed. 0.07 LB drop from thresholding alone.

Context / Trigger Conditions

Use this skill when:

• About to submit the final (or any) prediction to a Kaggle competition
• AutoGluon / sklearn / xgboost default predict() returns hard labels for classification
• Competition metric is ranking-based (see list below)
• CV score is great but LB score is suspiciously low
• You see sample_submission.csv with 0.0/1.0 values (those are the target format, NOT necessarily the submission format)

DO NOT threshold for ranking-based metrics:

• roc_auc, auc — area under ROC; needs continuous scores
• auc_mu — multi-class AUC; needs full probability matrix
• log_loss — penalizes confident wrong answers; needs probabilities
• MAP, NDCG — ranking metrics; need scores
• brier_score — squared error on probabilities
• mean_columnwise_auc — column-wise AUC
• rmse on log-target (RMSLE) — for log-transformed regression, submit log predictions directly

DO threshold (or round) for these metrics:

• accuracy — predicted class label
• f1, precision, recall — predicted class label
• quadratic_kappa — rounded integer (for ordinal)
• mae, rmse on raw target — continuous regression predictions are fine, but rounding is harmless