kaggle-optimal-blending

Kaggle Optimal Blending: The 80/20 Rule

Problem

Re-ranking (k-reciprocal, Jaccard, etc.) is powerful but using 100% re-ranked results can lose valuable signal diversity. Equal-weight blending often underperforms.

Context / Trigger Conditions

Use when:

• Implementing post-processing for Re-ID, retrieval, or recommendation tasks
• Re-ranking alone doesn't give expected boost
• Ensemble of multiple models shows little improvement
• Kaggle score plateaus despite strong individual models

Solution

The 80/20 Optimal Blending Rule

Key insight from 0.938 mAP solution (Jaguar Re-ID):

blend_raw_ratio = 0.20  # 20% raw, 80% re-ranked

final_similarity = (
    (1 - blend_raw_ratio) * rerank_score +
    blend_raw_ratio * raw_score
)

Why 80/20?

• 80% re-ranking: Captures neighborhood structure, removes false positives
• 20% raw: Preserves genuine long-range similarities, maintains diversity
• This ratio emerged as optimal through experimentation (0.938 mAP)

Implementation

import numpy as np

def optimal_blend(similarity_matrix, reranked_matrix, raw_ratio=0.20):
    """
    Blend re-ranked and raw similarity matrices

    Args:
        similarity_matrix: Original NxM similarity matrix
        reranked_matrix: Re-ranked version (k-reciprocal, etc.)
        raw_ratio: Weight for raw score (default 0.20)

    Returns:
        Blended similarity matrix
    """
    return (1 - raw_ratio) * reranked_matrix + raw_ratio * similarity_matrix

Re-Ranking Parameters (Optimized for 0.938 mAP)

# k-Reciprocal Re-ranking parameters
k1 = 20   # First k for reciprocal neighbors
k2 = 6    # Second k for final re-ranking
lambda_param = 0.2  # Jaccard weight

Complete Post-Processing Pipeline

def post_process_for_reid(query_features, gallery_features):
    """
    Complete post-processing: TTA + QE + Re-ranking + Optimal Blend
    """
    # 1. TTA (Test Time Augmentation)
    # query_features = (f_original + f_flipped) / 2

    # 2. Query Expansion (optional)
    # Expand with top-k gallery samples
    qe_top_k = 3

    # 3. Compute raw similarities
    raw_sim = cosine_similarity(query_features, gallery_features)

    # 4. Re-ranking (k-reciprocal)
    reranked_sim = k_reciprocal_reranking(
        raw_sim,
        k1=k1,
        k2=k2,
        lambda_param=lambda_param
    )

    # 5. Optimal blending (THE KEY!)
    final_sim = optimal_blend(
        raw_sim,
        reranked_sim,
        raw_ratio=0.20  # Don't use 0.0 or 0.5!
    )

    return final_sim