recommender-evaluation

Recommender Evaluation

This skill defines how the Vinyl Record Store recommender is measured. It exists because recommender quality is judged on ranking and catalog health, not MSE/RMSE — applying regression metrics to a top-k recommender is a classic, grade-costing mistake.

When to use

• You are about to compute or report a number about recommendation quality.
• You are designing the train/test split or deciding what counts as "relevant."
• You are comparing two algorithms and need a fair, side-by-side table.
• You are writing the Evaluation section of a CSX4207 report or slide.

Step 0 — Define "relevant" before touching metrics

Pin this down explicitly and write it in the report:

• Explicit ratings: "relevant" usually = rating ≥ threshold (e.g., ≥ 4 of 5).
• Implicit feedback: "relevant" = user interacted (play/purchase) in the held-out period; for ranking metrics, consider only items the user hasn't already consumed from training.

Ambiguity here invalidates every downstream number.

Step 1 — Split without leakage

• Leave-one-out per user (small data): hold out each user's most recent (or a random one) interaction for test; train on the rest. Standard for HitRate@k / NDCG@k on sparse academic datasets.
• Temporal split (preferred when timestamps exist): train on interactions before time T, test on after T. Closest to production reality.
• Never random row-shuffle split that lets a user appear in both train and test with overlapping context — it leaks and inflates every metric.
• For top-k ranking eval, sample negatives (items the user didn't interact with) to rank against the held-out positive, or rank against the full catalog (more honest, more expensive). State which.

Step 2 — Ranking-accuracy metrics (report at least NDCG@k + MAP@k)

For a user u, let the top-k recommendation list be R_k(u) and the set of relevant items be Rel(u).

• Precision@k = |Rel(u) ∩ R_k(u)| / k
• Recall@k = |Rel(u) ∩ R_k(u)| / |Rel(u)|
• HitRate@k = 1 if |Rel(u) ∩ R_k(u)| ≥ 1 else 0 (mean over users)
• MRR (Mean Reciprocal Rank) = mean over users of 1 / rank_of_first_relevant
• AP@k (Average Precision) = (1 / min(k, |Rel(u)|)) · Σ_{i=1..k} Precision@i · rel(i), where rel(i)=1 if item at rank i is relevant. MAP@k = mean of AP@k over users.
• DCG@k = Σ_{i=1..k} rel_i / log2(i + 1) (use 2^rel − 1 if graded relevance). IDCG@k = DCG of the ideal ordering. NDCG@k = DCG@k / IDCG@k ∈ [0,1].