Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add StamKavid/last-ds-mile --skill metric-selectiongit clone --depth 1 https://github.com/StamKavid/last-ds-mileWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/stamkavid/last-ds-mile/metric-selection)<a href="https://agentmods.dev/skills/stamkavid/last-ds-mile/metric-selection"><img src="https://agentmods.dev/badge/skills/stamkavid/last-ds-mile/metric-selection/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/stamkavid/last-ds-mile/metric-selection"><img src="https://agentmods.dev/badge/skills/stamkavid/last-ds-mile/metric-selection.svg" alt="Reviewed on agentmods" width="80" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00086 | $0.01439 |
| Opus 5 | $0.00043 | $0.00720 |
| Sonnet 5 | $0.00017 | $0.00288 |
| Haiku 4.5 | $0.00009 | $0.00144 |
Grade A, and why
metric-selection scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 10d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 88 lines — stays where its author put it; the contents beside it link to each section on GitHub.
metric-selection
Overview
The metric is not a technical afterthought — it's the operational definition of "good," and the wrong one can make an actively harmful model look great. This skill is a decision table, not a philosophy essay.
When to Use
- Picking the metric during
/ds-frame, or defending/re-checking it during/ds-evaluate. - The target is imbalanced (classification) or skewed (regression) and accuracy/plain RMSE feels wrong.
- NOT for: picking the validation split (see
ds-validate) — this skill is about how a prediction is scored, not how data is split for scoring. - Scope: tabular classification and regression. The ranking row is there because ranking metrics get confused with classification ones, not because this plugin covers recommenders. Forecasting metrics (MASE, sMAPE, pinball across horizons) and generation metrics are out of scope — see README → Scope.
Core Process
- Start from the decision this feeds (from
/ds-frame) — what does a false positive cost vs. a false negative? What does a $10,000 prediction error cost vs. a $1,000 one? - Match the problem type + cost asymmetry to a metric family in the table below.
- State explicitly why the chosen metric fits the decision — "we use F2 because missing a fraud case costs 5x more than a false alarm," not "F1 is standard."
Techniques/Patterns — metric decision table
| Problem type | Situation | Metric | Why |
|---|---|---|---|
| Binary classification | Balanced classes, symmetric cost | Accuracy, ROC-AUC | Both classes matter equally, straightforward to interpret |
| Binary classification | Imbalanced classes | PR-AUC (precision-recall AUC), not ROC-AUC | ROC-AUC is invariant to class balance — TPR and FPR are each computed within a class — so it keeps looking good while the model is unusable: at 600:1, an FPR of 1% is 6 false alarms for every true catch. PR-AUC uses precision, so it moves with that ratio |
| Binary classification | Asymmetric cost (e.g. missing fraud costs more than a false alarm) | F-beta (beta>1 weights recall higher; beta<1 weights precision higher), or a custom cost-weighted score | Directly encodes the real cost ratio instead of assuming false positives and negatives are equally bad |
| Binary classification | Need calibrated probabilities, not just ranking | Log loss / Brier score | Rewards a model for well-calibrated probabilities, not just correct ordering |
| Regression | Target spans multiple orders of magnitude (e.g. house prices) | RMSE/MAE on log(target) |
Keeps large-value errors from dominating the loss in relative terms |
| Regression | Outliers should not dominate the score | MAE (or Huber loss) over RMSE | RMSE squares errors, so a few large misses can swamp the average; MAE weighs every error linearly |
| Regression | Need an intuitive "% off" number for stakeholders | MAPE (with care — undefined/unstable near zero targets) | Communicates error in relative, business-readable terms |
| Regression | Cost of over- vs under-prediction is asymmetric (e.g. understaffing costs more than overstaffing) | Quantile (pinball) loss at a chosen service level, or an explicit asymmetric cost function | Symmetric RMSE/MAE assume a unit over-shoot and a unit under-shoot cost the same; a quantile target lets you deliberately bias predictions to the cheaper side and set the over/under rate on purpose, instead of bolting an arbitrary "buffer" onto a symmetric forecast |
| Ranking / recommendation | Position of correct items matters, not just presence | NDCG, MAP@k | Rewards getting the right answer near the top, not just somewhere in the list |
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 10d ago First seen · 88 lines · 86 tokens per session scan A 52fcc6f3e802
metric-selection is a skill published in the GitHub repository StamKavid/last-ds-mile (3 stars, last pushed 1mo ago), licensed MIT. It adds 86 tokens to every session and 1,439 once invoked, about $0.0004 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.
Other skills, from other repositories
developing-with-streamlit
Use for ALL Streamlit tasks: creating, editing, debugging, beautifying, styling, theming, optimizing, or deploying Streamlit apps. Also custom components, st.components.v2, HTML/JS/CSS work. Discovers and loads version-matched reference docs from the user's installed Streamlit (>=1.57). Triggers: streamlit, st.…
marimo-pair
Work inside the user's live marimo notebook from the code editor: run Python in the same kernel the user does, inspect live notebook state, and commit durable notebook changes through code mode. Use whenever you create, analyze, or improve the user's marimo notebook.
wax
Swift framework guidance for Wax on-device memory/RAG. Use when writing Swift code with the public Memory facade, experimental PhotoMemory / VideoMemory, BuiltInMultimodalEmbeddings, embedding providers, retrieval modes, or hybrid search. For agent operators using the Wax MCP server tools, use the separate wax-mcp…
wax-memory-maintenance
Maintain the Wax memory store used by the Hermes wax-memory provider.
ml-expert
Expert-level machine learning, deep learning, model training, and MLOps. Use when the user mentions machine learning, deep learning, neural networks, MLOps, or data science, or when the task involves Machine Learning Fundamentals, Data Preparation, or Model Training.
fast-dash
Build a Fast Dash web app from a Python function. Use when the user wants to turn a function into an interactive app, add a UI to an existing function, or build a dashboard / form / wizard. Fast Dash infers UI components from type hints, so a well-typed function becomes an app with one decorator.