Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add sawrus/agent-guides --skill model-monitoringgit clone --depth 1 https://github.com/sawrus/agent-guidesWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/sawrus/agent-guides/model-monitoring)<a href="https://agentmods.dev/skills/sawrus/agent-guides/model-monitoring"><img src="https://agentmods.dev/badge/skills/sawrus/agent-guides/model-monitoring/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/sawrus/agent-guides/model-monitoring"><img src="https://agentmods.dev/badge/skills/sawrus/agent-guides/model-monitoring.svg" alt="Reviewed on agentmods" width="80" height="20"></a>- NVIDIA SkillSpector pass
What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00000 | $0.00289 |
| Opus 5 | $0.00000 | $0.00144 |
| Sonnet 5 | $0.00000 | $0.00058 |
| Haiku 4.5 | $0.00000 | $0.00029 |
Grade A, and why
model-monitoring scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 9d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
What it actually says
Skill: Model Monitoring
When to load
When setting up monitoring for a deployed model or responding to drift alerts.
Monitoring Dimensions
1. Operational health
- Latency: p50, p95, p99
- Error rate: prediction failures, input validation failures
2. Data drift (vs training baseline)
- PSI (Population Stability Index) per feature
- PSI > 0.2 = significant shift → retrain likely needed
3. Model quality (when labels available)
- Accuracy metrics after ground truth arrives
- Business outcome correlation
PSI Drift Detection
def calculate_psi(expected: np.ndarray, actual: np.ndarray, buckets: int = 10) -> float:
"""PSI < 0.1: stable. 0.1-0.2: monitor. > 0.2: retrain."""
breakpoints = np.percentile(expected, np.linspace(0, 100, buckets + 1))
exp_counts = np.clip(np.histogram(expected, breakpoints)[0] / len(expected), 1e-4, None)
act_counts = np.clip(np.histogram(actual, breakpoints)[0] / len(actual), 1e-4, None)
return np.sum((act_counts - exp_counts) * np.log(act_counts / exp_counts))
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 9d ago First seen · 33 lines · 0 tokens per session scan A dd29273e2f3e
model-monitoring is a skill published in the GitHub repository sawrus/agent-guides (17 stars, last pushed 12d ago), licensed MIT. It costs nothing until one of its globs matches a file; then it loads 289 tokens. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-09-03.
Other skills, from other repositories
advanced-evaluation
This skill should be used when the user asks to "implement LLM-as-judge", "compare model outputs", "create evaluation rubrics", "mitigate evaluation bias", or mentions direct scoring, pairwise comparison, position bias, evaluation pipelines, or automated quality assessment.
agents-v2-py
Build container-based Foundry Agents with Azure AI Projects SDK (ImageBasedHostedAgentDefinition). Use when creating hosted agents with custom container images in Azure AI Foundry.
ai-ml
AI and machine learning workflow covering LLM application development, RAG implementation, agent architecture, ML pipelines, and AI-powered features.
ai-agent-development
AI agent development workflow for building autonomous agents, multi-agent systems, and agent orchestration with CrewAI, LangGraph, and custom agents.
arrowspace
Spectral vector search using graph Laplacian eigenstructure. Use when cosine/L2 similarity misses latent structure in your embeddings.
ai-engineer
Build production-ready LLM applications, advanced RAG systems, and intelligent agents. Implements vector search, multimodal AI, agent orchestration, and enterprise AI integrations.