Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add magnus919/agent-skills --skill agent-evals-and-observabilitygit clone --depth 1 https://github.com/magnus919/agent-skillsWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/magnus919/agent-skills/agent-evals-and-observability)<a href="https://agentmods.dev/skills/magnus919/agent-skills/agent-evals-and-observability"><img src="https://agentmods.dev/badge/skills/magnus919/agent-skills/agent-evals-and-observability/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/magnus919/agent-skills/agent-evals-and-observability"><img src="https://agentmods.dev/badge/skills/magnus919/agent-skills/agent-evals-and-observability.svg" alt="Reviewed on agentmods" width="80" height="20"></a>- NVIDIA SkillSpector pass
What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00102 | $0.01235 |
| Opus 5 | $0.00051 | $0.00617 |
| Sonnet 5 | $0.00020 | $0.00247 |
| Haiku 4.5 | $0.00010 | $0.00123 |
Grade A, and why
agent-evals-and-observability scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 9d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 64 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Agent Evals and Observability
Evaluation asks whether behavior meets a defined criterion on a declared dataset or production sample. Observability supplies traces, logs, metrics, correlations, and diagnostic context. Use both; neither proves what the other does.
Workflow
- Define the decision, risk, task contract, trajectory contract, and unacceptable outcomes. Select evidence by harm, reversibility, and deployment stage, not a staged completeness scale.
- Create an immutable dataset version and manifest before comparing versions. Declare provenance, rights/consent, slices, fixtures, expected side effects, contamination risk, limitations, retention, and changelog.
- Select complementary graders that observe the claimed property. Use deterministic checks for observable mechanics; use execution/environment checks for state and side effects; use human, model-judge, pairwise/ranking, domain, and safety review where appropriate.
- Run the candidate and baseline under comparable conditions. Preserve run configuration, stochastic repeats where variability affects the decision, failures/timeouts, and trajectory evidence.
- Report a multidimensional profile and uncertainty. Use paired comparisons where possible; inspect slices, missingness, base rates, confounders, effect sizes, and multiple comparisons. Non-significance is not equivalence.
- Apply a risk-tiered release gate: hard safety, privacy, authorization, and side-effect invariants cannot be averaged away. Record authority, insufficient-evidence outcomes, rollback, and follow-up. Load release-engineering when this evidence must be incorporated into artifact promotion, deployment, rollback, or a broader release train.
- Instrument production with minimized, redacted telemetry. Feed verified incidents and near misses into cases after consent, transformation, and contamination review.
Stop when the supported decision, evidence gaps, residual risks, and responsible owner are recorded. Escalate rather than infer a pass when required evidence is unavailable or conflicts.
What ships with it
19 files beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.
- evals/evals.json 8.1 KB
- README.md 1.7 KB
- references/datasets.md 1.3 KB
- references/evaluation-design.md 1.6 KB
- references/framework-routing.md 957 B
- references/graders.md 3.9 KB
- references/metrics-and-statistics.md 3.3 KB
- references/opentelemetry-genai.md 1.0 KB
- references/production-observability.md 3.1 KB
- references/regression-and-release.md 3.1 KB
- references/source-index.md 2.3 KB
- references/synthetic-scenarios.md 1.5 KB
- references/trajectory-review.md 2.5 KB
- templates/dataset-manifest.md 545 B
- templates/eval-plan.md 849 B
- templates/grader-specification.md 537 B
- templates/release-gate.md 577 B
- templates/run-report.md 514 B
- templates/trace-review.md 494 B
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 9d ago Changed · +1 lines · +16 tokens per session 8d400a09e2e5
- 12d ago First seen · 63 lines · 86 tokens per session scan A f762eec35c09
agent-evals-and-observability is a skill published in the GitHub repository magnus919/agent-skills (76 stars, last pushed yesterday), licensed MIT. It adds 102 tokens to every session and 1,235 once invoked, about $0.0005 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.
Other skills, from other repositories
taiyi-test
A project workflow skill for verifying an implementation and producing a TEST.md record. TDD means writing a failing test, implementing the change, and then making the test pass; this skill checks that process and other regression cases.
qa
QA test your code changes by reading your git diff, choosing the right validation path for frontend/browser and backend changes, and reporting pass/fail with evidence.
smoke-test
Run smoke tests against a deployed or local app based on your git diff. Each test uses Skyvern browser tools (navigate, act, validate, screenshot) with Chrome DevTools MCP as fallback. Posts screenshot evidence as PR comments.
review
Enter the Review phase of CocoBrew. Aggregates findings from Code Quality Advisor, CocoCupper intelligence, spec compliance check, and decision coverage gate. Produces review.md with decision points. Requires developer approval before $ship can proceed.
test
Enter the Test phase of CocoBrew. Reads spec.md test requirements, generates test cases, executes SQL validation and quality checks, records results in test.md. Can be re-run without full rebuild. Requires Build phase completion.
meter-compare
Compare CocoMeter accuracy and cost results with correctness-first ordering. Usage: $meter compare .