Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add hamelsmu/evals-skills --skill error-analysisgit clone --depth 1 https://github.com/hamelsmu/evals-skillsWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/hamelsmu/evals-skills/error-analysis)<a href="https://agentmods.dev/skills/hamelsmu/evals-skills/error-analysis"><img src="https://agentmods.dev/badge/skills/hamelsmu/evals-skills/error-analysis/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/hamelsmu/evals-skills/error-analysis"><img src="https://agentmods.dev/badge/skills/hamelsmu/evals-skills/error-analysis.svg" alt="Reviewed on agentmods" width="80" height="20"></a>- Socket pass
- Snyk pass
What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00053 | $0.01745 |
| Opus 5.5 | $0.00021 | $0.00698 |
| Sonnet 5.5 | $0.00011 | $0.00349 |
| Haiku 4.5 | $0.00005 | $0.00175 |
Grade A, and why
error-analysis scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 1mo ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
Copies of this mod
2 near-identical copies found in the catalogue:
- error-analysis — 100% identical, 0 lines differ
- error-analysis — 100% identical, 0 lines differ
How it starts
The opening of the file, as written. The whole thing — 165 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Error Analysis
Guide the user through reading LLM pipeline traces and building a catalog of how the system fails.
Overview
- Collect ~100 representative traces
- Read each trace, judge pass/fail, and note what went wrong
- Group similar failures into categories
- Label every trace against those categories
- Compute failure rates to prioritize what to fix
Core Process
Step 1: Collect Traces
Capture the full trace: input, all intermediate LLM calls, tool uses, retrieved documents, reasoning steps, and final output.
Target: ~100 traces. This is roughly where new traces stop revealing new kinds of failures. The number depends on system complexity.
From real user data (preferred):
- Small volume: random sample
- Large volume: sample across key dimensions (query type, user segment, feature area)
- Use embedding clustering (K-means) to ensure diversity
From synthetic data (when real data is sparse):
- Use the generate-synthetic-data skill
- Run synthetic queries through the full pipeline and capture complete traces
Step 2: Read Traces and Take Notes
Present each trace to the user. For each one, ask: did the system produce a good result? Pass or Fail.
For failures, note what went wrong. Focus on the first thing that went wrong in the trace — errors cascade, so downstream symptoms disappear when the root cause is fixed. Don't chase every issue in a single trace.
Write observations, not explanations. "SQL missed the budget constraint" not "The model probably didn't understand the budget."
Template:
| Trace ID | Trace | What went wrong | Pass/Fail |
|----------|-------|-----------------|-----------|
| 001 | [full trace] | Missing filter: pet-friendly requirement ignored in SQL | Fail |
| 002 | [full trace] | Proposed unavailable times despite calendar conflicts | Fail |
| 003 | [full trace] | Used casual tone for luxury client; wrong property type | Fail |
| 004 | [full trace] | - | Pass |
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 1mo ago First seen · 165 lines · 53 tokens per session scan A 2689164e29e3
error-analysis is a skill published in the GitHub repository hamelsmu/evals-skills (1,676 stars, last pushed 2mo ago), licensed MIT. It adds 53 tokens to every session and 1,745 once invoked, about $0.0002 per session on Opus 5.5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.
Other skills, from other repositories
debug-inference
Debug inference clients that use an attached provider and its native endpoint, including hosted APIs and host-local Ollama, vLLM, SGLang, TRT-LLM, LM Studio, or NIM. Use for provider attachment, endpoint policy, credential substitution, topology, and migration from the removed managed inference endpoint. Trigger…
spark-optimization
Optimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning. Use when improving Spark performance, debugging slow jobs, or scaling data processing pipelines.
notebooklm
Install, authenticate, troubleshoot, and operate Gemini Notebook through the notebooklm-py CLI or typed async Python API. Use for notebook and source management, grounded chat and research, and artifact generation or download when the user mentions Gemini Notebook, notebooklm-py, the notebooklm CLI, or its Python API.…
oh-my-posh
Install, configure, or troubleshoot Oh My Posh/ohmyposh: shell init, themes, segments, Nerd Font icons, and prompt setup on PowerShell, zsh, bash, or fish.
spark-engineer
Use when writing Spark jobs, debugging performance issues, or configuring cluster settings for Apache Spark applications, distributed data processing pipelines, or big data workloads. Invoke to write DataFrame transformations, optimize Spark SQL queries, implement RDD pipelines, tune shuffle operations, configure…
rocketride-debugging-pipelines
Use when a RocketRide pipeline run failed, errored, or produced wrong/empty output, and you need to find the failing node and fix it. Reads run status and execution traces, diagnoses the cause, and routes back to design or configuration. Also use directly when asked to debug a run.