Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add yonatangross/orchestkit --skill error-analysisgit clone --depth 1 https://github.com/yonatangross/orchestkitWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/yonatangross/orchestkit/error-analysis)<a href="https://agentmods.dev/skills/yonatangross/orchestkit/error-analysis"><img src="https://agentmods.dev/badge/skills/yonatangross/orchestkit/error-analysis/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/yonatangross/orchestkit/error-analysis"><img src="https://agentmods.dev/badge/skills/yonatangross/orchestkit/error-analysis.svg" alt="Reviewed on agentmods" width="80" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00061 | $0.03407 |
| Opus 5.5 | $0.00024 | $0.01363 |
| Sonnet 5.5 | $0.00012 | $0.00681 |
| Haiku 4.5 | $0.00006 | $0.00341 |
Grade A, and why
error-analysis scanned grade A with 1 finding against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 16d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Makes network callslowCapability
Not a fault in itself. Listed so you know the mod talks to something, and to what.
# Credentials ride in the Authorization header, so let curl enforce the scheme: How it starts
The opening of the file, as written. The whole thing — 194 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Error Analysis
Evals-first failure analysis for LLM applications. The method (Hamel Husain, Shreya Shankar) is qualitative research applied to traces: a human open-codes real failures, Claude axial-codes the notes into a named taxonomy, and only the recurring named modes earn an automated eval. Error analysis decides what to measure; it never writes an eval for a mode that has no name and no count.
When to Use
- An LLM feature (chatbot, RAG, agent, extractor) misbehaves and the team is guessing which evals to write
- You have production traces in Langfuse (or a JSONL export) and need the top failure modes with counts
- Eval scores exist but nobody trusts them because they were never grounded in real failures
- A judge or metric exists and its agreement with human labels is unknown
Do NOT use for Claude Code session errors (errors skill), failing CI runs (ci-debug), or writing the evaluators themselves once modes are named (testing-llm, ork:eval-runner).
Task Management (CC 2.1.16)
Multi-phase workflow: create tasks before Phase 1 and keep status current.
t_run = TaskCreate(subject="Error analysis: {target}", activeForm="Running error analysis on {target}")
t_pull = TaskCreate(subject="Pull traces", activeForm="Pulling traces")
t_open = TaskCreate(subject="Open-code failure notes", activeForm="Open-coding traces")
t_axial = TaskCreate(subject="Axial-code failure taxonomy", activeForm="Axial-coding notes")
t_tax = TaskCreate(subject="Write failure-taxonomy.md", activeForm="Writing taxonomy")
t_eval = TaskCreate(subject="Recommend evals + judge alignment", activeForm="Recommending evals")
TaskUpdate(taskId=t_open, addBlockedBy=[t_pull])
TaskUpdate(taskId=t_axial, addBlockedBy=[t_open])
TaskUpdate(taskId=t_tax, addBlockedBy=[t_axial])
TaskUpdate(taskId=t_eval, addBlockedBy=[t_tax])
Effort Scaling (CC 2.1.76)
| Effort | Trace pool | Notes | Outcome |
|---|---|---|---|
| low | 30 | human notes only, no Claude drafts | draft taxonomy |
| medium | 50 | drafts after 10 human notes | taxonomy + counts |
| high (default) | 100 | drafts after 30 human notes, saturation check | full deliverable |
What ships with it
4 files beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 16d ago Changed de4173b950b7
- 17d ago Changed · -1 lines 45ecd54b3056
- 21d ago First seen · 195 lines · 61 tokens per session scan A c402a757c248
error-analysis is a skill published in the GitHub repository yonatangross/orchestkit (295 stars, last pushed today), licensed MIT. It adds 61 tokens to every session and 3,407 once invoked, about $0.0002 per session on Opus 5.5. A static security scan graded it A with 1 finding (makes network calls). No closer match exists in the catalogue, so it is treated as the original; first seen 2026-09-25.
Other skills, from other repositories
examples-qa
Verify Instructor behavior through the repository's ./examples/ suite in pass, live record, or hermetic replay mode. Use when running selected examples or the corpus, capturing and reusing recorded LLM HTTP responses, diagnosing hub results, or distinguishing real errors, assertion failures, skipped examples, and…
credit-note-fixer
Fix the tiny credit-note formatting bug and rerun the exact targeted test command.
verify-omnigent
Drive Omnigent the way a user does and prove a behavior with recorded evidence, using an isolated server, runner, and mock model plus a feature map of every user entry point. Load before reproducing a user-facing bug, before claiming a fix works, or when reviewing whether a change covered every surface (session vs.…
resolve-author-fix
Find the root cause, implement a focused fix, and prove behavior with targeted tests and recordings.
resolve-repro-audit
Audit a recovered repro patch, assertions, and behavioral baseline before authoring or reviewing.
test
Enter the Test phase of CocoBrew. Reads spec.md test requirements, generates test cases, executes SQL validation and quality checks, records results in test.md. Can be re-run without full rebuild. Requires Build phase completion.