Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx agentmods add skills/growthxai/output/output-eval-error-analysisnpx skills add growthxai/output --skill output-eval-error-analysisgit clone --depth 1 https://github.com/growthxai/outputWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/growthxai/output/output-eval-error-analysis)<a href="https://agentmods.dev/skills/growthxai/output/output-eval-error-analysis"><img src="https://agentmods.dev/badge/skills/growthxai/output/output-eval-error-analysis.svg" alt="Measured on agentmods" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00038 | $0.02471 |
| Opus 5 | $0.00019 | $0.01236 |
| Sonnet 5 | $0.00008 | $0.00494 |
| Haiku 4.5 | $0.00004 | $0.00247 |
Grade A, and why
output-eval-error-analysis scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 6d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 260 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Error Analysis for Workflow Evaluation
Overview
Review real workflow traces and categorize how your workflow fails before writing any evaluators. Evaluators built without error analysis target generic qualities ("is this good?") instead of the specific ways your workflow actually breaks. This skill walks you through the process.
When to Use
- Starting a new eval project for an existing workflow
- Production quality has dropped and you need to understand why
- After significant prompt, model, or pipeline changes
- Before building your first evaluator for a workflow
Step 1: Collect Traces
Gather 50-100 representative workflow executions. More traces = more reliable failure categories.
From recent runs
List recent workflow executions and pull their traces:
# List recent runs for a workflow
npx output workflow runs list <workflowName>
# Pull a specific trace as JSON
npx output workflow debug <workflowId> --json
From production (bulk download)
Download production traces directly into dataset YAML files:
# Download up to 20 recent traces as dataset files
npx output workflow dataset generate <workflowName> --download --limit 20
This creates YAML files in tests/datasets/ with the input and last_output fields populated from real executions.
From scenario-driven generation
If production traces are sparse, generate traces from scenario inputs:
# Generate a dataset from a scenario file
npx output workflow dataset generate <workflowName> basic --name basic_trace
# Generate from inline JSON
npx output workflow dataset generate <workflowName> --input '{"topic": "AI safety"}' --name ai_safety_trace
Run enough inputs to get 50+ traces. Prioritize diversity over volume — vary inputs across the dimensions you expect to matter.
Step 2: Review Traces Individually
Review each trace one at a time. For each trace, record:
| Field | What to write |
|---|---|
| Trace ID | The workflow execution ID |
| Verdict | Pass or Fail (binary — no "partial" at this stage) |
| Root cause | If Fail: what specifically went wrong and why |
| Notes | Anything surprising or worth remembering |
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 6d ago First seen · 260 lines · 38 tokens per session scan A fc18ff302158
output-eval-error-analysis is a skill published in the GitHub repository growthxai/output (435 stars, last pushed yesterday), licensed Apache-2.0. It adds 38 tokens to every session and 2,471 once invoked, about $0.0002 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.
Other skills, from other repositories
technical-writing
Write, edit, review, or audit user-facing documentation for the eve repository. Use for changes under docs/, documentation tied to eve APIs or CLI behavior, docs work based on Slack or support feedback, and requests to make eve docs clearer, more natural, or less AI-patterned while verifying claims against current…
optimizing-clickhouse-and-hogql-queries
Workflow for optimizing ClickHouse and HogQL queries. Use when a HogQL query, query runner, insight, or report is too slow; when a hand-written ClickHouse query (via syncexecute or in a migration) is too slow; when ClickHouse times out or hits memory limits; when investigating a slow system.querylog row; or when…
dynamic-workflows
Designs and runs task-specific JavaScript harnesses with the workflow tool. Use for broad, long-running, highly structured, or adversarial work that benefits from many isolated agents: exhaustive audits, root-cause investigations, research, large triage queues, competing proposals, repeated verification, and…
adding-product-alerting
Recommended repo-engineering guide when adding alerting to a PostHog product or extending the shared alerts platform. Routes lifecycle state machines, AlertPolicy, destinations, HogFunction dispatch, email, fixed-cadence and calendar scheduling, insight evaluation, the AlertWizard, and shared alert editor components.…
adding-mcp-store-servers
Add a third-party MCP server (Linear, Notion, GitHub, ...) to the PostHog MCP store catalog. Use when asked to "add X to the MCP store", expand the MCP server marketplace, or fix a broken catalog entry. Covers finding the vendor's remote MCP endpoint, probing it (handshake, OAuth discovery, DCR), authoring the catalog…
subagent-orchestration
How and when to delegate work to subagents via the subagent tool (Explore, Plan, General). Use when a task involves codebase recon, implementation planning, or actual code changes that would benefit from an isolated context window instead of doing it all inline.