error-analysis

Error Analysis

Guide the user through reading LLM pipeline traces and building a catalog of how the system fails.

Overview

1. Collect ~100 representative traces
2. Read each trace, judge pass/fail, and note what went wrong
3. Group similar failures into categories
4. Label every trace against those categories
5. Compute failure rates to prioritize what to fix

Core Process

Step 1: Collect Traces

Capture the full trace: input, all intermediate LLM calls, tool uses, retrieved documents, reasoning steps, and final output.

Target: ~100 traces. This is roughly where new traces stop revealing new kinds of failures. The number depends on system complexity.

From real user data (preferred):

• Small volume: random sample
• Large volume: sample across key dimensions (query type, user segment, feature area)
• Use embedding clustering (K-means) to ensure diversity

From synthetic data (when real data is sparse):

• Use the generate-synthetic-data skill
• Run synthetic queries through the full pipeline and capture complete traces

Step 2: Read Traces and Take Notes

Present each trace to the user. For each one, ask: did the system produce a good result? Pass or Fail.

For failures, note what went wrong. Focus on the first thing that went wrong in the trace — errors cascade, so downstream symptoms disappear when the root cause is fixed. Don't chase every issue in a single trace.

Write observations, not explanations. "SQL missed the budget constraint" not "The model probably didn't understand the budget."

Template:

| Trace ID | Trace | What went wrong | Pass/Fail |
|----------|-------|-----------------|-----------|
| 001      | [full trace] | Missing filter: pet-friendly requirement ignored in SQL | Fail |
| 002      | [full trace] | Proposed unavailable times despite calendar conflicts | Fail |
| 003      | [full trace] | Used casual tone for luxury client; wrong property type | Fail |
| 004      | [full trace] | - | Pass |