error-analysis

Error Analysis

Guide the user through trace-grounded failure analysis and dataset labeling.

Interactive Q&A protocol (mandatory)

Ask one question at a time using the structured question tool (loaded per the HARD-GATE above).

Example question structure:

Which data source should we analyze first?
A) Existing Truesight dataset
B) New dataset to upload
C) Unsure, list datasets first

Rules:

• One question per message during setup.
• Use the structured question tool for every question. Structure each with a short header, 2-4 options with labels and descriptions, and place the recommended option first. Do not add "(Recommended)" or similar annotations to option labels.
• Ask one follow-up if response is ambiguous.

Core workflow

1. Select or create dataset:
   • If dataset exists, use list_datasets.
   • If not, use upload_dataset.
2. Collect representative traces:
   • Target approximately 100 traces when possible.
   • Use random plus stratified coverage when volume is high.
3. Analyze row by row:
   • Use get_dataset_rows with pagination.
   • For each row, call suggest_error_notes.
4. Persist annotations:
   • Save _ts_error_notes and _ts_error_category with update_dataset_row.
5. Consolidate categories:
   • Run consolidate_error_categories.
   • Review mapping proposals, then apply with apply_category_mappings.
6. Prioritize fixes:
   • Report most frequent categories first.
   • Recommend next skill based on failure type:
     • create-evaluation for new evaluation coverage
     • review-and-promote-traces for judgment backlog
     • eval-audit for broader process gaps

Analysis heuristics

• Focus on first root failure in each trace, not every downstream symptom.
• Let categories emerge from observed traces, not pre-baked labels.
• Iterate categories after 20 traces, then relabel for consistency.
• Stop when recent traces no longer reveal new failure categories.