agent-triage CLAUDE.md

agent-triage

CLI tool, MCP server, and Node.js library for diagnosing AI agents in production. Extracts behavioral policies from system prompts, evaluates traces against them, generates diagnostic reports.

Architecture

Traces (JSON/LangSmith/OTel)
  → Normalize to NormalizedConversation
  → Extract policies from system prompt (or use existing policies.json)
  → Evaluate: 12 metrics + policy compliance per conversation
  → Diagnose: root cause turn, cascade chain, blast radius, fix
  → Aggregate: failure patterns, recommendations
  → Report: self-contained HTML with inline CSS/JS

Core Types

• NormalizedConversation — universal trace format, all connectors normalize to this (src/ingestion/types.ts)
• Policy — extracted behavioral rule with id, name, description, category, complexity (src/policy/types.ts)
• PolicyResult — per-conversation verdict (pass/fail/not_applicable) with evidence and failing turns (src/evaluation/types.ts)
• Diagnosis — root cause analysis: turn, agent, summary, impact, cascade chain, fix, blast radius (src/evaluation/types.ts)
• Report — final output: conversations, policies, metrics, failure patterns, recommendations (src/evaluation/types.ts)

Key Modules

• src/llm/client.ts — unified LLM client (OpenAI + Anthropic), use createLlmClient()
• src/llm/prompts.ts — all prompt templates, return structured JSON
• src/llm/json.ts — safe JSON parsing for LLM responses (handles code fences, control chars, word-number coercion)
• src/report/ — HTML report: generator.ts (entry), sections.ts (render functions), styles.ts (inline CSS/JS/icons)

Rules

ESM Imports

All imports use .js extensions even for .ts files. This is required by Node16 module resolution.

// Correct
import { foo } from "./bar.js";
// Wrong — will fail at runtime
import { foo } from "./bar";
import { foo } from "./bar.ts";

LLM Response Parsing

Never use raw JSON.parse() on LLM output. Always use safeJSONParse() from src/llm/json.ts.