agent-observability

Agent Observability

Make agent behavior explainable from request entry through model, retrieval, tool, handoff, and response spans.

Use when

• Add telemetry to a new or existing agent workflow.
• Diagnose slow, costly, incorrect, looping, or failed executions.
• Define dashboards, alerts, service-level indicators, or audit evidence.
• Standardize traces across models, tools, and orchestration frameworks.

Inputs

Collect the workflow graph, runtime boundaries, incident questions, traffic and failure expectations, telemetry stack, data classification, retention policy, sampling limits, and owners. State what cannot be observed.

Output contract

Produce:

1. An observability objective and system boundary.
2. A trace and event schema with identifiers, span taxonomy, attributes, and redaction rules.
3. Metrics with definitions, units, dimensions, and ownership.
4. Dashboard and alert specifications tied to user impact.
5. A sampling, retention, access, and cost plan.
6. An investigation runbook and instrumentation verification results.

Workflow

1. Start with operational questions such as “Which tool causes timeouts?” or “Why did cost per resolved task rise?” Do not collect fields without a decision use.
2. Define one trace per user-visible attempt. Create spans for model calls, retrieval, tools, handoffs, approvals, retries, and final validation. Preserve parent-child relationships and propagate a correlation identifier across queues.
3. Record stable semantic fields. Include version identifiers, status, timing, token and cost measures, retry counts, tool names, policy outcomes, and evaluation tags when available. Read trace-schema.md before defining attributes.
4. Separate content from metadata. Default to content-free telemetry; allow prompt or response capture only through explicit authorization, redaction, access controls, and retention limits.
5. Derive a small set of service indicators: task success, critical-policy violations, end-to-end latency, tool failure rate, escalation rate, and cost per completed task. Define denominators and treatment of cancellations and timeouts.
6. Build dashboards from user outcome to dependency detail. Alert on actionable sustained impact, not individual noisy spans, and attach an owner and runbook.
7. Control cardinality, sampling, and storage cost. Retain all critical failures when permitted; use head or tail sampling for normal traffic without losing rare error classes.
8. Test trace propagation, redaction, retry linkage, clock handling, and degraded telemetry behavior before relying on the data.