audit

/audit Command

Scan existing agent codebase for telemetry coverage and anti-patterns.

Arguments

• path: Optional path to audit (defaults to current directory)

Workflow

Step 1: Launch Codebase Analyzer

Launch codebase-analyzer agent to:

1. Detect agent framework(s)
2. Find existing observability SDKs
3. Map agent architecture
4. Identify entry points and key files

Step 2: Load Audit References

Based on detection, load:

• references/methodology/agent-observability-tiers.md
• references/frameworks/{framework}.md
• references/vendors/{vendor}.md (if vendor detected)
• references/anti-patterns/

Step 3: Evaluate Coverage

Check each instrumentation area:

Area	Priority	Status	Check
SDK Initialization	P0		Vendor SDK configured?
LLM Call Tracing	P0		Model calls have spans?
Tool Call Tracing	P0		Tool executions tracked?
Error Capture	P0		Errors logged with context?
Token Tracking	P1		Input/output tokens recorded?
Cost Attribution	P1		Cost calculated per call?
Agent Identity	P1		Agent name/type in spans?
Multi-Agent Links	P1		Parent-child relationships?
Memory/RAG Spans	P2		Retrieval tracked?
Human-in-Loop	P2		Approval workflows traced?
Evaluations	P2		Quality metrics captured?
Session Context	P2		User/session attached?

Step 4: Anti-Pattern Scan

Launch instrumentation-reviewer agent to check for:

Critical:

• Full prompt/response logging
• Secrets in traces
• Missing parent spans
• Blocking telemetry calls

Important:

• No token tracking
• Missing error context
• High cardinality attributes
• Inconsistent naming

Step 5: Generate Audit Report

Output comprehensive report with:

• Coverage scorecard
• Gaps ranked by priority
• Anti-patterns with file:line refs
• Recommended fixes
• Quick wins vs. larger efforts

Output Format

## Agent Observability Audit: [Project Name]

### Executive Summary
- **Framework:** [LangGraph]
- **Vendor:** [Langfuse v2.1]
- **Coverage Score:** [65%] (based on tier completion)
- **Critical Issues:** [2]
- **Recommended Actions:** [5]

### Existing Telemetry
| SDK/Vendor | Version | Location | Status |
|------------|---------|----------|--------|
| Langfuse | 2.1.0 | requirements.txt | Active |
| OpenTelemetry | 1.20 | pyproject.toml | Partial |

### Coverage Assessment

#### Tier 0: Foundation
| Check | Status | Notes |
|-------|--------|-------|
| SDK Init | PASS | langfuse.init() in main.py:12 |
| Root Span | FAIL | No agent-level span |
| Error Capture | PASS | Exception handler present |

#### Tier 1: Core Tracing
| Check | Status | Notes |
|-------|--------|-------|
| LLM Spans | PARTIAL | Only OpenAI, missing Anthropic |
| Tool Spans | FAIL | Tools not instrumented |
| Agent Loop | PASS | @observe decorator on run() |

#### Tier 2: Context
| Check | Status | Notes |
|-------|--------|-------|
| Token Tracking | PASS | usage in span attributes |
| Cost Attribution | FAIL | No cost calculation |
| User Context | PARTIAL | user_id but no session |

#### Tier 3: Multi-Agent
| Check | Status | Notes |
|-------|--------|-------|
| Parent-Child | FAIL | Child agents orphaned |
| Handoff Logging | FAIL | No delegation tracking |

#### Tier 4: Evaluation
| Check | Status | Notes |
|-------|--------|-------|
| Quality Scores | FAIL | Not implemented |
| Feedback Capture | FAIL | Not implemented |

### Coverage Score

[##########..........] 50% Overall