debug-evals

Debug eval runs in Logfire

Eval runs (evaluations/) ship spans to Logfire under service_name = 'evals'. This skill finds a run, surfaces its metrics and failures, and drills into a single case. Read-only.

How to query

1. Confirm the current schema with mcp__logfire__query_schema_reference (spans and logs share the records table).
2. Run SQL with mcp__logfire__query_run (query + project: "haiku" + start_timestamp/end_timestamp, max 14 days). The remote MCP is org-scoped, so project is required; eval runs land in project haiku. The same SQL works pasted into Logfire's Explore UI.
3. Read span attributes as JSON: attributes->>'key', nested as attributes->'a'->'b'->>'c'. Cast when needed: (...)::float, (...)::int.
4. Hand back a clickable trace with mcp__logfire__project_logfire_link(trace_id, project="haiku").

When the Logfire MCP is not available

The mcp__logfire__* tools are not loaded in every session. The HTTP query API is the fallback and needs no MCP:

POST https://logfire-eu.pydantic.dev/v2/query      # EU projects
POST https://logfire-us.pydantic.dev/v2/query      # US projects
Authorization: Bearer <api-key>
Content-Type: application/json
{"sql": "...", "min_timestamp": "2026-08-23T00:00:00Z"}

• min_timestamp is mandatory and silently bounds every result. Too recent a value is indistinguishable from "no data".
• Read tokens are being replaced by API keys (pylf_v2_<region>_...). Same Authorization: Bearer header; the region is in the prefix.
• Keep the key in ~/.logfire-read-key (mode 600) and read it from there so it never lands in a transcript. Helper next to this skill: .claude/skills/debug-evals/lf-query.sh "<SQL>" [min_ts].
• The API is project-scoped. A key for the wrong project authenticates fine and returns zero rows — it does not error. Diagnose in this order:
  1. wrong region → HTTP 401 Invalid read token on the other host;
  2. wrong project → auth succeeds, count(*) over months is 0;
  3. right project → SELECT service_name, count(*) ... GROUP BY service_name shows evals, haiku-rag, haiku-ingester. Eval runs live in project haiku. There is an empty project named evals, which is the natural wrong guess.
• The API caps returned rows and does not say so. Aggregate server-side (count(*), avg(...), sum(CASE WHEN ...)) rather than pulling rows and counting them locally. A pass rate computed from a clipped page is wrong and looks fine.