agentsop-observability-setup

Observability Setup — Which Backend + One-Line Autolog

"Instrument before you need it. The cheapest debugging session is a trace you already have."

This is an ENHANCE overlay. The local skills [[langsmith]], [[phoenix]] and [[mlflow]] each teach one backend deeply. This skill sits one level up: it answers the question those skills cannot — which one, and how do I turn it on in a single line right now — then hands off to the chosen backend's own skill for depth.

---

1. 何时激活 (When to activate)

Activate at one of three moments — earlier is always cheaper:

Trigger	Signal
Starting any LM project	First dspy.LM / ChatOpenAI / LlamaIndex / agent graph in the repo, and no tracing wired yet
Before first deploy	About to ship an LM feature to real users with no trace UI — the AP-15 trap (§6)
"Why did it do that?" with no traces	A bug surfaced, you reach for history, and there is nothing recorded — you are debugging blind
Multi-component pipeline	Retriever + reranker + LLM + tools; per-call printing ([[agentsop-prompt-history-inspect]]) is no longer enough
Cost / latency regression	Need aggregate token & latency dashboards across runs, not a single printout

Do NOT activate when:

• You have already chosen a backend and just need its API — defer to [[langsmith]] / [[phoenix]] / [[mlflow]] directly.
• The task is classic ML experiment tracking with no LLM calls — that is plain MLflow, no decision needed.
• You only need to read one rendered prompt right now — that is [[agentsop-prompt-history-inspect]] (the cheaper first move).

The relationship to [[agentsop-prompt-history-inspect]]: that skill is the reactive first move (dump one prompt, no setup). This skill is the proactive layer — wire persistent tracing so the next "why" is answered by a trace that already exists, not a frantic re-run.

---

2. 核心心智模型 (Core mental model)

   Instrument BEFORE you need it
   ───────────────────────────────────────────────
   pick backend      one-line          verify         add eval
   by constraints →  autolog turns  →  a trace    →   hooks (judge,
   (stack/scale/      it on            actually        datasets,
    budget)           (~1 line)        landed          alerts)
   ───────────────────────────────────────────────
   cost of skipping each stage compounds: a missing trace at deploy
   becomes a multi-hour blind-debug later (AP-15, §6).