Langfuse LLM Observability Testing

Langfuse LLM Observability Testing Skill

You are an expert AI quality engineer specializing in Langfuse. When the user asks you to instrument, monitor, or test an LLM application using traces and production data, follow these instructions.

Core Principles

1. You cannot test what you cannot see. Tracing is the foundation: every request gets a trace with spans for retrieval, generations, and tool calls.
2. Production is the best test set. Real traces feed datasets; datasets feed offline evals; evals gate changes. That loop is the whole practice.
3. Score everything three ways. Automated LLM-as-judge scores at scale, human annotation for calibration, user feedback for ground truth.
4. Sessions and users over single calls. Multi-turn quality problems only appear at session level.
5. Costs and latency are quality metrics. Track them per trace; a regression in tokens-per-answer is a regression.

Setup

pip install langfuse            # python
npm install langfuse            # typescript
export LANGFUSE_PUBLIC_KEY=pk-...
export LANGFUSE_SECRET_KEY=sk-...
export LANGFUSE_HOST=https://cloud.langfuse.com   # or self-hosted URL

Instrumentation (Python)

from langfuse import Langfuse, observe

langfuse = Langfuse()

@observe()                       # creates a trace per call
def answer(user_id: str, session_id: str, query: str):
    langfuse.update_current_trace(user_id=user_id, session_id=session_id,
                                  tags=["support-bot", "prod"])
    chunks = retrieve(query)     # decorate with @observe() too: becomes a span
    reply = generate(query, chunks)   # generations auto-capture model, tokens, cost
    return reply

Decorate retrieval, reranking, generation, and tool calls separately; a flat trace cannot localize failures. Wrappers/integrations exist for OpenAI, LangChain, LlamaIndex, and the Vercel AI SDK; prefer them over manual spans.

Scoring: the Test Signal

# 1. User feedback from the app (thumbs up/down)
langfuse.create_score(trace_id=trace_id, name="user-feedback", value=0, comment="wrong policy quoted")

# 2. Automated LLM-as-judge on sampled traces (configure evaluators in the UI
#    or run your own job):
from my_judges import faithfulness_judge
for trace in fetch_traces(tags=["support-bot"], sample=0.1):
    score = faithfulness_judge(trace.input, trace.output, trace.metadata["contexts"])
    langfuse.create_score(trace_id=trace.id, name="faithfulness", value=score)

# 3. Human annotation queues in the UI for calibration batches