LLM Tracing and Observability Setup

LLM Tracing and Observability Setup

What this skill does

This skill sets up production-grade observability for LLM applications. Without tracing, debugging a broken LLM pipeline means guessing — you can't see what prompt was sent, what the model returned, which tool call failed, or why latency spiked. This skill configures the right tracing layer for your stack and shows what to instrument.

How to use

Claude Code / Cline

Copy this file to .agents/skills/llm-tracing-setup/SKILL.md in your project root.

Then ask:

• "Use the LLM Tracing Setup skill to add observability to our LangChain app."
• "Set up Langfuse tracing for our OpenAI API calls."

Provide:

• LLM framework in use (LangChain, direct API, LlamaIndex, custom)
• Preferred tracing backend (LangSmith, Langfuse, Helicone, or open to suggestions)
• Language (Python or TypeScript)
• Whether you need cost tracking, latency alerting, or user feedback collection

Cursor / Codex

Paste your LLM call code alongside these instructions and specify the tracing backend.

The Prompt / Instructions for the Agent

Step 1 — Choose a tracing backend

Backend	Best for	Cost model
LangSmith	LangChain / LangGraph apps	Free tier + usage
Langfuse	Any LLM stack, self-hostable	Free tier + open source
Helicone	OpenAI / Anthropic direct API	Per-request fee
OpenTelemetry + Jaeger	Full control, existing OTel infra	Self-hosted
Braintrust	Eval-heavy teams, prompt versioning	Per-event

Recommendation: Langfuse for most teams — framework-agnostic, self-hostable, free tier generous, good UI.

Step 2a — Langfuse setup (any stack)

# pip install langfuse
import os
from langfuse import Langfuse
from langfuse.decorators import observe, langfuse_context

os.environ["LANGFUSE_PUBLIC_KEY"] = "pk-lf-..."
os.environ["LANGFUSE_SECRET_KEY"] = "sk-lf-..."
os.environ["LANGFUSE_HOST"] = "https://cloud.langfuse.com"

langfuse = Langfuse()

# Decorate any function that calls an LLM
@observe()
def generate_response(user_query: str) -> str:
    # Automatically traces: input, output, latency, model
    response = openai_client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": user_query}]
    )
    return response.choices[0].message.content

# Add custom metadata
@observe()
def process_document(doc_id: str, query: str) -> str:
    langfuse_context.update_current_observation(
        metadata={"doc_id": doc_id, "pipeline_version": "v2.1"},
        tags=["document-qa", "production"]
    )
    return generate_response(query)