langsmith_observability

LLM Observability with LangSmith

This rule provides guidelines for setting up and utilizing LangSmith for LLM-native observability, which is crucial for debugging and monitoring non-deterministic LLM applications.

1. Environment Setup

To enable LangSmith tracing, set the following environment variables:

export LANGSMITH_TRACING=true
export LANGSMITH_API_KEY=<your-api-key>
export LANGSMITH_PROJECT=<your-project-name>

Note: LANGSMITH_PROJECT is supported in JS SDK versions >= 0.2.16; otherwise, use LANGCHAIN_PROJECT.

2. Tracing LLM Calls and Chains

LangSmith allows you to trace individual LLM calls and entire application chains.

2.1 Basic Tracing with traceable

Wrap your functions or methods with traceable to automatically log their execution as LangSmith runs. This is applicable even if you are using @google/genai or other LLM providers.

import { traceable } from "langsmith/traceable";
import { GoogleGenAI } from '@google/genai';

// Initialize GoogleGenAI client (assuming environment variables are set)
const genAI = new GoogleGenAI();

const tracedGenerateContent = traceable(
  async function generateContentWithTracing(modelName: string, prompt: string) {
    const response = await genAI.models.generateContent({
      model: modelName,
      contents: [{ role: 'user', parts: [{ text: prompt }] }],
    });
    return response.text;
  },
  { run_type: "llm" } // Designate as an LLM run
);

// Example usage:
// await tracedGenerateContent('gemini-pro', 'What is the capital of France?');

2.2 Tracing with wrapOpenAI (for OpenAI SDK)

If you are using the OpenAI SDK, you can directly wrap the client:

import { OpenAI } from "openai";
import { wrapOpenAI } from "langsmith/wrappers";

const openAIClient = wrapOpenAI(new OpenAI());

// Use openAIClient as usual, and calls will be traced automatically.
// const response = await openAIClient.chat.completions.create({...});

3. Collecting Feedback