langsmith_evaluation

LangSmith Evaluation with @google/genai

This rule provides a comprehensive guide for setting up and running evaluations with LangSmith, specifically tailored for integrations with @google/genai (Gemini on Vertex AI).

1. Setup Environment Variables

Before using @google/genai, ensure the following environment variables are set in your shell:

• GOOGLE_API_KEY (for Google API Key authentication)
• OR
• GOOGLE_GENAI_USE_VERTEXAI=true (to enable Vertex AI)
• GOOGLE_CLOUD_PROJECT (your Google Cloud project ID)
• GOOGLE_CLOUD_LOCATION (your Google Cloud location, e.g., us-central1)

These variables are automatically picked up by New GoogleGenAI({}).

2. Install Dependencies

You will need langsmith and @google/genai. Note: While openevals examples often use openai, you can define custom evaluators or potentially adapt openevals for @google/genai if it supports custom LLM integrations.

npm install langsmith @google/genai
# If you plan to use openevals for prebuilt evaluators, install it as well.
# npm install openevals

3. Create a LangSmith API Key

Create an API key in LangSmith settings and set LANGSMITH_API_KEY environment variable.

export LANGSMITH_TRACING=true
export LANGSMITH_API_KEY="<your-langsmith-api-key>"

4. Create a Dataset

Datasets define your test inputs and expected outputs.

import { Client } from "langsmith";

const client = new Client();

async function createAndAddExamples() {
  const dataset = await client.createDataset("My GenAI Dataset", {
    description: "Dataset for evaluating @google/genai models.",
  });

  const examples = [
    {
      inputs: { question: "What is the capital of France?" },
      outputs: { answer: "The capital of France is Paris." },
      dataset_id: dataset.id,
    },
    {
      inputs: { question: "Who painted the Mona Lisa?" },
      outputs: { answer: "Leonardo da Vinci painted the Mona Lisa." },
      dataset_id: dataset.id,
    },
  ];

  await client.createExamples(examples);
  console.log("Dataset and examples created successfully.");
}

// Call the function to create your dataset
// createAndAddExamples();