evalyn-setup

evalyn-setup

Overview

Guide a developer through instrumenting their LLM agent with evalyn so traces are captured for evaluation.

Pre-flight

Check evalyn is installed:

python -m pip show evalyn-sdk 2>/dev/null

If not installed:

pip install evalyn-sdk

Step 1: Detect Agent Framework

Scan the user's agent code for imports to determine the framework:

Supported frameworks (all auto-instrumented - decorator is sufficient):

langchain, langgraph, anthropic, openai, google.generativeai, google.adk, claude_agent_sdk

If no recognized framework: the decorator still works for any Python function, but LLM calls won't have token/cost details.

Step 2: Add the Decorator

Add to the agent's main entry function. The import evalyn_sdk line MUST come before any framework imports (it patches LLM clients via sys.meta_path):

import evalyn_sdk  # Must be FIRST import — patches LLM clients for tracing

from evalyn_sdk import eval

@eval(project="<project-name>", version="v1")
def agent_function(query: str) -> str:
    # existing agent code
    ...

Rules:

• import evalyn_sdk must be the very first import in the file
• project: descriptive kebab-case name (e.g., "my-research-agent")
• version: tracks iterations, start with "v1"
• Wrap the outermost function that represents one agent invocation
• Do NOT wrap internal helper functions
• Optional name parameter overrides the display name (defaults to function name)

Real example from the codebase:

import evalyn_sdk  # First import

from evalyn_sdk import eval

@eval(project="gemini-deep-research-agent", version="v1", name="research_agent")
def run_agent(question: str) -> str:
    ...

Step 3: Run the Agent

Tell the user to run their agent with at least 3 different inputs to generate traces:

python path/to/agent.py "first test query"
python path/to/agent.py "second different query"
python path/to/agent.py "third varied query"