langgraph-testing-evaluation

LangGraph Testing & Evaluation

Practical workflows for validating agent quality with:

• Unit/integration tests
• Trajectory evaluation
• LangSmith dataset evaluations
• A/B-style comparisons between versions

Use this file for high-level flow. Load references/* for detailed implementation.

Start Here

Choose the smallest approach that answers your question:

Goal	Primary method	Load first
Validate node logic quickly	Unit tests with mocks	references/unit-testing-patterns.md
Validate multi-step agent behavior	Trajectory evaluation	references/trajectory-evaluation.md
Track quality over datasets over time	LangSmith evaluation	references/langsmith-evaluation.md
Compare old vs new agent versions	A/B comparison	references/ab-testing.md

Recommended order:

1. Unit tests
2. Integration/trajectory checks
3. Dataset evaluation in LangSmith
4. A/B comparison before deployment

Quick Commands

Run from repo root.

Generate test scaffolding

# Python (preferred)
uv run skills/langgraph-testing-evaluation/scripts/generate_test_cases.py my_agent:graph --output tests/ --framework pytest

# JavaScript/TypeScript
node skills/langgraph-testing-evaluation/scripts/generate_test_cases.js ./my-agent.ts:graph --output tests/ --framework vitest

Run trajectory evaluation

# Python: LLM-as-judge
uv run skills/langgraph-testing-evaluation/scripts/run_trajectory_eval.py my_agent:run_agent my_dataset --method llm-judge --model openai:o3-mini

# Python: trajectory match
uv run skills/langgraph-testing-evaluation/scripts/run_trajectory_eval.py my_agent:run_agent dataset.json --method match --trajectory-match-mode strict --reference-trajectory reference.json

# JavaScript/TypeScript
node skills/langgraph-testing-evaluation/scripts/run_trajectory_eval.js ./agent.ts:runAgent my_dataset --method llm-judge --model openai:o3-mini --max-concurrency 4

Run LangSmith dataset evaluation

# Python
uv run skills/langgraph-testing-evaluation/scripts/evaluate_with_langsmith.py my_agent:run_agent my_dataset --evaluators accuracy,latency --max-concurrency 4

# Python (do not upload experiment results)
uv run skills/langgraph-testing-evaluation/scripts/evaluate_with_langsmith.py my_agent:run_agent my_dataset --evaluators accuracy --no-upload

# JavaScript/TypeScript
node skills/langgraph-testing-evaluation/scripts/evaluate_with_langsmith.js ./agent.ts:runAgent my_dataset --evaluators accuracy,latency --max-concurrency 4