pydantic-evals

Evaluating Non-Deterministic Functions with pydantic-evals

pydantic-evals is a code-first framework for evaluating stochastic functions (LLM calls, agents, pipelines). Define test cases, run them against a task function, and score results with evaluators.

Install: pip install pydantic-evals (or pip install 'pydantic-evals[logfire]' for Logfire integration).

Import Reference

from pydantic_evals import Case, Dataset, set_eval_attribute, increment_eval_metric
from pydantic_evals.evaluators import (
    Evaluator, EvaluatorContext, EvaluatorOutput, EvaluationReason,
    ReportEvaluator, ReportEvaluatorContext,
    LLMJudge, HasMatchingSpan,
)
from pydantic_evals.evaluators.common import Equals, EqualsExpected, Contains, IsInstance, MaxDuration
from pydantic_evals.otel import SpanQuery               # requires logfire extra
from pydantic_evals.generation import generate_dataset  # LLM-based dataset generation

Data Model

Dataset -> Cases -> Evaluators -> EvaluationReport. A Dataset holds Case objects and dataset-wide evaluators. Calling dataset.evaluate(task_fn) runs the task against all cases and returns an EvaluationReport. Both Case and Dataset are generic: Case[InputsT, OutputT, MetadataT].

Case

case = Case(
    name="simple",                           # identifier (optional, but recommended)
    inputs="What is the capital of France?", # any type — passed to the task function
    expected_output="Paris",                 # optional — available via ctx.expected_output
    metadata={"difficulty": "easy"},         # optional — available via ctx.metadata
    evaluators=(MyEvaluator(),),             # optional — case-specific evaluators
)

Dataset

dataset = Dataset(
    cases=[case1, case2],
    evaluators=[GlobalEvaluator()],          # applied to every case
    report_evaluators=[MyReportEvaluator()], # experiment-wide analysis (optional)
)

Method	Description
await dataset.evaluate(task_fn)	Run task against all cases (async)
dataset.evaluate_sync(task_fn)	Synchronous wrapper
dataset.add_case(...)	Add a case after construction
dataset.add_evaluator(ev, specific_case=None)	Add evaluator to all cases or a named case
Dataset.from_file("cases.yaml")	Load from YAML or JSON
dataset.to_file("cases.yaml")	Save to YAML or JSON