agento11y-experiments

Agent Observability experiments

Use this skill when adding framework-free offline evaluation to a Python project. The public SDK surface is agento11y.experiments; do not use removed v0 runner APIs.

This is the reference for the run-side API. If you don't yet know which evaluators you need or have no test cases, start with the agento11y-eval-starter skill — it reads your agent, recommends evaluators, writes a starter suite, and generates a minimal runner; come here for the deeper patterns (binding existing generations, auditable LLM judges, cross-process verifiers, pass@k/pass^k).

The normal setup cost for an already instrumented agent should be small:

1. Import experiments from agento11y.
2. Define a TestSuite with TestCases.
3. Wrap the existing agent call in with exp.trial(case) as trial:.
4. Bind the generation/conversation ids your normal instrumentation already produced, or call trial.record_io(...) when the harness owns the call.
5. Emit one final score and any supporting scores.

Setup

pip install "agento11y>=0.11.0"

Required environment:

export AGENTO11Y_ENDPOINT=https://agento11y-prod-<region>.grafana.net
export AGENTO11Y_AUTH_TOKEN=<grafana-cloud-ingestion-api-key>

# Optional when the endpoint requires tenant-scoped basic auth.
export AGENTO11Y_AUTH_TENANT_ID=<stack-id>

# Optional UI host for deep links when it differs from AGENTO11Y_ENDPOINT.
export AGENTO11Y_GRAFANA_URL=https://<your-stack>.grafana.net

Local-suite experiment ingest uses only the Cloud ingestion API key. Stored suite push/pull additionally uses AGENTO11Y_CONTROL_ENDPOINT and a Grafana service-account token in AGENTO11Y_SERVICE_ACCOUNT_TOKEN.

Experimental OTel eval spans/events are disabled by default. Opt in only when asked:

with experiments.experiment("nightly", use_experimental_otel=True) as exp:
    ...

Recommended Pattern

from agento11y import experiments

suite = experiments.TestSuite(
    suite_id="smoke",
    name="Smoke",
    version="2026-06-29",
    test_cases=[
        experiments.TestCase(test_case_id="capital-fr", input="Capital of France?", expected="Paris"),
    ],
)
verifier = experiments.Evaluator(evaluator_id="exact_match", version="2026-06-29", kind="deterministic")

with experiments.experiment(
    "PR experiment",
    experiment_id=f"pr-{git_sha}",
    suite=suite,
    planned_trial_count=len(suite.test_cases),
    candidate={"git_sha": git_sha, "model_name": "gpt-4o-mini"},
    tags=["ci"],
) as exp:
    for case in suite.test_cases:
        with exp.trial(case) as trial:
            answer = call_your_agent(case.input)

            # If normal instrumentation already created a conversation/generation,
            # bind those ids instead of recording duplicate I/O.
            # trial.bind_conversation(conversation_id)
            # trial.bind_generation(generation_id, conversation_id=conversation_id)
            trial.record_io(
                input=case.input,
                output=answer,
                model_provider="openai",
                model_name="gpt-4o-mini",
            )

            passed = str(case.expected).lower() in answer.lower()
            trial.final_score(
                1.0 if passed else 0.0,
                passed=passed,
                explanation=f"expected {case.expected!r}, got {answer!r}",
                evaluator=verifier,
            )

print(exp.url)