agento11y-experiments

Agent Observability experiments

Use this skill when adding framework-free offline evaluation to a Python project. The public SDK surface is agento11y.experiments; do not use removed v0 runner APIs.

This is the reference for the run-side API. If you don't yet know which evaluators you need or have no test cases, start with the agento11y-eval-starter skill — it reads your agent, recommends evaluators, writes a starter suite, and generates a minimal runner; come here for the deeper patterns (binding existing generations, auditable LLM judges, cross-process verifiers, pass@k/pass^k).

The normal setup cost for an already instrumented agent should be small:

1. Import experiments from agento11y.
2. Define a TestSuite with TestCases.
3. Wrap the existing agent call in with exp.trial(case) as trial:.
4. Bind the generation/conversation ids your normal instrumentation already produced, or call trial.record_io(...) when the harness owns the call.
5. Emit one primary-verdict score and any supporting diagnostic scores; an unannotated final score remains the legacy fallback.

Setup

pip install "agento11y>=0.11.0"

Required environment:

export AGENTO11Y_ENDPOINT=https://agento11y-prod-<region>.grafana.net
export AGENTO11Y_AUTH_TOKEN=<grafana-cloud-ingestion-api-key>

# Optional when the endpoint requires tenant-scoped basic auth.
export AGENTO11Y_AUTH_TENANT_ID=<stack-id>

# Optional UI host for deep links when it differs from AGENTO11Y_ENDPOINT.
export AGENTO11Y_GRAFANA_URL=https://<your-stack>.grafana.net

Local-suite experiment ingest uses only the Cloud ingestion API key. Stored suite push/pull additionally uses AGENTO11Y_CONTROL_ENDPOINT and a Grafana service-account token in AGENTO11Y_SERVICE_ACCOUNT_TOKEN.

Experimental OTel eval spans/events are disabled by default. Opt in only when asked:

with experiments.experiment("nightly", use_experimental_otel=True) as exp:
    ...

Recommended Pattern