Grafana Agent Observability is an SDK and plugin collection that collects telemetry from coding agents and AI agents built into applications, including sessions, tool calls, traces, tokens, costs, generations, and evaluations. Developers use it to monitor coding-agent usage or instrument agents in their own services, with SDKs for several programming languages. The catalogue add-ons help configure and use this observability workflow with coding agents.
Getting it into your agent
It runs from inside its repository, so the clone comes first — what it calls does not travel with the file alone.
git clone --depth 1 https://github.com/grafana/agento11ynpx agentmods add skills/grafana/agento11y/agento11y-eval-starterWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/grafana/agento11y/agento11y-eval-starter)<a href="https://agentmods.dev/skills/grafana/agento11y/agento11y-eval-starter"><img src="https://agentmods.dev/badge/skills/grafana/agento11y/agento11y-eval-starter/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/grafana/agento11y/agento11y-eval-starter"><img src="https://agentmods.dev/badge/skills/grafana/agento11y/agento11y-eval-starter.svg" alt="Reviewed on agentmods" width="80" height="20"></a>- NVIDIA SkillSpector warn
SkillSpector: 2 findings, up to medium
These are SkillSpector’s own severities. On a checked sample its high-severity flags on skills were ~96% false positives — a documented command, a public API, a “never do X” rule — so we show them as a caution to read, not a verdict. Why →
- medium Excessive Agency · line 38 Skill enables autonomous high-impact decisions without human-in-the-loop verification. Critical operations (destructive commands, financial transactions, data deletion) should require explicit user confirmation.Fix: Add human-in-the-loop confirmation for destructive, irreversible, or high-impact operations. Never auto-execute commands that modify files, send data, or alter system state.
- medium MCP Rug Pull · line 331 npx commands without a version suffix (e.g. @1.0.0) create a rug-pull risk if the upstream server is compromised and publishes a malicious update.Fix: Pin the version: npx @scope/[email protected]
What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00000 | $0.06241 |
| Opus 5 | $0.00000 | $0.03121 |
| Sonnet 5 | $0.00000 | $0.01248 |
| Haiku 4.5 | $0.00000 | $0.00624 |
Grade A, and why
agento11y-eval-starter scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 11d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 442 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Agent Observability eval starter
Help a developer who has an AI agent but no evaluation set up yet. The hard part isn't running an experiment — it's knowing what to evaluate and having cases to test against before there's any traffic. Answer both, grounded in the agent's actual code.
Always produce:
- A ranked, justified evaluator recommendation for this agent.
- A starter suite YAML the developer reviews and extends.
Then, depending on how runnable the agent is (Step 1):
- For an easily-invoked agent, a runner stub (
run_experiment.py, orrun-experiment.tsfor a TypeScript agent) that wires the suite to the SDK with one hole to fill — and optionally run it (Step 6), only with permission. For an agent that needs a harness or full runtime, point to the existing eval infra instead of a runner that can't actually call it.
This skill is language-agnostic — the reading, recommending, and YAML it produces do not
depend on the agent's language. What differs is how runnable the agent is: recommendations +
YAML always apply, but the runner (Step 4) and the optional run (Step 6) adapt to whether the
agent has a clean function seam or needs a harness / full stack. For deeper run-side patterns
(binding existing generations, cross-process verifiers) point to the per-language run skill
(Python: agento11y-experiments).
Rules
- Do not create, enable, or modify evaluators, rules, or guards in any Agent Observability tenant. No control-plane writes. (Running an offline experiment only publishes that run's scores — it does not create tenant-level evaluators/rules/guards — but only do it via Step 6.)
- Do not rewrite the agent's prompt, optimize, or redeploy.
- Never run the experiment without asking first (Step 6). Never run against a target the
developer did not configure — use their
AGENTO11Y_ENDPOINT+ credentials (Grafana Cloud); if they are not set, ask for them, do not invent an endpoint. - Never mint, generate, or store credentials. The developer owns their Grafana Cloud ingestion token; read it from the environment or ask them to paste it — do not create one.
- Never present the generated cases as validated. They are a draft to review and extend.
- Agent Observability is a Grafana Cloud product. Do not hardcode or assume any endpoint (no
localhost); the developer supplies their Cloud endpoint and token. - If a required input is missing (entrypoint, prompt, tools), ask the developer — don't guess.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 11d ago First seen · 442 lines · 0 tokens per session scan A 14b242fe046a
agento11y-eval-starter is a skill published in the GitHub repository grafana/agento11y (103 stars, last pushed yesterday), licensed Apache-2.0. It costs nothing until one of its globs matches a file; then it loads 6,241 tokens. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.
Other skills, from other repositories
research-engineer
An uncompromising Academic Research Engineer. Operates with absolute scientific rigor, objective criticism, and zero flair. Focuses on theoretical correctness, formal verification, and optimal implementation across any required technology.
tika-eval-compare
Compare extracts from two Tika builds over a corpus to detect regressions in content, encoding, exceptions, and embedded-document handling. Use for "compare before/after extracts", "eval this change against the corpus".
neuron-evaluation-engineer
Create and run AI evaluations with datasets, assertions, and output drivers in Neuron AI. Use this skill whenever the user mentions evaluation, testing AI systems, creating evaluators, dataset-driven testing, assertion-based validation, or wants to measure AI system performance. Also trigger for tasks involving…
jetson-validate-image
Use after jetson-flash-image to run static BSP checks, on-target smoke/regression tests on a flashed DUT, or both. Not for build or flash steps. Triggers: validate bsp, on-target validation.
atmos-validation
Validate Atmos projects, components, arbitrary JSON Schema inputs, EditorConfig, and GitHub Actions; use affected-file selection and native CI annotations.
skill-benchmark
Benchmark AI skill effectiveness by measuring implementation quality against legacy constraints.