agento11y-instrument

Agent Observability — instrument an LLM app

Help a developer wire their own LLM app or agent to Grafana Agent Observability, from zero or from a broken/partial state, and keep going until data actually lands in the stack. The value this skill adds over the static instrumentation prompt is two things a prompt can't do:

1. A mechanical gap checklist run against the real code — headed by the one failure that is completely silent (missing OTel providers → every metric lost, no error).
2. A verification loop through gcx: after each change, run the app and check the gcx agento11y agents / conversations / generations commands to confirm generations arrived. Diagnose the next gap from what's missing, not from guesswork.

The SDK reference (env vars, provider snippets, field lists, framework adapters, workflow steps) lives in agento11y's llms.txt "Path B". Fetch it rather than restating it here; this file holds the flow and the decision logic. A minimal fallback lives in references/instrumentation.md for when the fetch is unavailable.

Rules

• Reference, don't restate. Fetch SDK detail from https://raw.githubusercontent.com/grafana/agento11y/main/llms.txt (Path B). Only inline decision logic here. If the fetch fails, fall back to references/instrumentation.md.
• Never invent an endpoint or a token. Read them from the environment (AGENTO11Y_ENDPOINT, AGENTO11Y_PROTOCOL, AGENTO11Y_AUTH_MODE, AGENTO11Y_AUTH_TENANT_ID, AGENTO11Y_AUTH_TOKEN, OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_HEADERS) or ask the developer. Never fabricate a URL or mint a token.
• Target is Grafana Cloud. The developer supplies the endpoint + token (Step 0), and the gcx verification loop (Step 5) confirms data landing against the Cloud tenant. Never fabricate the endpoint or token — read them from the environment or ask.
• Write AGENTO11Y_* env vars, never SIGIL_*. SIGIL_* is a deprecated legacy fallback. Do this even if sibling apps or existing .env files in the repo use SIGIL_* — matching a stale local convention perpetuates it. If the app already reads SIGIL_*, add the AGENTO11Y_* names (the SDK still honors both) and note the old ones are deprecated. Do not "match the siblings."
• Gate every code WRITE on explicit confirmation. Report first (Step 4), apply only after the developer says yes (Step 5). Read-only gcx verification and re-running the app happen freely inside the loop; editing files does not.
• Keep diffs small; do not change app behavior. Instrumentation is additive. No refactors, no prompt rewrites, no dependency upgrades beyond the SDK/adapter packages actually needed.
• Never change the model, provider, or the app's LLM config — not even with permission, not even "just to run the verify loop." Instrument whatever model the app already uses. This is absolute: changing the model is out of scope for instrumentation, full stop. If a run fails because a provider API key is missing, the only allowed responses are: (a) ask the developer to provide the key for the model the app already uses, or (b) skip the live run and report the wiring as verified-by-construction, telling the developer to run it themselves. Do not offer to switch the provider, do not ask "which provider should I use?", and do not add a new provider dependency (e.g. langchain-anthropic) to make the run succeed. If the developer separately says they want a different model, that is an app change they own — tell them to make it and re-invoke this skill; do not fold it into the instrumentation diff. Swapping the model silently changes what the app does and what gets observed, which defeats the point. The provider API key (ANTHROPIC_API_KEY, OPENAI_API_KEY, …) is the app's own concern, not the instrumentation's — it authenticates the LLM call, not the telemetry export, and the app already has it if it runs at all. So don't ask for it, configure it, or rewire it; if the live verify-run fails on a missing provider key, skip the run and report verified-by-construction (see Step 5). Just don't conflate the two 401s: a 401 on generation ingest is observability auth and is yours to fix (usually a missing AGENTO11Y_PROTOCOL/AGENTO11Y_AUTH_MODE); an auth error from the model provider is not — surface it and let the developer handle their own key.
• Do not assume language symmetry. Verify the provider wrapper / framework adapter actually exists for the app's language before recommending it (Python has the most adapters, JS fewer, Go only google-adk, Java/.NET core + providers + google-adk). If it doesn't exist, hand-instrument with the core SDK. Prefer, in order: provider wrapper → framework adapter → hand-instrumentation.
• The loop is bounded. At most ~3–4 instrument→verify iterations. If data still isn't landing, stop and report what's checked and what remains — don't loop forever.
• Field-name traps: cache_write_input_tokens, NOT cache_creation_input_tokens. agent_version maps to the gen_ai.agent.version label and is required for per-version Performance charts. MessageRole (Python SDK) has only USER, ASSISTANT, TOOL — there is no SYSTEM (or DEVELOPER) member; MessageRole.SYSTEM raises AttributeError. Fold the system prompt into the USER message (or a text_part), and prefer the user_text_message() / assistant_text_message() / tool_result_message() helpers over hand-building Message(role=...). Always confirm enum members and helper names against the installed package before running — do not assume from llms.txt.
• Out of scope: offline test suites → agento11y-test-starter; tenant eval rules + guards on real traffic → agento11y-prod-setup. Coding-agent telemetry plugins (Claude Code, Cursor, …) → llms.txt "Path A". Any control-plane write.
• If a required input is missing (entrypoint, framework, endpoint, gcx auth), ask — don't guess.