Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add comet-ml/opik-skills --skill opik-online-evalgit clone --depth 1 https://github.com/comet-ml/opik-skillsWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/comet-ml/opik-skills/opik-online-eval)<a href="https://agentmods.dev/skills/comet-ml/opik-skills/opik-online-eval"><img src="https://agentmods.dev/badge/skills/comet-ml/opik-skills/opik-online-eval/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/comet-ml/opik-skills/opik-online-eval"><img src="https://agentmods.dev/badge/skills/comet-ml/opik-skills/opik-online-eval.svg" alt="Reviewed on agentmods" width="80" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00146 | $0.02934 |
| Opus 5.5 | $0.00058 | $0.01174 |
| Sonnet 5.5 | $0.00029 | $0.00587 |
| Haiku 4.5 | $0.00015 | $0.00293 |
Grade A, and why
opik-online-eval scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 22d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
This is a copy
100% identical to opik-online-eval — 52 lines differ, which has more behind it and is treated as the original. This page carries a canonical link to it rather than competing with it.
How it starts
The opening of the file, as written. The whole thing — 137 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Online Eval — Take a Judge Live
Definition of done: an online evaluation rule exists on the project, is enabled, and has scored at least one new trace — confirmed by reading a fresh trace's feedback score, not by the create call returning. The rule scores one failure mode, samples at a rate the project's volume can afford, carries a cost cap, and maps its variables to the fields the traces actually have. If traffic hasn't arrived yet, the done state is "rule live, unverified — watch this filter". If the rule can't be created, stop at the first genuine blocker and return exactly one next step.
Operate: one failure mode per rule, sample before you scale, cap the spend, verify on a real trace — and change no application code. This skill writes to Opik only.
Inputs
The entry point is /opik-online-eval <what to score> ("hallucinations", "the refund-window answer", "refusals"), /opik-online-eval right after /opik-evaluate validated a judge (take that judge live), or /opik-online-eval <project>. Infer the rest; treat these as optional overrides:
- project (default: configured) · scope (default: trace;
spanfor one model call,threadfor whole conversations) · sampling rate (default: 1.0 under ~1k traces/day, else 0.1–0.2) · filters (default: none; typical: anenvironmentor tag filter) · judge model · cost cap (default: set one) · variable mapping (default:input → input,output → output).
Ask only at a genuine, non-inferable blocker (see Blockers).
Activation — the only in-scope work
1. Resolve project, judge, and scope
Confirm Opik is reachable (~/.opik.config or OPIK_API_KEY; otherwise → Blocker). Resolve the project id from a trace or by name. Decide the judge:
- A judge
/opik-evaluatealready validated → reuse its prompt and output schema verbatim. - A named metric (hallucination, answer relevance, moderation) → a minimal binary judge for that one failure mode (
../opik-evaluate/references/write-judge-prompt.md). - A mechanical check (JSON valid, contains a disclaimer, latency budget) → a Python metric rule, not a judge.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 22d ago First seen · 137 lines · 146 tokens per session scan A 0611da9b553d
opik-online-eval is a skill published in the GitHub repository comet-ml/opik-skills (8 stars, last pushed yesterday), licensed Apache-2.0. It adds 146 tokens to every session and 2,934 once invoked, about $0.0006 per session on Opus 5.5. A static security scan graded it A with 0 findings. It is 100% identical to opik-online-eval, differing in 52 lines, and is treated as a copy.
Other skills, from other repositories
opik-optimize
Improve a prompt with the Opik Agent Optimizer — resolve the prompt, a dataset, and a metric, pick the algorithm, run a bounded optimization, check the gain on held-out data, and save the winner as a new prompt version with the optimization run link. Runs via the opik-optimizer package; reads prompts and datasets via…
opik-online-eval
Take a judge live on production traffic — create an Opik online evaluation rule (LLM-as-judge or Python metric) on a project with sampling, filters, variable mapping, and a cost cap, then confirm new traces are being scored. Works over the SDK's REST client; reads rules and score names via the MCP when connected.…
opik-test
Turn a failing Opik trace (or a described failure) into a repeatable regression check — a test-suite item with the trace's input and one or two binary assertions — so a fix can be verified by the compare skill. Works over the SDK; uses the MCP write tool when connected. Returns the suite, the item, and the assertion.…
opik-instrument
Add Opik tracing to an existing app and verify a real trace lands. Installs the Opik package, detects the language and LLM framework, adds the minimum tracing, runs a safe representative path, confirms a trace in Opik, and returns the trace link. Use for "instrument my code", "add opik tracing", "add observability"…
opik-evaluate
Build an LLM evaluation and run it against the app, returning an Opik experiment with scores and its link. Picks a test suite with judge assertions or a dataset with metrics, sources cases from traces or synthetic data, scores heuristics-first then one-failure-mode judges, runs client-side via the SDK or server-side…
llm-app-patterns
Production-ready patterns for building LLM applications. Covers RAG pipelines, agent architectures, prompt IDEs, and LLMOps monitoring. Use when designing AI applications, implementing RAG, building agents, or setting up LLM observability.