Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add comet-ml/opik-mcp --skill opik-online-evalgit clone --depth 1 https://github.com/comet-ml/opik-mcpWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/comet-ml/opik-mcp/opik-online-eval)<a href="https://agentmods.dev/skills/comet-ml/opik-mcp/opik-online-eval"><img src="https://agentmods.dev/badge/skills/comet-ml/opik-mcp/opik-online-eval/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/comet-ml/opik-mcp/opik-online-eval"><img src="https://agentmods.dev/badge/skills/comet-ml/opik-mcp/opik-online-eval.svg" alt="Reviewed on agentmods" width="80" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00146 | $0.02959 |
| Opus 5.5 | $0.00058 | $0.01184 |
| Sonnet 5.5 | $0.00029 | $0.00592 |
| Haiku 4.5 | $0.00015 | $0.00296 |
Grade A, and why
opik-online-eval scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 19d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
Copies of this mod
2 near-identical copies found in the catalogue:
- opik-online-eval — 100% identical, 52 lines differ
- opik-online-eval — 100% identical, 52 lines differ
How it starts
The opening of the file, as written. The whole thing — 157 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Online Eval — Take a Judge Live
Definition of done: an online evaluation rule exists on the project, is enabled, and has scored at least one new trace — confirmed by reading a fresh trace's feedback score, not by the create call returning. The rule scores one failure mode, samples at a rate the project's volume can afford, carries a cost cap, and maps its variables to the fields the traces actually have. If traffic hasn't arrived yet, the done state is "rule live, unverified — watch this filter". If the rule can't be created, stop at the first genuine blocker and return exactly one next step.
Operate: one failure mode per rule, sample before you scale, cap the spend, verify on a real trace — and change no application code. This skill writes to Opik only.
Inputs
The entry point is /opik-online-eval <what to score> ("hallucinations", "the refund-window answer", "refusals"), /opik-online-eval right after /opik-evaluate validated a judge (take that judge live), or /opik-online-eval <project>. Infer the rest; treat these as optional overrides:
- project (default: configured) · scope (default: trace;
spanfor one model call,threadfor whole conversations) · sampling rate (default: 1.0 under ~1k traces/day, else 0.1–0.2) · filters (default: none; typical: anenvironmentor tag filter) · judge model · cost cap (default: set one) · variable mapping (default:input → input,output → output).
Ask only at a genuine, non-inferable blocker (see Blockers).
Activation — the only in-scope work
1. Resolve project, judge, and scope
Confirm Opik is reachable (~/.opik.config or OPIK_API_KEY; otherwise → Blocker). Resolve the project id from a trace or by name. Decide the judge:
- A judge
/opik-evaluatealready validated → reuse its prompt and output schema verbatim. - A named metric (hallucination, answer relevance, moderation) → a minimal binary judge for that one failure mode (
../opik-evaluate/references/write-judge-prompt.md). - A mechanical check (JSON valid, contains a disclaimer, latency budget) → a Python metric rule, not a judge.
What ships with it
9 files beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.
- evals/.gitignore 7 B
- evals/cases.yaml 2.8 KB
- evals/fixtures/traffic/pyproject.toml 118 B
- evals/fixtures/traffic/seed.py 3.9 KB runs code
- evals/fixtures/traffic/traffic.py 914 B runs code
- evals/grader.py 4.5 KB runs code
- evals/HARNESS.md 2.7 KB
- evals/metrics.py 2.4 KB runs code
- evals/run_evals.py 9.2 KB runs code
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 19d ago Changed · +20 lines 38e0293e0faa
- 26d ago First seen · 137 lines · 146 tokens per session scan A 0611da9b553d
opik-online-eval is a skill published in the GitHub repository comet-ml/opik-mcp (222 stars, last pushed 6d ago), licensed Apache-2.0. It adds 146 tokens to every session and 2,959 once invoked, about $0.0006 per session on Opus 5.5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-09-18.
Other skills, from other repositories
local-rag-search
Efficiently perform web searches using the mcp-local-rag server with semantic similarity ranking. Use this skill when you need to search the web for current information, research topics across multiple sources, or gather context from the internet without using external APIs. This skill teaches effective use of…
cqlib-kernel
A controlled workflow for running a cqlib quantum-style angle kernel and quantum support vector machine (QSVM), a classifier that separates labelled data. It compares the result with independent mathematical and RBF baselines and states the method's scientific limits.
qiskit-machine-learning
A Qiskit Machine Learning coding skill for quantum kernels, quantum support-vector models, quantum neural networks, gradients, training, and prediction. It uses quantum circuits as part of machine-learning models.
squlearn-learning
A Python toolkit that connects quantum circuits and measurements with scikit-learn, a common machine-learning library. It supports quantum kernels, which compare data using quantum circuits, and scikit-learn estimators.
luma-vision
A skill for analyzing images with several external vision models. It is activated only when a user starts a request with /skill luma-vision and includes an image.
research-topic
Multi-step research orchestration. Use when user asks "research X", "summarize current state of Y", "what's the latest on Z", or compares approaches. Calls extract(action="agent") which searches the web, extracts top results, then synthesises a citation-preserving Markdown answer with one configured LLM.