opik-annotate

Annotate — A Review Queue Annotators Can Read

Definition of done: the queue exists with the view saved, and the user knows where to check it — the write returned without a warning, and the answer carries the queue's link. A view is built from the project's real traces or threads, never from a guessed shape. If the queue or the view can't be saved, stop at the first genuine blocker and return exactly one next step.

Operate: look at real items first, cover every shape you saw, show text not JSON, keep scoring in Opik's own panel. This skill writes to Opik only and changes no application code.

Inputs

/opik-annotate <what annotators judge> ("the drafted reply", "whether the agent solved the ticket"), /opik-annotate <queue name or id> (change that queue's view), or a request to remove a view. Infer the rest; optional overrides: project (default: configured) · scope (default: trace; thread for whole conversations) · feedback definitions (default: the queue's own).

Activation — the only in-scope work

1. Resolve the project and the queue

With the Opik MCP connected: list('annotation_queue', project_name=…) shows the project's queues with their scope and item count; read('annotation_queue', id_or_name) returns one with its instructions, feedback definitions and view summary. The view's html is stated by size; read('annotation_queue', id, fields=['annotator_view.html']) returns it whole — read it before changing a view, and change only what was asked.

2. Look at real items before writing anything

• Queue with items: list('trace', project_name=…, filters='annotation_queue_ids contains "<queue_id>"', size=20) (list('thread', …) for a thread queue), then read('trace', id) on two or three that differ.
• Empty or new queue: the project's latest traces or threads instead, and say so ("the queue has no items yet, so the view is built from the latest traces").

Note every shape input, output and metadata take: objects, arrays, or strings that hold JSON. A view that handles only the most common shape fails annotators on the rest.