judge-eval

Judging a denkeeper eval run

Rubric version: v1 (2026-08-17)

You are the judge in a blinded A/B comparison. A denkeeper eval run executed every test case in a task set against two agent configurations — an incumbent and a candidate — through the real engine, with writes suppressed. Your job is to say which of two responses is better, without knowing which configuration produced either.

You never learn the assignment. Model, provider, variant name, cost, latency and token usage are withheld server-side. Do not speculate about them, and do not let a guess influence a verdict. If you find yourself reasoning about "this looks like a bigger model", stop and go back to the response.

Setup

The tools come from denkeeper's MCP server. The judging key needs exactly two scopes and nothing else:

denkeeper keys create judge --scopes eval:read,eval:write

A judge key with wider scopes can start runs and spend money; this one can read pairs and write verdicts, and that is all it should ever be able to do.

Add the server to Claude Code (adjust host and key):

claude mcp add --transport http denkeeper https://your-denkeeper/api/v1/mcp --header "Authorization: Bearer $DENKEEPER_JUDGE_KEY"

Tools

Tool	What it does
eval_run_status	Run status, progress, pairs produced, items still pending
eval_pending	The queue. run_id scopes it; sample_n draws a random subset
eval_get_pair	One blinded item: prompt, notes, pinned history, Response A and B with tool traces
eval_verdict	Record {item_id, winner, dimensions, notes, rubric_version}
eval_summary	Unblinds: gate table, win-rate, per-category breakdown, verdict

Each pair of responses is queued twice, with the presentation order swapped. Both items must be judged before the pair counts, and if your two verdicts name different sides the pair records as a tie — that is the position-bias control, and it only works if you judge each item on its own. Never try to recognise a pair you have already seen, and never look up your earlier verdict.