boxel: Skill for Claude Code

AI assistant evals

packages/ai-assistant-evals holds a Playwright spec that does, per model, what a person did by hand all week: log in, create a workspace, open a room, pick the model, send "create a hello world card and show it", wait until the bot is idle, check a card rendered, and read the room's events for the numbers. One row per model.

It runs against the REAL local stack and the REAL provider. Every run spends money (a few cents for a good model, up to a few dollars for a bad one) and leaves a workspace and a room behind on the local synapse and realm server. Do not run it to "see what happens", do not loop it, and do not rerun a model whose result you have not read yet.

Bare prompt or evaluation

pnpm eval:models is the fixed hello-world prompt, graded mechanically, nothing written anywhere but eval-results/. pnpm eval <evaluation-card-url> is the same browser flow fed by an EvaluationCard: the card's prompt and follow-up prompts, its initial cards and files copied into the test workspace first, a per-session results directory, and one EvaluationResultCard per model plus one EvaluationReportCard written into the workspace the evaluation lives in (the writer's AI Assistant Evaluations workspace, at the endpoint evals, made by pnpm eval:setup). The quality score on a result is a judge's, not the runner's: the /run-ai-assistant-eval command in .claude/commands walks through running, judging against the card's success criteria, and recording the score with pnpm eval:judge. Everything below about watching a run, grades, verdicts and benchmarks applies to both. For an evaluation with follow-up prompts the turn and time targets are multiplied by the number of prompts.

A change to the boxel-skills content that targets one of the assistant's core use cases needs an evaluation that shows the problem and the with/without numbers: the skill-change-eval skill covers writing it and what counts as evidence.

Prerequisites

• The dev stack is up: mise run dev-all with OPENROUTER_API_KEY set for the ai-bot. Check: curl -sk -o /dev/null -w '%{http_code}' https://localhost:4200/ prints 200.
• Every eval user holds at least 3000 credits. Running out mid-session turns the assistant's reply into "There was an error processing your request" and wastes the run, so top up first — the balance is the sum of that user's credits_ledger rows and a top-up is one extra_credit row. The /run-ai-assistant-eval command's preflight carries the query.
• The local matrix users ai-assistant-eval-user-1 to ai-assistant-eval-user-5 / password exist (see EVAL_USERS below). pnpm eval:users registers whoever is missing, and pnpm eval refuses to start without them rather than failing on the first browser action of a paid run. For evaluations, pnpm eval:setup has created the AI Assistant Evaluations workspace of the writer user (user locally), at the endpoint evals, and pushed the cards.
• Every model you name has a ModelConfiguration card in the SystemCard the host uses (packages/catalog/contents/SystemCard/default.json), or the picker will not offer it.
• Playwright's Chromium is installed (pnpm exec playwright install chromium in packages/ai-assistant-evals if a run complains).