boxel: Command for Claude Code

Run an AI assistant evaluation end to end: $ARGUMENTS

The first token is the EvaluationCard URL, for example https://localhost:4201/user/evals/Evaluation/hello-world. The second, optional, token is a comma-separated list of models as the picker names them ("Claude Sonnet 4.6,GPT-5.5").

Every model run spends real money and takes minutes. Run each model once per session. Never rerun a model whose result you have not read, and never loop. A plan that repeats runs, as the skill-change-eval skill's does, runs each repeat as its own session once the previous one is judged.

1. Ask what to run

Ask before anything else. The only work that comes first is reading the files the options are built from, named below. Nothing else happens until the answer comes back — no curl of the stack, no pnpm eval:setup, no run. The preflight is step 2 for a reason: starting it first burns time on a run that may not be the one the user wanted, and a stack that is down is worth reporting after the choice, not instead of it.

Look at $ARGUMENTS. Ask with one AskUserQuestion call carrying every question it leaves unanswered:

• no EvaluationCard URL and no evaluation name → ask which evaluation
• no model list → ask which models
• nothing about the browser → ask whether to watch it

If it answers all three, skip straight to the preflight. Otherwise every unanswered one goes in the same call, so the user answers once and the run proceeds without further interruption.

Which evaluation. Read every eval-realm/Evaluation/*.json under packages/ai-assistant-evals and take data.attributes.cardInfo.name and .summary from each — never a hardcoded list, because evaluations are added by dropping a file in that directory. List all of them in the reply, numbered, one line each as name — summary, with the summary copied word for word: it already says what each prompt asks and what the eval tests, so do not shorten or rephrase it. Then ask, with hello-world first (it is the cheapest and the one to start from) and the rest by ascending cost. Each option's label is the name, and its description is the summary, word for word. Say in the question that "Other" takes any file name from the list. An evaluation's URL is https://localhost:4201/user/evals/Evaluation/<file name without .json>.

Which models. Read every model the SystemCard offers (packages/catalog/contents/SystemCard/default.json → relationships.modelConfigurations.* → each card's cardInfo.name). That list is the whole of what can be asked for: a model the SystemCard does not list is not in the picker, and naming it fails the run at model selection.

Print all of them in the reply as a numbered list, name and lab per line, so the third option below has something to refer to.

Then ask, three options:

• Claude Sonnet 4.6 — the recommended single model. Minutes and cents.
• GPT-5.6 Luna — a single model from another lab, for a second opinion on the same evaluation.
• Specify manually — all for every model on the list, or any names from it, comma-separated. Say in the description that this is where the other thirteen live and that all costs dollars and the better part of an hour for a one-prompt evaluation, more for a two-prompt one.

An answer typed into the box is names or all, taken as given.

Take an answer the user typed into "Other" as given — it is a picker name or an evaluation file name, not a new question.

Whether to watch the browser. Three options, watching first as the default, because a run takes minutes and what the assistant does on screen is most of what there is to see:

• Watch it — pass no flag. One headed browser with a window per model, all still running side by side, so watching costs no wall-clock.
• Headless — pass --headless. Nothing on screen.
• One at a time — pass --headed. A plain window, one model per turn; say in the description that this makes a sweep as long as the sum of its models.