Agent
Analyze AgentV evaluation results to identify weak assertions, suggest deterministic upgrades for LLM-grader graders, flag cost/quality improvements, and surface cross-run benchmark patterns. Use when reviewing eval quality, improving evaluation configs, or triaging flaky/expensive evaluations.
Agent
Perform bias-free blind comparison of evaluation outputs from multiple providers or configurations. Randomizes labeling, generates task-specific rubrics, scores N-way comparisons, then unblinds results and attributes improvements. Dispatch this agent when comparing outputs across targets or iterations.
Agent
Execute an AgentV evaluation test case by performing the task described in the input. Reads input.json from the test directory, carries out the task using available tools, and writes response.md with the result. Dispatch one executor subagent per test case, all in parallel.
Agent
Grade a candidate response for an AgentV evaluation test case. Evaluates all assertion types natively — deterministic checks via string operations, LLM grading via Claude's own reasoning, script-grader via Bash script execution. Zero CLI dependency. Dispatch this agent after a candidate completes a test case.
Agent
Generate improved versions of the artifact under test (skill, prompt, config, or directory of related files) based on failure analysis. Reads the current best artifact from the working tree, applies targeted mutations to address failing assertions, and writes changes in place. Supports single files and multi-file…