Light-weight AI agent evaluation and optimization framework
This repository also configures its own agents. See what agentv tells them →
Agent Claude Code
Analyze AgentV evaluation results to identify weak assertions, suggest deterministic upgrades for LLM-grader graders, flag cost/quality improvements, and surface cross-run benchmark patterns. Use when reviewing eval quality, improving evaluation configs, or triaging flaky/expensive evaluations.
Agent Claude Code
Perform bias-free blind comparison of evaluation outputs from multiple providers or configurations. Randomizes labeling, generates task-specific rubrics, scores N-way comparisons, then unblinds results and attributes improvements. Dispatch this agent when comparing outputs across targets or iterations.
Agent Claude Code
Execute an AgentV evaluation test case by performing the task described in the input. Reads input.json from the test directory, carries out the task using available tools, and writes response.md with the result. Dispatch one executor subagent per test case, all in parallel.
Agent Claude Code
Grade a candidate response for an AgentV evaluation test case. Evaluates all assertion types natively — deterministic checks via string operations, LLM grading via Claude's own reasoning, script-grader via Bash script execution. Zero CLI dependency. Dispatch this agent after a candidate completes a test case.
Agent Claude Code
Generate improved versions of the artifact under test (skill, prompt, config, or directory of related files) based on failure analysis. Reads the current best artifact from the working tree, applies targeted mutations to address failing assertions, and writes changes in place. Supports single files and multi-file…
At most 3 mods per repository are shown here, and a mod shipped inside a plugin is left to that plugin's page — the rest are on their repository pages: