10 tagged evaluation framework, measured the same way as everything else here.
Browse within: agent-evaluation 5coding-agents 5llms 5vscode 5
Agent
Configure and run the default Claude Code agent in Coder Eval — the full agent-config surface, direct vs. Bedrock authentication, permission modes, sandbox isolation, skills/plugins, early stop, and token telemetry.
Agent
Run OpenAI Codex as the agent under evaluation in Coder Eval — installation, authentication, task configuration, and how Codex telemetry maps to sandboxed, weighted scoring.
Agent
Run OpenCode, the open-source terminal coding agent, as the agent under evaluation in Coder Eval — installation, provider authentication, model selection, and how its event stream maps to sandboxed, weighted scoring.
Agent
Analyze AgentV evaluation results to identify weak assertions, suggest deterministic upgrades for LLM-grader graders, flag cost/quality improvements, and surface cross-run benchmark patterns. Use when reviewing eval quality, improving evaluation configs, or triaging flaky/expensive evaluations.
Agent
Perform bias-free blind comparison of evaluation outputs from multiple providers or configurations. Randomizes labeling, generates task-specific rubrics, scores N-way comparisons, then unblinds results and attributes improvements. Dispatch this agent when comparing outputs across targets or iterations.
Agent
Grade a candidate response for an AgentV evaluation test case. Evaluates all assertion types natively — deterministic checks via string operations, LLM grading via Claude's own reasoning, script-grader via Bash script execution. Zero CLI dependency. Dispatch this agent after a candidate completes a test case.