benchmark agents

benchflow-ai/benchflow

Agent Codex

Analyze blind comparison results to understand WHY the winner won and generate improvement suggestions.

benchflow-ai/benchflow

Agent Codex

Compare two outputs WITHOUT knowing which skill produced them.

benchflow-ai/benchflow

Agent Codex

Evaluate expectations against an execution transcript and outputs.

microsoft/STATE-Bench

Agent

StateBenchAgent is the provided standard tool-calling agent loop for Azure AI Foundry models and OpenAI models. It uses the OpenAI v1 client to call the Responses API with the benchmark's domain tools and the locked prompt protocol. If your model supports this tool-calling path, no agent code is required.

microsoft/STATE-Bench

Agent

Use this path to evaluate models from a different provider or write a custom tool-calling agent. STATE-Bench does not ship third-party adapters — provider integration is user-owned.

vitaliikapliuk/modelharness

Agent

Fresh-context verifier. Use PROACTIVELY after completing any multi-step task - verifies work against its specification by running real checks, immune to the implementer's rationalizations.

MetriLLM/metrillm

Agent

Analyzes and compares MetriLLM benchmark results. Recommends the best local model for a given use case based on performance and quality scores.

typedef-ai/ade-bench-plugin

Agent

Scans a dbt project directory and returns a structured analysis of its models, sources, dependencies, database type, and complexity. Read-only — does not modify any files.