01-auto-arena

Auto Arena Skill

End-to-end automated model comparison using the OpenJudge AutoArenaPipeline:

1. Generate queries — LLM creates diverse test queries from task description
2. Collect responses — query all target endpoints concurrently
3. Generate rubrics — LLM produces evaluation criteria from task + sample queries
4. Pairwise evaluation — judge model compares every model pair (with position-bias swap)
5. Analyze & rank — compute win rates, win matrix, and rankings
6. Report & charts — Markdown report + win-rate bar chart + optional matrix heatmap

Prerequisites

# Install OpenJudge
pip install py-openjudge

# Extra dependency for auto_arena (chart generation)
pip install matplotlib

Gather from user before running

Info	Required?	Notes
Task description	Yes	What the models/agents should do (set in config YAML)
Target endpoints	Yes	At least 2 OpenAI-compatible endpoints to compare
Judge endpoint	Yes	Strong model for pairwise evaluation (e.g. gpt-4, qwen-max)
API keys	Yes	Env vars: OPENAI_API_KEY, DASHSCOPE_API_KEY, etc.
Number of queries	No	Default: 20
Seed queries	No	Example queries to guide generation style
System prompts	No	Per-endpoint system prompts
Output directory	No	Default: ./evaluation_results
Report language	No	"zh" (default) or "en"

Quick start

CLI

# Run evaluation
python -m cookbooks.auto_arena --config config.yaml --save

# Use pre-generated queries
python -m cookbooks.auto_arena --config config.yaml \
  --queries_file queries.json --save

# Start fresh, ignore checkpoint
python -m cookbooks.auto_arena --config config.yaml --fresh --save

# Re-run only pairwise evaluation with new judge model
# (keeps queries, responses, and rubrics)
python -m cookbooks.auto_arena --config config.yaml --rerun-judge --save

Python API

import asyncio
from cookbooks.auto_arena.auto_arena_pipeline import AutoArenaPipeline

async def main():
    pipeline = AutoArenaPipeline.from_config("config.yaml")
    result = await pipeline.evaluate()

    print(f"Best model: {result.best_pipeline}")
    for rank, (model, win_rate) in enumerate(result.rankings, 1):
        print(f"{rank}. {model}: {win_rate:.1%}")

asyncio.run(main())