evalbench AGENTS.md

EvalBench: Agentic Onboarding & Development Guide

Welcome to EvalBench! This guide is specifically designed for AI coding assistants and new developers to help you understand the repository's architecture, evaluation modes, codebase structure, and contribution flows.

---

Table of Contents

• Overview & Objectives
• Project Architecture
• Module Directory Layout
• Tool Paradigms & Generators
• Core Workflows & Lifecycle
• Configuration Schemas
• Local Development & Execution
• Troubleshooting & Tips

---

Overview & Objectives

EvalBench is a highly flexible testing and evaluation framework designed to measure the quality of Generative AI (GenAI) workflows, specifically focusing on:

• Database specific tasks: Generating valid DDL, DML, and DQL across multiple database types (AlloyDB, BigQuery, Spanner, Postgres, SQLite, etc.).
• Agent Multi-turn journeys: Simulating complex user interactions where an LLM-based simulated user responds to the agent's questions or drives the conversation via a predetermined plan.
• Extensible Scoring: Evaluating outputs with a plug-and-play suite of deterministic and LLM-based scorers.

[!NOTE] The core design principle of EvalBench is isolation. Each multi-turn execution runs in a sandboxed home directory (.venv/fake_home/ or .venv/fake_home_claude/) to prevent local machine environment contamination.

---

Project Architecture

EvalBench separates evaluation into distinct orchestrators, evaluators, generators, and scorers.

graph TD
    Config[Run Config YAML] -->|Loads| Orch[Orchestrator]
    Dataset[Dataset JSON] -->|Parsed| Orch
    Orch -->|Splits scenarios by Dialect/DB| Eval[Evaluator]
    
    subgraph Per Scenario Evaluation Loop
        Eval -->|Turn 1 Prompt| Gen[Agent / Model Generator]
        Gen -->|Tool Calls / Text| StepLogic{Is Multi-turn?}
        StepLogic -->|Yes| VUser[Simulated User LLM]
        VUser -->|Turn N Next Response| Gen
        StepLogic -->|Done / Final output| Exec[SQL Execution / Scorer Work]
    end
    
    Exec -->|Store Results| Report[Reporting: CSV / BigQuery]