openbench CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Essential Setup

• Always source the virtual environment before running Python commands: source .venv/bin/activate
• This project uses UV as the package manager, not pip
• Dependency management: When adding dependencies with UV, use >= constraints (e.g., uv add "package>=1.2.3")
  • Exception: inspect-ai must remain pinned to a specific version for stability
  • Use the latest stable version as the minimum to keep dependencies healthy and secure
  • Check latest versions with uv pip list --outdated or on PyPI

Key Commands

Development Setup

# Initial setup
uv venv && uv sync --dev
source .venv/bin/activate

Running Tests

# Run all unit tests
pytest

# Run integration tests (requires API keys)
pytest -m integration

# Run specific test file
pytest tests/test_registry.py

# Run with coverage
pytest --cov=openbench

Code Quality

# Format code
ruff format .

# Lint code
ruff check .

# Type checking
mypy .

# Run all pre-commit hooks
pre-commit run --all-files

Using the CLI

# List available benchmarks
bench list

# Describe a specific benchmark
bench describe mmlu

# Run evaluation
bench eval mmlu --model groq/llama-3.1-70b --limit 10

# View previous results
bench view

Installing Dependencies

# Install core dependencies only (runs most benchmarks)
uv sync

# Install with specific benchmark dependencies
uv sync --group scicode       # For SciCode benchmark
uv sync --group jsonschemabench # For JSONSchemaBench

# Install with development tools
uv sync --dev

# Install everything
uv sync --all-groups

Publishing to PyPI

# Build the package
uv build

# Publish to PyPI (requires PyPI API token)
uv publish

Architecture Overview

Project Structure

• src/openbench/ - Main package directory
  • _cli/ - CLI commands implementation
  • datasets/ - Dataset loaders (MMLU, GPQA, HumanEval, SimpleQA)
  • evals/ - Benchmark implementations built on Inspect AI
  • metrics/ - Custom scoring metrics
  • scorers/ - Scoring functions used across benchmarks
  • utils/ - Shared utilities
  • _registry.py - Dynamic task loading system
  • config.py - Benchmark metadata and configuration