evals-context

Evals Codebase Context

When to Use This Skill

Use this skill when the task involves:

• Modifying or debugging the evals execution infrastructure
• Adding new eval exercises or languages
• Working with the evals web interface (apps/web-evals)
• Modifying the public evals display page on roocode.com
• Understanding where evals code lives in this monorepo

When NOT to Use This Skill

Do NOT use this skill when:

• Working on unrelated parts of the codebase (extension, webview-ui, etc.)
• The task is purely about the VS Code extension's core functionality
• Working on the main website pages that don't involve evals

Key Disambiguation: Two "Evals" Locations

This monorepo has two distinct evals-related locations that can cause confusion:

Component	Path	Purpose
Evals Execution System	packages/evals/	Core eval infrastructure: CLI, DB schema, Docker configs
Evals Management UI	apps/web-evals/	Next.js app for creating/monitoring eval runs (localhost:3446)
Website Evals Page	apps/web-roo-code/src/app/evals/	Public roocode.com page displaying eval results
External Exercises Repo	Roo-Code-Evals	Actual coding exercises (NOT in this monorepo)

Directory Structure Reference

packages/evals/ - Core Evals Package

packages/evals/
├── ARCHITECTURE.md          # Detailed architecture documentation
├── ADDING-EVALS.md          # Guide for adding new exercises/languages
├── README.md                # Setup and running instructions
├── docker-compose.yml       # Container orchestration
├── Dockerfile.runner        # Runner container definition
├── Dockerfile.web           # Web app container
├── drizzle.config.ts        # Database ORM config
├── src/
│   ├── index.ts             # Package exports
│   ├── cli/                 # CLI commands for running evals
│   │   ├── runEvals.ts      # Orchestrates complete eval runs
│   │   ├── runTask.ts       # Executes individual tasks in containers
│   │   ├── runUnitTest.ts   # Validates task completion via tests
│   │   └── redis.ts         # Redis pub/sub integration
│   ├── db/
│   │   ├── schema.ts        # Database schema (runs, tasks)
│   │   ├── queries/         # Database query functions
│   │   └── migrations/      # SQL migrations
│   └── exercises/
│       └── index.ts         # Exercise loading utilities
└── scripts/
    └── setup.sh             # Local macOS setup script