compare-attempts

Compare Benchmark Attempts

Overview

This SOP compares evaluated brazil-bench attempts across multiple dimensions to produce a ranked leaderboard and detailed comparison summary. It supports up to 10 attempts in the "Top 10" format, automatically pruning lower-ranked entries when more are added.

Parameters

• results_dir (optional, default: ./results): Directory containing evaluation reports
• output_file (optional, default: ./results/LEADERBOARD.md): Output comparison file
• max_entries (optional, default: 10): Maximum entries in leaderboard

Steps

1. Discover Evaluation Results

Find all completed evaluation reports in the results directory.

Constraints:

• You MUST scan for *.md files in the results directory (excluding LEADERBOARD.md)
• You MUST verify each file is a valid evaluation report (has "# Evaluation:" header)
• You MUST extract the attempt name from each report
• You SHOULD skip any malformed or incomplete reports

ls {results_dir}/*.md | grep -v LEADERBOARD.md

2. Extract Metrics from Each Report

Parse each evaluation report to extract comparable metrics.

Constraints:

• You MUST extract all metrics from the Metrics table in each report
• You MUST extract the pattern/orchestration type
• You MUST extract spec compliance (X/Y format)
• You MUST extract test counts if available
• You MUST extract git metrics (commits, duration, fix commits)
• You SHOULD normalize duration to hours for comparison
• You MUST handle missing metrics gracefully (mark as "N/A")
• You MUST check for evaluation methodology differences (see Step 2a)

Required Metrics:

Metric	Source	Notes
Pattern	Summary section	swarm, hive, solo, crew, custom
Spec Compliance	Summary or checklist	Count of implemented requirements
Lines of Code	Metrics table	Source code only
Files	Metrics table	Python files in src
Dependencies	Metrics table	Package count
Commits	Metrics table	Total git commits
Duration	Metrics table	Development time
Fix Commits	Git Analysis	Commits with "fix" in message
Test Scenarios	Test Summary	BDD or unit test count
Token Usage	Token Usage section	From prompts.txt if available
Phase Durations	Duration Breakdown	Initial coding, tests working, human intervention