evaluate-attempt

Evaluate Benchmark Attempt

Overview

This SOP evaluates a completed brazil-bench attempt against the spec.md requirements, capturing metrics for comparison across orchestration patterns. Supports both Python and Swift/iOS implementations.

Parameters

• attempt_repo (required): Repository name (e.g., attempt-3)
• output_dir (optional, default: ./results): Where to write evaluation results

Steps

0. Detect Project Language

Identify the primary language/platform of the implementation.

Detection Commands:

cd ./reviews/{attempt_repo}

# Python indicators
ls pyproject.toml setup.py requirements.txt 2>/dev/null

# Swift/iOS indicators
ls Package.swift *.xcodeproj *.xcworkspace 2>/dev/null

# Check file extensions
find . -name "*.py" -not -path "./.venv/*" | head -5
find . -name "*.swift" | head -5

Language Detection Matrix:

Files Found	Language	Test Framework
pyproject.toml, *.py	Python	pytest
Package.swift, *.swift	Swift Package	swift test
*.xcodeproj, *.swift	iOS/Xcode	xcodebuild test
Both Python and Swift	Multi-language	Run both

Constraints:

• You MUST detect the language before running tests
• You MUST use appropriate commands for the detected language
• You SHOULD note the detected language in the report

1. Clone Attempt

Fetch the attempt repository for local analysis.

Constraints:

• You MUST clone into ./reviews/{attempt_repo}
• You MUST verify the clone succeeded before proceeding
• You MUST NOT modify any files in the cloned repo

gh repo clone brazil-bench/{attempt_repo} ./reviews/{attempt_repo}

2. Verify Spec Integrity

Confirm the spec.md was not modified from the template.

Constraints:

• You MUST compare spec.md against the template version
• You MUST fail the evaluation if spec.md was modified
• You SHOULD use a checksum comparison

gh repo clone brazil-bench/benchmark-template ./reviews/_template --depth 1
diff ./reviews/{attempt_repo}/spec.md ./reviews/_template/spec.md