re-evaluate

Re-Evaluate Attempt After Changes

Overview

This SOP monitors attempt repositories for closed issues and code changes, re-runs relevant evaluations, updates the main evaluation report, and files new issues for any regressions or newly discovered problems.

Parameters

• attempt_repo (required): Repository name (e.g., 2025-12-13-python-claude-hive)
• evaluation_path (optional, default: ./results/{attempt_repo}.md): Path to existing evaluation report
• full_reeval (optional, default: false): If true, run complete evaluation instead of targeted checks

Steps

1. Check Repository for Changes

Determine if the repository has changed since the last evaluation.

Constraints:

• You MUST check for new commits since last evaluation
• You MUST identify closed issues that triggered changes
• You MUST NOT proceed if no changes detected (unless full_reeval=true)

# Navigate to existing clone or re-clone
cd ./reviews/{attempt_repo} || gh repo clone brazil-bench/{attempt_repo} ./reviews/{attempt_repo}

# Fetch latest changes
git fetch origin

# Check for new commits on main/master
git log HEAD..origin/main --oneline 2>/dev/null || git log HEAD..origin/master --oneline

# Pull latest changes
git pull origin main 2>/dev/null || git pull origin master

# Get the latest commit info
git log -1 --format="%H %ai %s"

2. Identify Closed Issues

List issues that have been closed since the last evaluation.

Constraints:

• You MUST fetch all closed issues from the repo
• You MUST categorize issues by type ([Missing], [Test Quality], [Docs], etc.)
• You SHOULD note which issues were addressed vs closed without fix

# List all closed issues
gh issue list -R brazil-bench/{attempt_repo} --state closed --json number,title,closedAt,labels

# Get details of recently closed issues
gh issue list -R brazil-bench/{attempt_repo} --state closed --limit 20 --json number,title,body,closedAt

# Check issue comments for resolution notes
gh issue view {issue_number} -R brazil-bench/{attempt_repo} --json comments