arbor-agent-merge-eval

Arbor Merge And Eval

Use this whenever scores, metadata, merge decisions, or final validation are involved.

Dataset Discipline

• B_dev is for routine iteration, executor experiments, score tracking, and idea selection.
• B_test is for milestone checks only: before merging a branch and at final report time.
• If B_test diverges from B_dev, do not hand-wave it. Investigate overfitting, noise, data split mismatch, or eval contamination.

Tree Metadata

Persist evaluation metadata early and update it after merges:

• baseline_score: unmodified B_dev score.
• trunk_score: current trunk B_dev score.
• test_baseline_score: unmodified B_test score.
• test_trunk_score: current trunk B_test score.
• eval_cmd: B_dev command.
• eval_cmd_test: B_test command.
• eval_timeout, eval_retries, eval_retry_base_delay, eval_retry_max_delay.
• dataset_info: paths and split descriptions.
• metric_direction: maximize or minimize.
• trunk_branch: non-protected branch that receives verified merges.
• submission_path, sample_submission_path.

Use {cwd} and {node_id} placeholders. Example:

cd {cwd} && uv run python run_eval.py --split dev --run-name {node_id}

Score Semantics

• Tree node score is an absolute B_dev metric value.
• Merge verification uses B_test.
• metric_direction controls improvement:
  • maximize: higher is better.
  • minimize: lower is better.
• Do not compare deltas with absolutes.
• If output has JSON with score, prefer it. Otherwise extract the primary metric from text (primary_score, score, accuracy, acc, etc.).

Merge Procedure

Native GitMergeBranch:

1. Refuses target main or master.
2. Resolves target to configured trunk_branch. main/master are base branches, not merge targets.
3. Creates an isolated worktree at source_branch.
4. Runs eval_cmd_test with {cwd} and {node_id} substituted.
5. Retries transient failures if configured.
6. Extracts verified B_test score.
7. Rejects the merge if B_test does not improve over test_trunk_score or test_baseline_score.
8. Checks plugin protected paths and required outputs.
9. Merges source into trunk with --no-ff.
10. Reports the verified test score and instructs the coordinator to update tree metadata and node status.