file-run-issues

File Run Issues

Overview

evaluate-run produces a findings.jsonl per run — one JSON object per observation. This skill aggregates those findings into assessment.json, a compact machine-readable summary used by FindingsScorer and downstream reporting. It does not create beads issues or GitHub issues.

Parameters

• run_dir (required): Same path evaluate-run used, e.g. experiment-1/runs/language=rust_model=opus_tooling=beads/rep2/
• min_severity (optional, default: info): Skip findings below this severity when computing counts. Options: critical, high, medium, low, info
• dry_run (optional, default: false): Print the assessment JSON without writing assessment.json

Steps

1. Load findings

test -f {run_dir}/findings.jsonl || { echo "no findings.jsonl — run evaluate-run first"; exit 1; }

Read the file line-by-line. Each line is one finding with at minimum these fields:

{"id": "R3", "kind": "requirement_missing", "severity": "high", "title": "...", "evidence": "...", "suggestion": "..."}

2. Filter by severity

Drop findings whose severity is below min_severity. The severity ordering from highest to lowest is: critical, high, medium, low, info.

3. Count severities

Count how many findings fall into each severity bucket:

{"critical": 0, "high": 2, "medium": 5, "low": 3, "info": 1}

4. Compute penalty_score

start = 1.0
subtract: critical * 0.25 + high * 0.10 + medium * 0.03 + low * 0.01
clamp result to [0.0, 1.0]

A run with no findings scores 1.0. A run with one critical finding scores 0.75. A run with four critical findings scores 0.0 (clamped).

5. Collect top findings

Select the top 5 findings by severity (critical first, then high, medium, low, info). Within the same severity level, preserve the original order from findings.jsonl. Include all fields from the original finding object.

6. Compute requirement_coverage

Count findings with kind in requirement_missing or requirement_partial — these represent requirements the agent did not fully implement. Estimate total requirements from R<N> IDs present in findings plus any implemented ones (inferred from evaluation.md if available, otherwise estimate from the highest R-number seen).