benchmark-report

Benchmark Report

Aggregate benchmark data from interactive sessions (JSONL logs) and SOVA autonomous runs (DB), then generate a velocity comparison report.

Arguments: $ARGUMENTS

Instructions

Step 1: Parse Arguments

Extract optional flags from $ARGUMENTS:

• --since YYYY-MM-DD: only include data from sessions after this date
• --issues N,N,N: only include these specific issue numbers (comma-separated)
• --update-docs: write the report to docs/benchmark-results.md (default: terminal output only)

If $ARGUMENTS is empty or contains none of these flags, use all available data and output to terminal only.

Step 2: Collect Interactive Data (JSONL)

Find all benchmark log files:

ls .claude/benchmark/issue-*.jsonl 2>/dev/null

If no files exist, note "No interactive benchmark data found" and skip to Step 3.

For each file:

1. Extract the issue number from the filename. Files named issue-null.jsonl or with non-numeric IDs: count them in the file summary but exclude from per-issue metrics.
2. If --issues was specified, skip files whose issue number is not in the list.
3. If --since was specified, check the first session_start event's ts field; skip the file if it predates the cutoff.

For each included issue file, extract metrics using jq:

Wall clock (total across all sessions):

jq -s '
  [.[] | select(.event == "session_start") | .ts] | first as $start |
  [.[] | select(.event == "session_end" or .event == "session_summary") | .ts] | last as $end |
  {start: $start, end: $end}
' .claude/benchmark/issue-{N}.jsonl

For multi-session issues (multiple session_start/session_end pairs), sum the durations of each session separately.

Human idle time:

jq -s '
  [.[] | select(.event == "human_idle_start" or .event == "human_idle_end")]
  | [range(0; length; 2) as $i | (.[($i+1)].ts | split("T") | .[1] | split("Z") | .[0]) as $end_t |
     (.[$i].ts | split("T") | .[1] | split("Z") | .[0]) as $start_t |
     {start: .[$i].ts, end: .[($i+1)].ts}]
' .claude/benchmark/issue-{N}.jsonl

Compute human idle as the sum of all human_idle_start to human_idle_end intervals (in minutes). If an odd number of idle events exists (unclosed idle period), ignore the unpaired event.