test-health

/test-health — Test Suite Health Audit

Audit the test suite for flaky tests, dead/trivial tests, coverage gaps on recent changes, missing assertions, and structural issues. Produce a health report with prioritized recommendations.

Arguments

• $ARGUMENTS may contain:
  • --flaky-runs N — number of times to run the suite for flaky detection (default: 5)
  • --coverage — only run the coverage gap analysis (skip flaky/dead detection)
  • --quick — skip flaky detection (most time-consuming), run everything else
  • No arguments — full audit

Phase 0 — Pre-flight

1. Confirm we're in the codegraph repo root
2. Verify vitest is available: npx vitest --version
3. Parse $ARGUMENTS:
   • FLAKY_RUNS=N from --flaky-runs N (default: 5)
   • COVERAGE_ONLY=true if --coverage
   • QUICK=true if --quick
4. Discover all test files:

   find tests/ \( -name '*.test.js' -o -name '*.test.ts' \) | sort
   

5. Count total test files and categorize by directory (integration, parsers, graph, search, unit)

Phase 1 — Flaky Test Detection

Skip if COVERAGE_ONLY or QUICK is set.

Run the full test suite FLAKY_RUNS times and track per-test pass/fail:

RUN_DIR=$(mktemp -d /tmp/test-health-XXXXXX)
for i in $(seq 1 $FLAKY_RUNS); do
  timeout 180 npx vitest run --reporter=json > "$RUN_DIR/run-$i.json" 2>"$RUN_DIR/run-$i.err"
  exit_code=$?
  if [ $exit_code -eq 124 ]; then
    echo '{"timeout":true}' > "$RUN_DIR/run-$i.json"
  elif [ $exit_code -ne 0 ] && [ $exit_code -ne 1 ]; then
    # Use jq to safely JSON-escape stderr content (may contain quotes, newlines, backslashes)
    stderr_content=$(cat "$RUN_DIR/run-$i.err")
    jq -n --argjson code "$exit_code" --arg stderr "$stderr_content" \
      '{"error":true,"exit_code":$code,"stderr":$stderr}' > "$RUN_DIR/run-$i.json"
  fi
done

For each run, parse the JSON reporter output from $RUN_DIR/run-$i.json to get per-test results.

After all runs are parsed and analysis is complete, clean up the temporary directory:

rm -rf "$RUN_DIR"