regression-sweep

Regression Sweep

Walk the entire validated/*.json tree, re-fire each finding's poc.py, compare output against the recorded poc_output.txt, and write a weekly drift report. Mounted onto the cloud-agent task #4.

Trigger

Cron weekly (default Mondays 02:00 UTC). May also be invoked ad-hoc after a major patch deployment.

Workflow

1. Index validated findings. Glob validated/*.json and resolve each entry's FINDING_DIR (under findings/finding-NNN/).
2. Per finding:
   • Re-run python3 poc.py with a 60-second timeout.
   • Capture stdout/stderr into findings/finding-NNN/evidence/validation/regression-{week}-rerun.txt.
   • Diff against findings/finding-NNN/evidence/validation/poc-rerun-output.txt (the validator's original re-run output) using a normalized line-set comparison (strip timestamps, request IDs, ephemeral tokens).
   • Re-check the finding's CVE via tools/nvd-lookup.py — has severity changed?
3. Classify each finding into one of:
   • still_valid — re-run matches baseline within tolerance, CVSS unchanged.
   • drift_severity — re-run matches, but CVSS shifted ≥1.0 (NVD re-scored).
   • newly_invalid — re-run output diverges, exploit no longer fires. Likely patched.
   • newly_revalidated — finding had been marked REJECTED later, but now fires again. Regression.
   • inconclusive — re-run errored (network, target unreachable). Retry next sweep.
4. Write report to artifacts/regression-{YYYYWww}.json + human-readable regression-{YYYYWww}.md. Transitions (newly_revalidated, drift_severity, newly_invalid) appear in the report under explicit headers for analyst review.

Output

{OUTPUT_DIR}/
  artifacts/
    regression-{YYYYWww}.json         # machine-readable result
    regression-{YYYYWww}.md           # human summary
  findings/finding-NNN/evidence/validation/
    regression-{YYYYWww}-rerun.txt    # captured re-run output

regression-{week}.json schema:

{
  "week": "2026W19",
  "swept_at": "2026-05-13T02:00:00Z",
  "counts": {"still_valid": 47, "drift_severity": 2, "newly_invalid": 5, "newly_revalidated": 1, "inconclusive": 3},
  "findings": [
    {"finding_id": "finding-012", "asset": "asset42", "cve": "CVE-2024-12345",
     "verdict": "newly_invalid", "reason": "PoC output diverged: response now 404",
     "baseline_cvss": 9.8, "current_cvss": 9.8}
  ]
}