cve-hunt-benchmark-harness

CVE-Hunt Benchmark Harness

Derived from T3MP3ST's src/benchmark/index.ts and bench/ corpora (Apache-2.0). The point of a ground-truth benchmark: measure finding quality — precision AND recall — instead of counting scanner output.

1. Challenge definition

A benchmark challenge bundles everything needed to score a hunt:

id, name, description, difficulty(easy|medium|hard|expert),
target{address, type, zone},
tasks[{name, description, phase, operatorType, priority}],
groundTruth[{id, title, severity, category, matchKeywords[], points}],
timeLimitSec

• groundTruth is the answer key: every KNOWN vulnerability in the target with its severity, category, matchKeywords (what a correct finding should mention), and a points value.
• tasks defines the hunting workflow phases so runs are comparable.
• timeLimitSec bounds the run — time-to-finding is a scored metric.

2. Scoring — precision, recall, F1

For a run that produced findings[] against groundTruth[]:

• A finding matches a ground-truth vuln when it hits the vuln's matchKeywords (keyword containment over the finding's title/description/category).
• truePositives = findings that matched ≥1 ground-truth vuln.
• falsePositives = findings that matched nothing.
• precision = TP / (TP + FP) — how trustworthy the tool's output is.
• recall = matched_ground_truth / total_ground_truth — how much was found.
• f1 = 2 * precision * recall / (precision + recall).
• pointsScored sums matched vulns' points; report scorePercent against maxPoints.
• timeToFirstFinding = seconds until the first valid finding; also report findingsPerMinute and totalDurationSec.

Severity breakdown: per severity, {found, total} — a tool that finds all the lows but none of the criticals is not done.

3. Decoys — measuring the false-positive rate honestly

Include CLEAN samples in the corpus (T3MP3ST's DECOY-clean-c, DECOY-clean-java pattern): code that LOOKS vulnerable but is not. Findings against decoys are false positives by construction. A hunter that scores well on vulnerable samples but flags decoys is pattern-matching, not analyzing — and the decoy FP rate is the number that proves it.