snyk-vulnbench: Skill for Claude Code

Benchmark Add New Fixture

Instructions

Drive a new benchmark fixture from an empty or existing fixtures/<name>/ tree through a complete, scoreable setup: SARIF-backed ground truth, Snyk parser alignment, eval tasks, and verification. The outcome is that pnpm run benchmark (once the rest of the task matrix loads) can run find-vulns and fix-vulns against the fixture and snyk-code command configs produce findings that match ground truth types.

Principles (read before executing):

• VulnBench 1.0 ground truth lives in fixtures/<fixture-name>/findings.json. VulnBench 2.0 attacker-reachable ground truth lives in findings-attacker-reachable.json and is selected by task metadata ("groundTruth": "attacker-reachable"). Both stay outside project/, where the sandbox prevents the agent from reading them. See docs/benchmark-management.md and docs/benchmark.md.
• Every fixture root also contains fixture.json, outside project/. It is the project-level metadata and provenance manifest; keep vulnerability types, code flows, payloads, and answer-key details in the findings files instead.
• Find-vulns scoring matches findings to known vulns by VulnType only, in a greedy order: iterate parsed findings in array order; each finding consumes the first unmatched ground-truth row with the same type. Therefore vulnerabilities[] order should mirror snyk code test results[] order whenever multiple rows share a type (e.g. two hardcoded-credentials, two other or two of any same type).
• Snyk maps SARIF results[].ruleId (e.g. javascript/TooPermissiveCorsHeader) to benchmark type via mapRuleId() in src/parsers/snyk-code.ts. Driver metadata in runs[0].tool.driver.rules (id, name, shortDescription.text, properties.cwe) names the vulnerability class — use it to pick or add VulnType strings and regex patterns.

Read docs/benchmark-management.md (Adding a New Eval Task, Ground-Truth JSON, Updating When You Add a New Vulnerability Type, Maintaining Snyk Code ruleId mappings) and docs/benchmark.md (Command configs and Snyk Code, How Vuln Type Matching Works) before editing if you are unsure of field shapes or scoring behavior.