snyk-vulnbench: Skill for Claude Code

Benchmark Add New Category

Instructions

Add a new eval category so a set of tasks can be filtered with --category <id> and scored with the appropriate pipeline. The outcome is that the new category appears in dry-run output, tasks using it load and score correctly, and docs reflect the change.

Categories determine three things: (1) how tasks are grouped for CLI filtering, (2) what system/user prompts the agent receives, and (3) which scoring pipeline runs (F1 find-vulns or LLM-judge fix-vulns). Adding a find-* category reuses the existing F1 scorer; adding a fix-* category reuses the LLM judge scorer.

---

Step 1: Gather requirements

Confirm with the user:

1. Category ID — kebab-case string used in --category flag (e.g. llm-find-vulns).
2. Category constant name — UPPER_SNAKE for the EVAL_CATEGORIES key (e.g. LLM_FIND_VULNS).
3. Scoring type — does this category use find-vulns scoring (F1 precision/recall) or fix-vulns scoring (LLM judge)? Most new categories are find-vulns variants.
4. Which existing tasks should move to this category (task JSON files to update).
5. Prompt customization — any special emphasis for the system prompt (e.g. "focus on LLM-specific risks") or whether the default find-vulns/fix-vulns prompt is fine.

Done when: you have the category ID, scoring type, affected tasks, and prompt direction.

---

Step 2: Add the category to EVAL_CATEGORIES

Edit src/types.ts. Add a new entry to the EVAL_CATEGORIES object before the closing } as const satisfies Record<string, EvalCategory>. Each entry needs:

• id — the CLI-facing category ID string
• name — human-readable name
• description — one-sentence description
• defaultSystemPrompt — the system prompt agents receive (must include the FINDINGS_JSON output format block for find-* categories)
• defaultPrompt — the user message sent to the agent

For find-* categories, copy the FINDINGS_JSON format block from FIND_VULNS verbatim — only customize the introductory text and defaultPrompt. The JSON schema must stay identical so the scorer can parse it.