judges adding-a-judge.instructions.md

Adding a New Judge — Step-by-Step

When adding a new judge to the Judges Panel, follow these steps exactly. All judges now load from agent files (agents/<id>.judge.md). The registry bootstraps from the agents directory at runtime. Evaluators remain in src/evaluators/ and are referenced via the script: field in the agent frontmatter.

ℹ️ Back-compat: legacy .agent.md files are still readable, but all new agents must use .judge.md.

The legacy side-effect imports in src/judges/index.ts have been removed. If you need to add a judge, follow the agent-native workflow below.

---

0. Create/Update Agent File (agents/<id>.judge.md)

Frontmatter fields:

• id, name, domain, rulePrefix, description, tableDescription, promptDescription
• script: relative path to evaluator module (e.g., ../src/evaluators/cybersecurity.ts)
• priority: integer ordering (lower runs earlier; 999 reserved for false-positive-review)

Body becomes the system prompt (persona + evaluation criteria). See existing agents for patterns.

You can regenerate agents from the registry with:

npm run generate:agents        # skip existing
npm run generate:agents:force  # overwrite
npm run validate:agents        # sanity check against registry

---

1. Create the Evaluator

Create src/evaluators/{judge-id}.ts with an analyze function:

import type { Finding } from "../types.js";

export function analyzeSupplyChain(code: string, language: string): Finding[] {
  const findings: Finding[] = [];
  const prefix = "SCS";
  // ... deterministic analysis logic ...
  return findings;
}

Key requirements:

• The function signature is (code: string, language: string) => Finding[]
• Use the same rulePrefix from the judge definition
• Import shared utilities from ./shared.js as needed
• Follow existing evaluators as templates (e.g. data-security.ts, cybersecurity.ts)

2. Create the Judge Definition (with self-registration)