judge-llm-batch

The calls.ndjson file lives at <state-root>/runs/<run-id>/calls.ndjson, where <state-root> is the resolved leerie state directory (default $HOME/.leerie/<basename>/, overridable via LEERIE_STATE_DIR / --state-dir / leerie.toml state_dir; /leerie-state inside the container) — never a path relative to CWD.

Output shape:

{
  "call_type": "<from filter>",
  "run_id": "<from file>",
  "judged_at": "<ISO-8601>",
  "judge_model": "claude-sonnet-4-6",
  "verdicts": [
    {
      "call_id": "<UUID from the capture>",
      "schema_ok": true,
      "schema_rationale": "<one sentence>",
      "factually_grounded": true,
      "factual_rationale": "<one sentence>",
      "hallucination_free": true,
      "hallucination_rationale": "<one sentence>",
      "pass": true,
      "worst_offender": "<optional — 1-line quote when any dimension failed>"
    }
  ],
  "aggregate": {
    "n": 0,
    "schema_pass": 0,
    "factual_pass": 0,
    "hallucination_free_pass": 0,
    "overall_pass": 0
  }
}

pass is true only when all three dimensions are true.

<execution_context> Arguments parsed from $ARGUMENTS:

• First positional: path to calls.ndjson (required). Can also be a <state-root>/runs/<run-id>/ directory — the skill will find calls.ndjson inside it.
• --call-type <name> (required): one of classifier, planner, reconciler, implementer, integrator, conformer. Filters the NDJSON to only lines with this call_type value.
• --run-id <id> (optional): if provided, resolves the path as <state-root>/runs/<run-id>/calls.ndjson, where <state-root> is the resolved leerie state directory (default $HOME/.leerie/<basename>/, overridable via LEERIE_STATE_DIR / --state-dir / leerie.toml state_dir; /leerie-state inside the container) — never a path relative to CWD.
• --out <path> (optional): explicit verdict output path; defaults to <ndjson-dir>/judge-out/<call_type>-verdicts.json.

The NDJSON line shape (from IMPLEMENTATION.md §10):

{
  "call_id": "<UUID v4>",
  "run_id": "<str>",
  "call_type": "<str>",
  "model": "<str>",
  "system_prompt": "<str>",
  "user_content": "<str>",
  "response_content": "<str>",
  "parsed_ok": true,
  "input_tokens": 0,
  "output_tokens": 0,
  "latency_ms": 0,
  "success": true,
  "ts": "<ISO-8601>"
}

</execution_context>

The judge skill operates post-run: it reads the archive, scores a batch, and writes verdicts. The llm-self-heal skill consumes these verdicts to propose prompt patches.

Each call_type maps to exactly one system-prompt source (IMPLEMENTATION.md §10 call_type → prompt table):

call_type	Prompt source
classifier	prompts/classifier.md
planner	prompts/planner.md
reconciler	prompts/reconciler.md
implementer	prompts/implementer.md
integrator	prompts/integrator.md
conformer	prompts/conformer.md

The system_prompt field in the NDJSON capture is the actual verbatim text injected — so the judge can always derive what the worker was asked to do from the capture alone.

Step 1: Locate and read the NDJSON file

Resolve the input path from $ARGUMENTS. If the path is a directory, append /calls.ndjson. Read line-by-line and parse each line as JSON.

Filter to lines where call_type matches the --call-type argument.

If zero lines match, emit: No captures for call_type=<name> in <path> and stop.