bkit-evals

bkit Evals — Skill Quality Evaluation Runner

v2.1.11 Sprint β FR-β2. Wraps evals/runner.js with input validation, result persistence, and structured reporting. Replaces the bare node evals/runner.js <skill> invocation that previously required users to remember argv structure and ignored timeout / sandbox concerns.

Arguments

If no argument is provided, render the same output as list.

Behavior

run <skill>

1. Validate skill against /^[a-z][a-z0-9-]{0,63}$/. Reject anything else (no shell metacharacters, no slashes, no spaces) — see Security below.
2. Spawn node evals/runner.js --skill <skill> via child_process.spawnSync (argv form, no shell). Default timeout 30 s, max 120 s. The --skill flag form is mandated by the runner CLI and locked by L3 contract test.
3. Capture stdout / stderr. Parse the trailing JSON block via balanced-brace fallback (string-aware).
4. Apply fail-closed defense: if parsed === null and stdout includes Usage:, return reason: 'argv_format_mismatch'; if parsed === null otherwise, return reason: 'parsed_null'. Exit code 0 alone NEVER implies success — the parsed JSON must be present.
5. Persist the structured result to .bkit/runtime/evals-{skill}-{ISO timestamp}.json with stdout/stderr tails (2000 chars each), parsed payload, and reason field.
6. Render a one-line summary in the chat:
   • exit code
   • parsed pass/fail counts (if available)
   • path of the persisted result file

list

1. Read evals/config.json to enumerate skill classifications.
2. For each classification (workflow, capability, hybrid), list skills that have evals/{classification}/{skill}/eval.yaml.
3. Render a category-grouped table with skill name + a one-line note from the eval YAML (description field if present).