graphjin-eval

GraphJin Eval

Use this skill when a user wants to create or run a GraphJin agent benchmark, add a real business question to the suite, establish a baseline, compare a candidate, or understand an evaluation failure.

Rules

• Always use graphjin eval commands with --json for machine-readable state.
• Never edit eval/suite.yml, hidden oracle definitions, tolerances, reward weights, reports, or baseline files by hand.
• Never hand-edit website/data/benchmarks/<benchmark>.yaml or website/content/benchmarks/<benchmark>/runs/. graphjin eval publish is the only supported writer; it writes one row and one run page and never runs Git.
• Treat eval publish --label as presentation only. Supersession uses the normalized provider and model identity, not the display label.
• Use graphjin eval rm <task-id> as the supported task-removal path; never delete a task from the suite file manually.
• Never invent an oracle, field, threshold, or business interpretation.
• Treat exit code 2 as a broken suite, not a model regression.
• Treat the suite generator version as part of the scoring contract. Bump eval.GeneratorVersion whenever generated task semantics change, including method-rule dialect support, then regenerate every committed/frozen suite. A binary must refuse suites from any other generator version.
• Treat exit code 3 as an environment problem, not a model regression.
• Treat exit code 130 as an interrupted checkpoint. Resume it; do not score it.
• Provider-backed commands can incur cost. Explain the expected call count and use --yes only after the user approves provider traffic.
• Read both usage views in the report: finalized tokens measure agent efficiency, while provider tokens include failed attempts and retries. On a compatible baseline, report the total-token and tokens-per-episode direction and percentage; treat cross-model or differently shaped comparisons as advisory.
• Check provider_usage.complete. If false, unknown_attempts counts provider calls that returned no usage and all recorded token totals are lower bounds. Never compare token percentages across accounting versions, providers, models, configured max_steps, or incomplete provider usage.
• Before calling two runs a same-build comparison, require matching provenance.binary_fingerprint. It is the SHA-256 of the exact CLI executable and catches runtime changes that do not alter prompt hashes.
• Full prompts, answers, rows, and executed queries stay in local episode files. Share reports, not episode files, unless the user explicitly asks for the private trajectory.
• Failed/interrupted provider attempts stay under .graphjin-evals/attempts/. They are private, and no persisted file may contain a credential.
• Use GOOGLE_API_KEY as the canonical Google credential name.
• Publishing does not refuse a low score. Never rerun a completed benchmark to make the public board look better; publish the observed result with its accepted state.
• Do not publish a report marked scoring_suspect until the scorer/runtime mismatch is understood. --allow-suspect-scoring is an explicit audited override, not a routine publishing flag.
• Publish with the exact binary that ran the benchmark. A missing graphjin_commit or mismatched binary_fingerprint is a broken provenance chain and must be rerun, not waived.
• Never publish an off-suite run unless the user explicitly asks for it. When asked, use --allow-off-suite and verify it appears as unranked with the mismatch reason.