holmesgpt-eval

HolmesGPT Eval — Agent-Native (Self-Answer)

Primary purpose

• Evaluate the current agent against HolmesGPT's eval fixtures by having the agent answer each user_prompt itself.
• No HTTP bridge or shell wrapper for answering — the agent's own tools should be used to produce the answer string that gets scored.
• Supports two modes: bulk run (all or filtered tests) and individual run (single specific test case).

Agent-native flow (what the agent should do)

1. Check for existing fixtures:
   • If vendor/holmesgpt/tests/llm/fixtures/test_ask_holmes exists, skip fetch and use existing fixtures.
   • Otherwise, call scripts/fetch_fixtures.sh via exec to download/unzip HolmesGPT and return tests/llm/fixtures/test_ask_holmes.
2. Determine run mode:
   • Bulk run: If no specific test case is specified, iterate through all case directories (optionally filtered by tags).
   • Individual run: If a specific test case is specified (e.g., "01_how_many_pods", "09_crashpod"), run only that single test.
3. Iterate each case directory containing test_case.yaml:
   • Parse test_case.yaml and determine case controls before executing anything:
     • If tags are specified for the run, check if the test_case.yaml contains a matching tag in its tags field. If not, ignore this case completely (do not record, process, or report it).
     • If evaluation.correctness.expected_score == 0 → SKIP this case entirely (do not run before_test/after_test). Record as skipped with reason.
     • Determine evaluation mode for this case (see Scoring Rules → Mode selection).
   • For non-skipped cases:
     • Run before/after as bash scripts (materialize then execute):
       • Write the before_test block verbatim to .before.sh in the case directory.
       • Write the after_test block verbatim to .after.sh in the case directory.
       • Execute with a non-interactive shell and strict flags from the case directory:
         • before: bash --noprofile --norc -eo pipefail ./.before.sh
         • after: bash --noprofile --norc -eo pipefail ./.after.sh
       • Always set cwd to the case directory so relative paths resolve.
       • Capture stdout, stderr, and exit code for both before and after.
     • Ask the agent itself the user_prompt (inline in the current session) and capture the raw textual reply.
       • The agent MUST answer the prompt directly without creating script files (.py, .sh, etc.) unless explicitly required by the test case.
       • The agent should use available tools to investigate and provide answers in the current session.
       • If the test runs longer than 5 minutes, stop the test immediately and mark it as failed with reason "timeout_exceeded".
     • Score the agent's answer against expected_output.
     • Always run after_test (as above) to clean up, even if before/ask/score failed.
4. Write results under evals/holmesgpt-eval/results/:
   • results.json — array of case results (prompt, expected, output, pass/fail, skipped, details)
   • report.md — concise summary (pass/fail/skip counts; per-case status, expected list, output, missing elements if any)
   • latest-results.md — points to the latest report

Scoring rules for the agent

• Mode selection (per-case):

  • Default to loose mode.

  • If the test_case.yaml contains evaluation.correctness.type, use that value to set the mode for this case.

    • Supported values (case-insensitive): strict, loose
    • Unknown/absent values: fall back to loose

  • Example YAML snippet:

    evaluation: correctness: type: strict

• Modes:

  • strict: pass only if ALL expected elements are sufficiently present in the output
  • loose: pass if the output reasonably matches the expected content overall

• “Sufficiently present” guidance:

  • Case-insensitive; ignore trivial punctuation/whitespace differences
  • Numeric normalization OK when unambiguous (e.g., 14 vs fourteen)
  • Minor paraphrases allowed if the essence is clear; elements may span multiple sentences

• Deterministic baseline when unsure:

  • strict = all expected substrings present (case-insensitive)
  • loose = any expected substring present