oxy-evals

Oxy evals: draft, run, diagnose, repair

An Oxy eval is a .test.yml file: a list of prompts, each with the answer a correct agent gives. oxy test runs the target agent on every prompt several times and asks an LLM judge whether each answer carries the expected facts.

Drafting the tests and repairing the agent are one loop:

1. Draft cases and their expected answers from evidence.
2. Run them and read the results.
3. Diagnose each failure and repair the agent until it passes reliably.

The hard rule: the expected answer is the specification. Never weaken a test to make it pass. Don't loosen expected to match what the agent said, delete or skip a failing case, or rewrite a prompt to dodge the failure. If a test itself is wrong (a bad expected value, an ambiguous prompt), say so, show the evidence, and let the user decide. That's a different change from a repair, and you don't make it silently.

Reference docs (field tables, CI usage): https://www.oxygen-hq.com/docs/guide/build/test-evaluate/testing

What can be tested

• Only .agentic.yml agents. oxy test rejects any other target, including classic .agent.yml agents (that runtime was removed). Inline tests: blocks inside agent or automation files are gone too. A .test.yml file is the only entry point.
• Only the final answer text is judged. The judge sees the question, expected, and the agent's answer. It doesn't see the SQL, the tool calls, or the rows.
• tool: on a case is accepted but not enforced. It has no effect on pass/fail. Don't rely on it to assert that the agent used a particular path.

The .test.yml format

name: Sales performance          # optional display name
target: agents/sales.agentic.yml # path from the project root; see "target" below

settings:
  runs: 3                        # times each case runs (default 3)
  concurrency: 5                 # agent runs in flight at once (default 5)
  judge_model: judge             # a model `name:` from config.yml (default: first model there)

cases:
  - name: total-revenue-2025     # optional, but always add it: it makes --case usable
    prompt: What was total revenue across all stores in 2025?
    expected: Total revenue across all stores in 2025 was approximately $6.7 billion.
    tags: [revenue, critical]