run-eval

Run Eval Workflow

Available eval suites

!ls connectors/*.eval.yaml .elliot/eval/*.json 2>/dev/null || echo "(no eval suites found — see example at connectors/my-saas.eval.yaml)"

Steps

1. Find or create the eval suite

elliot_run_eval accepts either:

• path — a path to an *.eval.yaml (preferred) or *.json suite file, OR
• suite_id — the bare id of a JSON suite under .elliot/eval/<id>.json.

If a *.eval.yaml already lives next to the connector, call elliot_run_eval(path="connectors/<slug>.eval.yaml").

If none exists, offer to create one. The canonical shape is YAML:

name: My Connector Evals
connector: my-connector
version: "1.0.0"
cases:
  - id: list-items-no-filter
    description: Returns at least one row with the expected fields
    tool_id: list_items
    arguments: {}
    expect:
      no_error: true
      min_rows: 1
      fields_present: [id, name]
      max_token_estimate: 500
  - id: list-items-rejects-bad-status
    description: A bad enum value is rejected, not silently ignored
    tool_id: list_items
    arguments: { status: not-a-real-status }
    expect:
      error_code: INVALID_PARAM_VALUE

Cover the error paths, not just the happy path. A tool that returns good rows for good input but silently accepts bad input is not agent-ready — the agent gets an empty or wrong result with no signal. Add at least one case per tool that asserts a bad argument is rejected: set expect.error_code to the code you expect (INVALID_PARAM_VALUE for a bad enum/bound, MISSING_PARAM for an omitted required param, UNKNOWN_PARAM for a stray key). The case passes only if the tool raises that code, and fails if the call succeeds — so you prove the contract rejects what it should. (In a legacy JSON suite the equivalent is "expect_error": "INVALID_PARAM_VALUE" on the case.)

Evaluate your skills, not just your tools. A deterministic skill is served as one callable tool, so a case's tool_id can name a skill id — the runner executes the whole step chain end-to-end (each step bound to the last) and the same expect block applies to the skill's final output. Add a case for every multi-step workflow you ship (fields_present on the fields the last step should return, max_token_estimate on the one-call cost) so a connector's headline workflows are validated before publish, not just their individual steps.