agentv-eval-writer

AgentV Eval Writer

Comprehensive docs: https://agentv.dev Promptfoo parity matrix: https://agentv.dev/docs/reference/promptfoo-parity/

Authoring Principle

Treat YAML as the canonical portable model. Prefer authoring .eval.yaml / EVAL.yaml first, then use TypeScript helpers, Python scripts, or executable graders only when they lower to the same fields or when the evaluation logic must actually run code.

Eval files define what is tested and how it runs: prompts, datasets, assertions, task fixtures, top-level providers, and suite run controls. Use field-local file refs such as tests: file://..., prompts: file://..., default_test: file://..., and environment: file://.... String-valued tests and string entries inside tests[] are raw-case refs for direct paths, directories, and globs. Run several full eval suites directly with CLI multi-file selection and tags. Use scoped run: on individual tests only for threshold, repeat, timeout_seconds, and legacy budget_usd; keep provider selection at top-level providers or CLI --provider, put suite budget caps under evaluate_options.budget_usd, authored concurrency under evaluate_options.max_concurrency, suite repeat policy under evaluate_options.repeat, coding-agent testbed setup under environment, provider environment overrides under env, and lifecycle hooks under extensions.

Use @agentv/sdk for TypeScript helper imports. Do not use @agentv/eval for new evals, examples, scaffolds, or skill guidance; it was a deprecated compatibility package and has been removed from this repository.

Authoring Checklist

• Put grading criteria in assert, not in test-level criteria. Plain assertion strings become an llm-rubric grader.
• Prefer plain assertion strings for semantic checks when the default rubric grader can judge them. Use type: llm-rubric for structured criteria, custom prompts, custom grader providers, or assertion-level transforms. Use type: agent-rubric when the grader itself must be an agent-capable provider that can inspect the workspace. Use type: script when grading must execute code.
• Put reference answers in tests[].vars.expected_output or default_test.vars.expected_output, and consume them with an explicit assertion such as type: llm-rubric with value: "Matches the reference answer: {{ expected_output }}". Do not write criteria, scoring instructions, or "the agent should..." rubric prose as the reference answer.
• For historical or repo-state evals, materialize the repo through a pinned environment setup recipe. Mentioning a SHA only in prompt prose is not enough because the agent needs an actual checkout to inspect.