agentv AGENTS.md

AgentV Agent Guide

This file is the root index for repo-facing agent instructions. It carries baseline rules that always apply plus a Routing table that maps your change to the deeper .agents/*.md guide you must read before starting that work. The rules in this file are a summary; each linked guide is authoritative for its area, so a matching Routing trigger means "read the guide," not "the summary is enough." Read STRATEGY.md plus ROADMAP.md before making product-boundary calls.

Product Direction

AgentV aims to be the repo-native, workspace-native evaluation framework for AI agents.

• Repo-native evals: run against real repos, multi-repo workspaces, setup scripts, and existing harnesses.
• Zero-infra local to CI: keep the default path lightweight so the same eval contract works on a laptop and in CI.
• Portable run artifacts: treat run bundles, traces, and summaries as the source of truth for comparison, gating, and export.
• Adapter boundaries: integrate with Phoenix, Harbor, Opik, and provider-specific systems through narrow adapters instead of absorbing their concepts into core.
• AI-native extensibility: keep the core small and composable so engineers and coding agents can extend it with plugins, wrappers, and harness-specific glue.

Eval authoring mental model:

• AgentV adopts Promptfoo-compatible matrix authoring where it helps: an eval expands prompts x tests/vars x providers, then applies repeat samples/retries as run policy. Compatibility is at the provider declaration layer, not a claim that every AgentV config runs in Promptfoo unchanged.
• Author systems under test as providers. A provider entry id names the backend/spec; label is the stable AgentV selection and result identity.
• Treat AgentV as a Promptfoo-compatible superset for provider declarations. Promptfoo-shaped provider entries can be strings like openai:gpt-4.1-mini, complete package provider strings like package:@agentv/promptfoo-providers:CodexCliProvider or package:@agentv/promptfoo-providers/codex-cli:Provider, option objects with id, label, config, env, prompts, transform, delay, and inputs, or provider maps like { "openai:gpt-4": { label, config } }. Package provider strings must include the exported class/function segment after the final colon. AgentV-only fields such as top-level environment and provider-local runtime/testbed overlays remain AgentV semantics; Promptfoo may ignore or strip unknown keys and will not execute AgentV environment setup without a transpiler or wrapper.
• Use environment recipes for coding-agent testbeds, including host/Docker setup, repo materialization, fixtures, services, and cwd. Do not use Promptfoo extensions or public workspace authoring as the canonical testbed contract.
• Use top-level env for provider/eval environment variables. Use extensions for lifecycle hooks. Use field-local file:// refs for reusable prompts, tests, defaults, and environments.
• Use tags and run-bundle metadata for grouping and Dashboard navigation. Do not use experiment path buckets, Vercel path layout, or model-as-experiment grouping as canonical AgentV semantics.
• AgentV run bundles, traces, transcripts, datasets, indexes, and Git-backed artifacts stay AgentV-owned. Do not design an Opik export path or Phoenix projection path for them; Phoenix correlation is link-out only when external_trace metadata already exists.