adaptation-parity

Adaptation + Parity

This repo's thesis: the agent you ship is the agent you benchmark. This skill makes that real in two steps — adapt an agent to BenchFlow, then verify parity (it behaves the same inside the eval harness as standalone).

1. Adapt (agent → BenchFlow ACP)

An agent becomes a BenchFlow agent by speaking ACP over stdio and registering via the public register_agent extension point. Pattern (see ai-sdk/acp, ai-sdk/harness-pi for worked examples):

• server.mjs — a pure-JS ACP-over-stdio server. Newline-delimited JSON-RPC; stdout = protocol only (logs → stderr). Handle initialize, session/new, session/set_model, session/prompt (run the agent loop, stream session/update events, return {stopReason, usage}), session/cancel. Map the agent's stream → ACP: text→agent_message_chunk, reasoning→agent_thought_chunk, tool call→tool_call (name+args in title), tool result→tool_call_update. Route the model at OPENAI_BASE_URL (the gateway).
• register.py — register_agent(name, install_cmd, launch_cmd, protocol="acp", api_protocol="openai-completions", env_mapping={BENCHFLOW_PROVIDER_*→agent vars}, acp_model_format="bare", requires_env=[]). install_cmd bootstraps node, base64-deploys server.mjs, and npm-installs deps in the sandbox.

For prod parity, the same server.mjs/loop must be the production runtime — no eval-only reimplementation. That's the whole point.

Scaffold a new adapter: python scripts/scaffold_ai_sdk_agent.py <name> (prints a ready-to-edit package skeleton mirroring ai-sdk/acp).

2. Verify parity (inside == outside)

Behavioral parity = given the same model responses, the agent sends the same request and takes the same actions whether driven inside BenchFlow or standalone. Verify at two levels:

(a) Wire parity — drive the agent's ACP server against a capturing mock upstream, twice: standalone, and through BenchFlow's gateway. Diff the upstream requests.