AgentCheck AGENTS.md

AGENTS.md — working on AgentCheck

Instructions for AI coding agents and new contributors working in this repository. Read this before changing anything under agentcheck/.

What this project is

AgentCheck is an evidence authority for AI-agent behavior, not a framework compatibility layer. It imports a trusted local agent, derives adversarial scenarios from what it finds, and runs them in same-user child processes where declared tool calls are simulated. It emits PASS / FAIL / INCONCLUSIVE / INFRA_ERROR, an HTML report, and a replay manifest.

The product's value is that its verdicts can be trusted. Every rule below exists to protect that.

Keep four evidence dimensions separate: contained, observed, semantically understood, and policy-verifiable. One never implies the next. A blocked and observed runtime action with unknown semantics remains INCONCLUSIVE, never a PASS. Static discovery is lower-bounded, not exhaustive.

Non-negotiable invariants

Breaking any of these is a correctness bug, not a style issue:

1. The original tool handler never executes during a simulated evaluation. Interception replaces the invoker before the handler is reached.
2. Unknown tools fail closed. Never synthesize a tool result.
3. No real mutations. Only the simulated world changes.
4. Worker isolation. Scenarios run in child processes; the environment allowlist is empty by default. This is trusted-code process isolation, not a hostile-code sandbox.
5. Network denied by default, and containment failures surface. The current Python guard is not a kernel security boundary and must not be described as one.
6. INCONCLUSIVE and INFRA_ERROR never collapse into PASS.
7. Redaction at the artifact and log boundary, before writing or printing.
8. Never overstate replay. It reproduces inputs and harness behavior, not model determinism.

Layout

Path	Role
agentcheck/domain/	Contract models: scenarios, specs, runs, verdicts, findings. Fingerprinted and versioned.
agentcheck/adapters/	The only layer allowed to import a framework SDK.
agentcheck/inspect/	Import a target and extract an AgentSpec without running a turn.
agentcheck/generate/	Derive, lint, select, and freeze suites.
agentcheck/coverage/	Derived behavioral coverage over a spec and a scenario set. Pure contract analysis; imports no target.
agentcheck/runner/	Orchestrator, worker, tool gateway, simulated world, budgets, network guard.
agentcheck/evaluate/	Oracle evaluation and verdict assignment.
agentcheck/replay/	Manifests, source binding, filesets.
agentcheck/report/, agentcheck/baseline/, agentcheck/review/	Reporting, CI gating, human decisions on findings.
agentcheck/regression/	Run-to-run behavioral comparison over stored artifacts. Executes nothing.
agentcheck/gate.py	The CI decision: runs the suite, compares the baseline, and answers whether a change blocks the build. Orchestrates the above; decides no verdicts of its own.
agentcheck/identity.py	Portable target identity and its bounded legacy compatibility path.
agentcheck/redaction.py, agentcheck/privacy.py	Credential redaction for artifacts and logs.
agentcheck/cli.py	The agentcheck command.
spikes/environment_containment/	Repository-only, provider-neutral research contracts. Control metadata stays outside its minimal target leaves and never ships in distributions.