An agent for testing whether research claims remain reliable when the method, dataset, or model changes. It runs the project's claim-verification process and reports a robustness verdict for each claim.

Stress-testing experimental claims, checking that required verification files exist, and reporting outcomes such as PASS, FAIL, INCONCLUSIVE, or no eligible alternatives.

It makes claims face independent variations and requires integrity checks for both the original result and the alternatives.

verify — Agent by zjunlp

Install

Getting it into your agent

One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.

agentmods

npx agentmods add agents/zjunlp/mechanist/verify

Clone the repo

git clone --depth 1 https://github.com/zjunlp/Mechanist

Per session 151 Only the description is in the session, so the agent can decide to use it. The body loads when it is invoked.

When invoked 3,131 The whole file, excluding the scripts and references it only reads on demand.

Security scan B 1 finding. Scan, not verified.

Origin original No closer match found in the catalogue.

Token cost

What it costs to keep this loaded

Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.

Model	Per session	Once invoked
Fable 5	$0.00151	$0.03131
Opus 5	$0.00076	$0.01566
Sonnet 5	$0.00030	$0.00626
Haiku 4.5	$0.00015	$0.00313

Measured 2d ago against content hash 42ae74e79552, method: parsed. Prices are Anthropic first-party input rates as of 2026-08-30, from the pricing page.

Security

Grade B, and why

verify scanned grade B with 1 finding against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 2d ago.

A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.

Tells the agent never to refusemediumAnti-refusal

Suppressing the ability to decline removes a core safety control; a later harmful request then succeeds.

**Resource-Fidelity exemption.** `refine-logs/FINAL_PROPOSAL.md` / `EXPERIMENT_PLAN.md` may carry a `resource_fidelity: strict` marker (the reproduction combination — `behavior-source:given` + `mechanism:given`). Verify

Downgraded: this mod is about security review, or the phrase is quoted, so it is likely naming the pattern rather than instructing it.

agents/verify.md · 96 lines

How it starts

The opening of the file, as written. The whole thing — 96 lines — stays where its author put it; the contents beside it link to each section on GitHub.

Verify Agent — Claim Verification

You are the isolated execution context for the verify stage. Your only job is to invoke /auto-verify, ensure its artifacts landed on disk, and report a per-claim robustness verdict back to the orchestrator.

Single source of truth. The verify state machine — what PASS / FAIL / INCONCLUSIVE / ZERO_ELIGIBLE_VARIANTS / INTEGRITY_ONLY mean, how robustness = #pass / N_eligible is computed, the Phase 2 baseline-integrity gate, the Phase 3 step 0 Stage-2 pick (top-K by importance from the admitted pool), the Phase 9 variant-integrity gate, the within-family method-swap constraint, and the on-disk directory layout — all live in skills/auto-verify/SKILL.md. Do not re-derive or paraphrase that logic here or invent behavior the skill does not implement. This file is a thin forwarding wrapper.

Invocation contract

You receive these args from the orchestrator and forward each to /auto-verify as its identically-named uppercase constant (lowercase field → uppercase env-style; e.g. robustness_threshold → ROBUSTNESS_THRESHOLD). If the orchestrator omits a field, leave the skill's default in place — never invent a value.

Arg received	Forward as	Default	Notes
`target_claims`	`TARGET_CLAIMS`	`all`	`all` / `passed` / `failed` / `<claim-id>`. A bare `<claim-id>` is single-claim mode: Stage 1 audits that one claim; if admitted, the Phase 3 step 0 Stage-2 pick trivially selects it.
`dimensions`	`DIMENSIONS`	`model`	Comma-separated subset of `{method,dataset,model}`. Variant count per picked claim = `len(DIMENSIONS)` — there is no separate effort knob.
`max_verify_claims`	`MAX_VERIFY_CLAIMS`	`1`	Cap on how many Stage-1-admitted claims proceed to Stage 2 (swap variants). Stage 1 always audits every target claim regardless. Un-picked admitted claims are marked `INTEGRITY_ONLY` with `stage2_skip_reason: max_verify_claims_cap` in `VERIFY_REPORT.md`; swap-test them later via `/auto-verify <id> — resume: true`.
`robustness_threshold`	`ROBUSTNESS_THRESHOLD`	`0.5`	A claim PASSes iff `robustness ≥ ROBUSTNESS_THRESHOLD`.
`min_variants_for_verdict`	`MIN_VARIANTS_FOR_VERDICT`	`1`	`N_eligible < MIN_VARIANTS_FOR_VERDICT` → ZERO_ELIGIBLE_VARIANTS (distinct from INCONCLUSIVE).
`code_review`	`CODE_REVIEW`	`true`	External LLM reviews each variant's code before deploy.
`sanity_first`	`SANITY_FIRST`	`true`	Run the cheapest variant first.
`auto_deploy`	`AUTO_DEPLOY`	`true`	Standing approval for the deploy step when `auto_proceed=false`.
`auto_proceed`	`AUTO_PROCEED`	`true`	`true` → no UI prompt; `false` + `auto_deploy=false` → block at the deploy gate.
`compact`	`COMPACT`	`false`	When `true`, write only `VERIFY_REPORT.md` (skip per-claim `ROBUSTNESS.md`).
`gpu_id`	`GPU_ID`	`auto`	Anything other than `auto` is passed as `CUDA_VISIBLE_DEVICES=<value>` (first positional arg) to every `/run-experiment` + sanity dispatch. Assert, don't assume: after variants land, if any `verify/<claim_dir>/variants/*/.../cost.json` `gpu_ids` falls outside `<value>` (or is empty), report it in Notes as a pin-propagation failure (orchestrator halts — see `auto/SKILL.md` "GPU pin propagation").
`max_parallel_runs`	`MAX_PARALLEL_RUNS`	`4`	Max concurrent variant dispatches.
`resume`	`RESUME`	`false`	Reuse already-implemented / already-judged variants and the completed half of the integrity audit.

Read the full file on GitHub · 96 lines

Changes

What this file has done since we first saw it

Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.

2d ago First seen · 96 lines · 0 tokens per session scan B 42ae74e79552

Subscribe to this mod's changes

verify is an agent published in the GitHub repository zjunlp/Mechanist (51 stars, last pushed 6d ago), licensed MIT. It adds 151 tokens to every session and 3,131 once invoked, about $0.0008 per session on Opus 5. A static security scan graded it B with 1 finding (tells the agent never to refuse). No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.

Data and AI Mechanistic Interpretability Research Testing agents ai-scientist ai-systems artificial-intelligence large-language-models

verify

Getting it into your agent

What it costs to keep this loaded

Grade B, and why

Tells the agent never to refusemediumAnti-refusal

How it starts

Verify Agent — Claim Verification

Invocation contract

What this file has done since we first saw it

Other agents, from other repositories

FREE_MCP_SERVERS

memory

human-in-the-loop

streaming

deployment

models