Borrowing it
Nothing to install: this file belongs to sema-lisp/sema. Take a copy, put it at the same path in your own repository, and replace the rules that are about this project with yours.
curl -O https://raw.githubusercontent.com/sema-lisp/sema/main/.claude/commands/llm-verify.mdgit clone --depth 1 https://github.com/sema-lisp/semaWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/commands/sema-lisp/sema/llm-verify)<a href="https://agentmods.dev/commands/sema-lisp/sema/llm-verify"><img src="https://agentmods.dev/badge/commands/sema-lisp/sema/llm-verify/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/commands/sema-lisp/sema/llm-verify"><img src="https://agentmods.dev/badge/commands/sema-lisp/sema/llm-verify.svg" alt="Reviewed on agentmods" width="80" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00000 | $0.00988 |
| Opus 5.5 | $0.00000 | $0.00395 |
| Sonnet 5.5 | $0.00000 | $0.00198 |
| Haiku 4.5 | $0.00000 | $0.00099 |
Grade A, and why
llm-verify scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 2mo ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 60 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Verify LLM / Agentic Features
Verify a change to Sema's LLM/agent layer (sema-llm) using the two-tier flow
that's caught real bugs (see CHANGELOG 1.21.x). $ARGUMENTS names the feature(s)
to focus on (e.g. "tool loop", "fallback", "caching", "reasoning-effort"); if
empty, cover the agent loop, fallback, cache, and budget.
Tier 1 — Deterministic, keyless (required, runs in CI)
Use sema_llm::fake::FakeProvider (scripted replies / tool calls / errors /
streamed chunks) installed via register_test_provider; assert on
FakeRecorder requests. Tests live in crates/sema/tests/llm_fake_test.rs.
Hooks: set_retry_base_ms(0) (no sleeps), set_network_max_retries.
cargo test -p sema-lang --test llm_fake_test
Add a FakeProvider test for any change to the agent loop, retry, cache, budget,
or a provider serializer — this is the regression oracle. Prefer asserting on
behavior the model can't fake: round-2 tool-result correlation, retry attempt
counts, cache hit ⇒ recorder.call_count() unchanged and zero added usage.
Tier 2 — Live integration (when feasible; keys are in the env)
Keys present: ANTHROPIC_API_KEY, OPENAI_API_KEY, GEMINI_API_KEY,
MISTRAL_API_KEY. Cheap models for testing (don't hammer gpt-5.5):
| Provider | Model | configure |
|---|---|---|
| OpenAI | gpt-5.4-mini (dots, not the dashed snapshot form) |
(llm/configure :openai {:api-key (env "OPENAI_API_KEY") :default-model "gpt-5.4-mini"}) |
| Anthropic | claude-haiku-4-5-20251001 |
(llm/configure :anthropic {:api-key (env "ANTHROPIC_API_KEY") :default-model "...") |
| Gemini | gemini-2.5-flash |
(llm/configure :gemini {:api-key (env "GEMINI_API_KEY") :default-model "gemini-2.5-flash"}) |
| Mistral | mistral-small-latest |
(llm/configure :mistral {:api-key (env "MISTRAL_API_KEY") :default-model "mistral-small-latest"}) |
Write a /tmp/*.sema script, build (cargo build), run ./target/debug/sema /tmp/x.sema.
Verify across all three major families for serializer-level changes (tool
loop, reasoning-effort) — quirks differ per provider.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 2mo ago First seen · 60 lines · 0 tokens per session scan A 2698da7c207e
llm-verify is a command published in the GitHub repository sema-lisp/sema (60 stars, last pushed 9d ago), licensed MIT. It costs nothing until one of its globs matches a file; then it loads 988 tokens. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.
llm-verify compared with commands from other repositories
| Mod | Type | Stars | Last push | Licence | Scan | Tokens | Works with |
|---|---|---|---|---|---|---|---|
| llm-verifysema-lisp/sema | Command | 60 | 9d ago | MIT | A | 0 | Claude Code |
| rust-buildaffaan-m/ECC | Command | 298k | 4d ago | MIT | A | 8 | Claude Code, OpenCode |
| fix-issuescan1357/oh-my-pi | Command | 38k | yesterday | MIT | A | 0 | Claude Code |
| fix-issueyonatangross/orchestkit | Command | 299 | yesterday | MIT | A | 45 | Claude Code, Cursor |
| devSmart-AI-Memory/attune-ai | Command | 11 | yesterday | Apache-2.0 | A | 16 | Claude Code |
| gsd:audit-fixmrboups/xbrain | Command | 2 | 1mo ago | MIT | A | 25 | Claude Code |
| gsd:forensicsswisstruthorg/swiss-truth-mcp | Command | 0 | 6mo ago | none | A | 30 | Claude Code |