What is vitest-evals AGENTS.md?

Repository instructions for vitest-evals, a toolkit for testing and evaluating AI agent runs. They describe its pnpm commands, package layout, file-specific checks, and development conventions.

Use them when editing vitest-evals packages or apps, running focused linting, formatting, tests, type checks, builds, or evaluation commands.

They make it easier to run the right check for a changed file and understand where the project’s adapters, reports, tests, and demos belong.

How getsentry/vitest-evals configures Codex and OpenCode

Reuse

Borrowing it

Nothing to install: this file belongs to getsentry/vitest-evals. Take a copy, put it at the same path in your own repository, and replace the rules that are about this project with yours.

Copy the file

curl -O https://raw.githubusercontent.com/getsentry/vitest-evals/main/AGENTS.md

Clone the repo

git clone --depth 1 https://github.com/getsentry/vitest-evals

Made for: Codex, OpenCode.

Wrote this? Show the measurements

A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.

README.md

[![agentmods](https://agentmods.dev/badge/instructions/getsentry/vitest-evals/agents-md/github.svg)](https://agentmods.dev/instructions/getsentry/vitest-evals/agents-md)

Your own site

<a href="https://agentmods.dev/instructions/getsentry/vitest-evals/agents-md"><img src="https://agentmods.dev/badge/instructions/getsentry/vitest-evals/agents-md/github.svg" alt="Measured on agentmods" height="20"></a>

Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.

Your own site · 80×15

<a href="https://agentmods.dev/instructions/getsentry/vitest-evals/agents-md"><img src="https://agentmods.dev/badge/instructions/getsentry/vitest-evals/agents-md.svg" alt="Reviewed on agentmods" width="80" height="20"></a>

Per session 632 This file is loaded in full into every session.

When invoked 632 The same file — it is already loaded in full.

Security scan A 0 findings. A grade says what 26 rules found in the file — not that it is safe.

Origin original No closer match found in the catalogue.

Token cost

What it costs to keep this loaded

Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.

Model	Per session	Once invoked
Fable 5.1	$0.00632	$0.00632
Opus 5.5	$0.00253	$0.00253
Sonnet 5.5	$0.00126	$0.00126
Haiku 4.5	$0.00063	$0.00063

Measured 1mo ago against content hash fb41ca05f1b3, method: parsed. Prices are Anthropic first-party input rates as of 2026-10-07, from the pricing page.

Security

Grade A, and why

vitest-evals AGENTS.md scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 1mo ago.

A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.

Nothing flagged

None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.

AGENTS.md · 46 lines

How it starts

The opening of the file, as written. The whole thing — 46 lines — stays where its author put it; the contents beside it link to each section on GitHub.

Agent Instructions

Package Manager

Use pnpm: pnpm install, pnpm test, pnpm typecheck, pnpm build, pnpm evals

Repository Structure

packages/
  core/                   # normalized session/run types, transcript events, report metadata
  vitest-evals/           # root describeEval, judges, matchers, reporter, legacy; re-exports core
  harness-ai-sdk/         # AI SDK adapter into HarnessRun
  harness-openai-agents/  # OpenAI Agents adapter into HarnessRun
  harness-pi-ai/          # pi-ai adapter, wrapped tool runtime, tool replay
  github-reporter/        # GitHub Action reporter: collection, summary, annotations
  report-ui/              # HTML report UI and its transcript/trace model
  docs/                   # docs site, generated API reference, llms.txt
apps/                     # demo-ai-sdk, demo-openai-agents, demo-pi (realistic demo fixtures)

File-Scoped Commands

Task	Command
Lint file	`pnpm exec biome lint path/to/file.ts`
Format file	`pnpm exec biome format --write path/to/file.ts`
Test file	`pnpm exec vitest run path/to/file.test.ts -c vitest.config.ts`
Eval file	`pnpm exec vitest run path/to/file.eval.ts -c vitest.config.ts --reporter=./packages/vitest-evals/src/reporter.ts`

Key Conventions

Root API work is harness-first and judge-first.
Legacy scorer-first changes stay under packages/vitest-evals/src/legacy/....
Keep normalized session and run data JSON-serializable.
Keep UsageSummary to stable usage units such as tokens, USD cost, tools, retries, provider, and model. Set costUsd only when cost is known; provider-specific pricing details belong in usage.metadata.
GitHub reporting is action-first: document workflows with uses: getsentry/vitest-evals@v0, not CLI commands.
Sharded eval reporting uses distinct JSON artifacts per matrix job and one reducer job that publishes the combined action report.
Root GitHub Action releases keep source and bundled tags separate: Craft publishes the source release, vX.Y.Z-src preserves the source baseline, and vX/vX.Y.Z point at the bundled action commit. Keep the Craft github target filtered away from package artifacts.
Update README.md, packages/vitest-evals/README.md, and relevant docs when product shape changes.
Follow policies/code-comments.md for exported-function JSDoc and non-obvious comments only.

Read the full file on GitHub · 46 lines

Changes

What this file has done since we first saw it

Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.

1mo ago Changed · +6 tokens per session fb41ca05f1b3
2mo ago First seen · 46 lines · 626 tokens per session scan A c823b5ec8dee

Subscribe to this mod's changes

vitest-evals AGENTS.md is an instructions file published in the GitHub repository getsentry/vitest-evals (358 stars, last pushed 1mo ago), licensed Apache-2.0. It adds 632 tokens to every session, about $0.0025 per session on Opus 5.5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.

tag-non-production

XcodeBuildMCP-iOS-Template copilot-instructions.md

Instructions for getsentry/XcodeBuildMCP-iOS-Template, a project described as: A scaffold template for iOS app projects.

getsentry/XcodeBuildMCP-iOS-Template · 344 tokens

vscode buildNext.instructions.md

Working notes and architecture documentation for the new esbuild-based build system in build/next. Use when making changes to the new build pipeline (transpile/bundle commands, NLS plugin, source-map handling, resource copying, or self-hosting watch tasks).

microsoft/vscode · 9,715 tokens

vitest-evals: Instructions file for Codex