What is llm-evaluation-system CLAUDE.md?

A guide to the llm-evaluation-system repository for coding agents. It explains the repository's packages, web application, important files, and common development commands.

Use it when changing the MCP package, the optional FastAPI and React web app, the local results viewer, or their deployment and build setup.

It gives an agent the context needed to work in an unfamiliar codebase. It reduces guesswork about where code lives and how the different parts are built and deployed.

How awslabs/llm-evaluation-system configures Claude Code

About the project

LLM Evaluation System is an agent-guided platform for evaluating language models and agents, generating datasets and configuring multiple judges from natural-language requests before producing an analysis report. It is for comparing model responses, testing agents, and creating document-grounded evaluation data.

awslabs/llm-evaluation-system · 26 stars · on GitHub

Reuse

Borrowing it

Nothing to install: this file belongs to awslabs/llm-evaluation-system. Take a copy, put it at the same path in your own repository, and replace the rules that are about this project with yours.

Copy the file

curl -O https://raw.githubusercontent.com/awslabs/llm-evaluation-system/main/CLAUDE.md

Clone the repo

git clone --depth 1 https://github.com/awslabs/llm-evaluation-system

Made for: Claude Code.

Wrote this? Show the measurements

A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.

README.md

[![agentmods](https://agentmods.dev/badge/instructions/awslabs/llm-evaluation-system/claude-md/github.svg)](https://agentmods.dev/instructions/awslabs/llm-evaluation-system/claude-md)

Your own site

<a href="https://agentmods.dev/instructions/awslabs/llm-evaluation-system/claude-md"><img src="https://agentmods.dev/badge/instructions/awslabs/llm-evaluation-system/claude-md/github.svg" alt="Measured on agentmods" height="20"></a>

Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.

Your own site · 80×15

<a href="https://agentmods.dev/instructions/awslabs/llm-evaluation-system/claude-md"><img src="https://agentmods.dev/badge/instructions/awslabs/llm-evaluation-system/claude-md.svg" alt="Reviewed on agentmods" width="80" height="20"></a>

Per session 7,449 This file is loaded in full into every session.

When invoked 7,449 The same file — it is already loaded in full.

Security scan A 1 finding. A grade says what 26 rules found in the file — not that it is safe.

Origin original No closer match found in the catalogue.

Token cost

What it costs to keep this loaded

Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.

Model	Per session	Once invoked
Fable 5.1	$0.07449	$0.07449
Opus 5.5	$0.02980	$0.02980
Sonnet 5.5	$0.01490	$0.01490
Haiku 4.5	$0.00745	$0.00745

Measured 2mo ago against content hash aa2bc8a37df5, method: parsed. Prices are Anthropic first-party input rates as of 2026-10-07, from the pricing page.

Security

Grade A, and why

llm-evaluation-system CLAUDE.md scanned grade A with 1 finding against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 2mo ago.

A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.

Makes network callslowCapability

Not a fault in itself. Listed so you know the mod talks to something, and to what.

**Testing the web app for real** — pytest greens don't prove the web app works; exercise it against the live `make dev` stack in layers: (1) `pytest` for pure logic; (2) `curl`/`urllib` against `:4001` for `/api/*` route

CLAUDE.md · 403 lines

How it starts

The opening of the file, as written. The whole thing — 403 lines — stays where its author put it; the contents beside it link to each section on GitHub.

CLAUDE.md

This is the canonical agent-facing doc for the repo — Claude Code reads it directly, and AGENTS.md is a thin pointer back here so other tools that follow the agents.md convention (Codex, Cursor) land in the same place.

What's in this repo

Two deployables that share some code:

eval_mcp/ — the MCP package published to PyPI as llm-evaluation-system (entry point eval-mcp). Self-contained, no database, no web app. This is what 99% of users install.
backend/ + frontend/ — the optional EKS web app (FastAPI chat + Vite/React UI + Cognito auth). ./deploy.sh is its entry point; make dev runs it locally via Docker Compose.

frontend/ is a single Vite + React SPA (client-side routing via react-router). vite build produces a static bundle served two ways: bundled into eval_mcp/viewer_static/ for the MCP's local results viewer (npm run build:viewer), and served from a private S3 bucket via CloudFront OAC for the EKS web deployment (no frontend pod — see ARCHITECTURE.md). Locally, nginx serves the bundle and proxies the gated paths (/api, /inspect) to the backend, mirroring that CloudFront/S3 split. Changing frontend code therefore affects the PyPI wheel — the viewer static is package data per pyproject.toml.

Key files

File	Purpose
`eval_mcp/server.py`	Unified MCP server — every tool is registered here
`eval_mcp/tools/`	Tool handlers (QA gen, judge, config, run, …)
`eval_mcp/core/bedrock_client.py`	Bedrock client + cross-region inference + API-key auth
`eval_mcp/core/judge_config.py`	Default judge models and criteria
`eval_mcp/core/pricing.py`	Live model pricing from LiteLLM (24h cache → vendored snapshot fallback); no hand-maintained price table
`eval_mcp/core/litellm_pricing_snapshot.json`	Vendored offline fallback for pricing; refresh with `make sync-pricing`
`backend/core/agent.py`	EKS web app's agent system prompt + loop (the MCP itself doesn't host an agent)
`Makefile`	Local dev commands (`make dev`, `make logs`, `make restart`, `make stop`, `make release`)

Read the full file on GitHub · 403 lines

Changes

What this file has done since we first saw it

Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.

2mo ago First seen · 403 lines · 7,449 tokens per session scan A aa2bc8a37df5

Subscribe to this mod's changes

llm-evaluation-system CLAUDE.md is an instructions file published in the GitHub repository awslabs/llm-evaluation-system (26 stars, last pushed yesterday), licensed Apache-2.0. It adds 7,449 tokens to every session, about $0.0298 per session on Opus 5.5. A static security scan graded it A with 1 finding (makes network calls). No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.

llm-evaluation-system CLAUDE.md next to 6 instructions from other repositories
Mod	Type	Stars	Last push	Licence	Scan	Tokens	Works with
llm-evaluation-system CLAUDE.mdawslabs/llm-evaluation-system	Instructions file	26	yesterday	Apache-2.0	A	7,449	Claude Code
vscode buildNext.instructions.mdmicrosoft/vscode	Instructions file	195k	yesterday	MIT	A	9,719	GitHub Copilot
next.js AGENTS.mdvercel/next.js	Instructions file	143k	yesterday	MIT	A	7,461	Codex, OpenCode
langchain AGENTS.mdlangchain-ai/langchain	Instructions file	149k	yesterday	MIT	A	4,680	Codex, OpenCode
vscode oss-third-party-notices.instructions.mdmicrosoft/vscode	Instructions file	195k	yesterday	MIT	A	5,516	GitHub Copilot
vscode copilot-instructions.mdmicrosoft/vscode	Instructions file	195k	yesterday	MIT	A	3,093	GitHub Copilot
deepseek-harness AGENTS.mddeepseek-ai/deepseek-harness	Instructions file	274k	2d ago	MIT	A	4,313	Codex, OpenCode

llm-evaluation-system: Instructions file for Claude Code