llm-eval-harness

llm-eval-harness is a plugin for Claude Code from daymade/claude-code-skills. Its token cost is not measured, scanned A, original, MIT.

A test and evaluation tool for language models accessed through OpenAI- or Anthropic-compatible APIs. It checks whether requests work correctly and measures model behavior and performance.

In plain words
What is it for?
Use it to evaluate or benchmark model endpoints, onboard providers, verify speed claims, test failover options, and investigate models that ignore instructions.
Why use it?
It helps reveal unavailable models, dropped prompts or tools, protocol problems, slow responses, concurrency limits, and quality regressions.

Plugin for Claude Code

Written for Claude Code: a Claude Code plugin manifest.

Good fit Use it to evaluate or benchmark model endpoints, onboard providers, verify speed claims, test failover options, and investigate models that ignore instructions.

Compare 6 plugins from other repositories ↓
Install in Claude Code
/plugin marketplace add daymade/claude-code-skills/plugin install llm-eval-harness
About the project

Claude Code Skills Marketplace is a collection and marketplace of skills, plugins, agents, and instructions that extend Claude Code with specialized development workflows. It is for developers who want to install existing workflows or create, validate, and package their own Claude Code skills.

daymade/claude-code-skills · 1,381 stars · on GitHub

Install

Getting it into your agent

One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.

Clone the repo
git clone --depth 1 https://github.com/daymade/claude-code-skills

Made for: Claude Code.

Wrote this? Show the measurements

A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.

agentmods badge for llm-eval-harness

README.md
[![agentmods](https://agentmods.dev/badge/plugins/daymade/claude-code-skills/llm-eval-harness.svg)](https://agentmods.dev/plugins/daymade/claude-code-skills/llm-eval-harness)
Your own site
<a href="https://agentmods.dev/plugins/daymade/claude-code-skills/llm-eval-harness"><img src="https://agentmods.dev/badge/plugins/daymade/claude-code-skills/llm-eval-harness.svg" alt="Measured on agentmods" height="20"></a>
Per session not measured What this adds to a session before it is invoked.
When invoked not measured Not applicable: nothing here is loaded into a session.
Security scan A 1 finding. A grade says what 26 rules found in the file — not that it is safe.
Origin original No closer match found in the catalogue.
Security

Grade A, and why

llm-eval-harness scanned grade A with 1 finding against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 4d ago.

A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.

Makes network callslowCapability

Not a fault in itself. Listed so you know the mod talks to something, and to what.

"description": "Test/evaluate any LLM behind an OpenAI- or Anthropic-compatible endpoint: availability (max_tokens-aware), request fidelity (does system prompt/tools/history REACH the model, or does the gateway silently
.claude-plugin/marketplace.json#llm-eval-harness · 25 lines

What it actually says

{
  "name": "llm-eval-harness",
  "description": "Test/evaluate any LLM behind an OpenAI- or Anthropic-compatible endpoint: availability (max_tokens-aware), request fidelity (does system prompt/tools/history REACH the model, or does the gateway silently drop it), speed (TTFT+tok/s), concurrency (before a workshop), Anthropic protocol compliance, quality regression, vendor bug reports, deployment gates, resident canaries. Reach for this BEFORE hand-rolling a curl loop against any LLM endpoint — that instinct is the trap: it skips the N=10 sampling, Connection:close, and env-var key handling this bakes in. Use when someone tests/benchmarks/测评/压测 a model/endpoint, onboards a provider, decides whether to switch or temporarily fail over to an alternate channel (e.g. outage or quota exhaustion), writes a supported-models list, debugs \"model ignores system prompt\", or verifies a tok/s claim. Triggers on \"benchmark this model\", \"测一下这个模型/渠道/API\", \"接入新模型先测一下\", \"system prompt 不生效\", \"这个渠道能不能用/稳不稳\", \"临时切换过去顶一阵子\" — even without \"eval\", even wrapped in business narrative.",
  "source": "./llm-eval-harness",
  "strict": false,
  "version": "1.4.0",
  "category": "developer-tools",
  "keywords": [
    "llm",
    "eval",
    "benchmark",
    "model-comparison",
    "speed-test",
    "concurrency",
    "protocol-compliance",
    "tokens-per-second",
    "ttft",
    "thinking-aware",
    "availability",
    "fidelity",
    "system-prompt",
    "gateway-testing",
    "vendor-bug-report"
  ]
}
Files

What ships with it

1 file beside marketplace.json#llm-eval-harness in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.

Changes

What this file has done since we first saw it

Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.

  1. 4d ago First seen · 25 lines scan A da095b119fb7

Subscribe to this mod's changes

llm-eval-harness is a plugin published in the GitHub repository daymade/claude-code-skills (1,381 stars, last pushed today), licensed MIT. Its token cost is not measured: this kind of file is read by the harness, not the model. A static security scan graded it A with 1 finding (makes network calls). No closer match exists in the catalogue, so it is treated as the original; first seen 2026-09-03.

Related

Other plugins, from other repositories

evals

LLM evaluation methodology and eval-suite design, distilled from Anthropic's official evaluation guidance — a knowledge router over success criteria, eval design, and grading methods (/evals:methodology), plus an action skill that interviews for measurable success criteria and scaffolds a graded eval suite for an LLM…

melodic-software/claude-code-plugins · not measured

rust-intel

Defense against LLM Rust failure modes: 59 categories of bugs that survive rustc, clippy, and cargo test — async cancellation, unsafe/FFI, concurrency, crypto, supply-chain, tests-that-pass-by-luck, semantic conformance, and systemic performance cost. Ships the rust-intel skill plus /rust-intel:audit, /rust-intel:fix.

PHPCraftdream/rust-intel · not measured

evals-skills

Skills for building LLM evaluations: pipeline audit, error analysis, synthetic data generation, LLM-as-Judge design, evaluator validation, RAG evaluation, and annotation interfaces.

hamelsmu/evals-skills · not measured

nextjs

Official Next.js skills: adopt and optimize Cache Components, adopt Partial Prefetching, and verify runtime behavior against a running dev server.

vercel/next.js · not measured

themes-example

themes-example. An open-source AI agent that brings the power of Gemini directly into your terminal.

google-gemini/gemini-cli · not measured

claude-plugins-official marketplace

Directory of popular Claude Code extensions including development tools, productivity plugins, and MCP integrations.

anthropics/claude-plugins-official · not measured