A command-line workflow for running Twinkle Eval, a tool that tests an AI model against question sets and reports scores. The model must already be available through an OpenAI-compatible API endpoint.

Use it to create and validate evaluation settings, download or select built-in datasets, run or resume a benchmark, and inspect results.

It provides the configuration and checks needed to run an evaluation without confusing a failed model connection with a scoring problem. It can also help diagnose answers that the result parser cannot read.

How ai-twinkle/Eval configures Claude Code

Reuse

Borrowing it

Nothing to install: this file belongs to ai-twinkle/Eval. Take a copy, put it at the same path in your own repository, and replace the rules that are about this project with yours.

Copy the file

curl -O https://raw.githubusercontent.com/ai-twinkle/Eval/main/.claude/skills/run-eval/SKILL.md

Clone the repo

git clone --depth 1 https://github.com/ai-twinkle/Eval

Made for: Claude Code.

Wrote this? Show the measurements

A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.

README.md

[![agentmods](https://agentmods.dev/badge/skills/ai-twinkle/eval/run-eval/github.svg)](https://agentmods.dev/skills/ai-twinkle/eval/run-eval)

Your own site

<a href="https://agentmods.dev/skills/ai-twinkle/eval/run-eval"><img src="https://agentmods.dev/badge/skills/ai-twinkle/eval/run-eval/github.svg" alt="Measured on agentmods" height="20"></a>

Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.

Your own site · 80×15

<a href="https://agentmods.dev/skills/ai-twinkle/eval/run-eval"><img src="https://agentmods.dev/badge/skills/ai-twinkle/eval/run-eval.svg" alt="Reviewed on agentmods" width="80" height="20"></a>

Per session 129 Skills are progressive disclosure: only the name and description are preloaded; the body loads when the skill is used.

When invoked 2,952 The whole file, excluding the scripts and references it only reads on demand.

Security scan A 0 findings. A grade says what 26 rules found in the file — not that it is safe.

Origin original No closer match found in the catalogue.

Token cost

What it costs to keep this loaded

Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.

Model	Per session	Once invoked
Fable 5.1	$0.00129	$0.02952
Opus 5.5	$0.00052	$0.01181
Sonnet 5.5	$0.00026	$0.00590
Haiku 4.5	$0.00013	$0.00295

Measured 1mo ago against content hash 7fdd652f72a2, method: parsed. Prices are Anthropic first-party input rates as of 2026-10-07, from the pricing page.

Security

Grade A, and why

run-eval scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 1mo ago.

A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.

Nothing flagged

None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.

.claude/skills/run-eval/SKILL.md · 214 lines

How it starts

The opening of the file, as written. The whole thing — 214 lines — stays where its author put it; the contents beside it link to each section on GitHub.

跑一次評測

前提：本專案不啟動模型服務

Twinkle Eval 只做一件事——拿題目去呼叫已經在外部運行的 OpenAI 相容端點。模型要自己先起好（vLLM、Ollama、OpenAI、NVIDIA Build 都行），再把 base_url 填進 config。

端點沒回應時，本專案的正確行為是依 max_retries 重試後報錯退出，不會也不該嘗試重啟服務（CLAUDE.md 原則 G）。

1. 產生 config

twinkle-eval --init                  # 列出全部 11 個範本
twinkle-eval --init multiple_choice  # 產生 configs/multiple_choice.yaml
twinkle-eval --init all              # 全部產生到 configs/

範本存放在 twinkle_eval/templates/，--init 直接掃該目錄。

config 骨架

llm_api:
  base_url: "http://localhost:8000/v1"   # 必填
  api_key: "EMPTY"                       # 必填（本地 vLLM 隨便填）
  api_rate_limit: -1                     # QPS，-1 為不限
  max_retries: 3
  timeout: 600
  disable_ssl_verify: false

model:
  name: "my-model"                       # 必填，會寫進結果路徑與紀錄
  temperature: 0.0
  top_p: 0.9
  max_tokens: 4096
  extra_body:                            # 傳給 API 的額外參數

evaluation:
  dataset_paths:                         # 必填，list（即使只有一個）
    - "datasets/example/tmmluplus/"
  evaluation_method: "box"               # 必填，見下表
  repeat_runs: 1                         # >1 時算平均與標準差
  shuffle_options: false                 # 選項隨機排列

logging:
  level: "INFO"

每個 evaluation_method 的必填欄位與 strategy_config 參數不同，完整對照見 references/config-reference.md。

挑 evaluation_method

方法	用在	備註
`pattern`	選擇題，通用首選	正則比對，含中英文預設模式
`box`	選擇題，推理模型	提取 `\boxed{}` / `\box{}`；需要 `system_prompt`
`logit`	多選項題	比較各選項 log-probability，不依賴輸出格式；選項數不限
`math`	數學推理	`\boxed{}` + MathRuler；需 `[math]`
`regex_match`	BBH 之類自由格式	⚠️ `system_prompt` 不生效，見下方說明
`custom_regex`	自訂格式	必須設 `strategy_config.patterns`
`ifeval` / `ifbench`	指令遵循	需 `[ifeval]` / `[ifbench]` + nltk 資料
`bfcl_fc` / `bfcl_prompt`	函式呼叫	FC 走 tools API，prompt 走注入
`niah`	長文本大海撈針
`ragas`	RAG 品質
`text2sql`	Text-to-SQL	需設 `text2sql_db_base_path`
`asr`	語音辨識	需 `[asr]`；`llm_api.type: whisper` 或多模態
`vision_mcq`	視覺多選題	需 `[vision]`（縮放用）

Read the full file on GitHub · 214 lines

Files

What ships with it

1 file beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.

references/config-reference.md 9.2 KB

Changes

What this file has done since we first saw it

Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.

1mo ago First seen · 214 lines · 129 tokens per session scan A 7fdd652f72a2

Subscribe to this mod's changes

run-eval is a skill published in the GitHub repository ai-twinkle/Eval (117 stars, last pushed 1mo ago), licensed MIT. It adds 129 tokens to every session and 2,952 once invoked, about $0.0005 per session on Opus 5.5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-09-12.

Analytics and metrics Data and AI Evaluation LLM eval

run-eval next to 6 skills from other repositories
Mod	Type	Stars	Last push	Licence	Scan	Tokens	Works with
run-evalai-twinkle/Eval	Skill	117	1mo ago	MIT	A	129	Claude Code
llm-evaluationmartineserios/thebrana	Skill	3	today	MIT	A	31	Claude Code
eval-designerNickCrew/Claude-Cortex	Skill	66	4mo ago	MIT	D	62	Claude Code, Codex
ai-engineering-toolkitsickn33/agentic-awesome-skills	Skill	49k	yesterday	MIT	A	47	Claude Code
ai-prompt-regression-testingsickn33/agentic-awesome-skills	Skill	49k	yesterday	MIT	A	37	Claude Code
prompt-optimizationlangwatch/langwatch	Skill	5.0k	yesterday	Apache-2.0	A	105	Claude Code, Codex
experimentslangwatch/langwatch	Skill	5.0k	yesterday	Apache-2.0	A	58	Claude Code, Codex

Eval: Skill for Claude Code