Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add agentscope-ai/OpenJudge --skill 01-graders-and-pipelinegit clone --depth 1 https://github.com/agentscope-ai/OpenJudgeWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/agentscope-ai/openjudge/01-graders-and-pipeline)<a href="https://agentmods.dev/skills/agentscope-ai/openjudge/01-graders-and-pipeline"><img src="https://agentmods.dev/badge/skills/agentscope-ai/openjudge/01-graders-and-pipeline/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/agentscope-ai/openjudge/01-graders-and-pipeline"><img src="https://agentmods.dev/badge/skills/agentscope-ai/openjudge/01-graders-and-pipeline.svg" alt="Reviewed on agentmods" width="80" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00105 | $0.01289 |
| Opus 5.5 | $0.00042 | $0.00516 |
| Sonnet 5.5 | $0.00021 | $0.00258 |
| Haiku 4.5 | $0.00011 | $0.00129 |
Grade A, and why
01-graders-and-pipeline scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 1mo ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 160 lines — stays where its author put it; the contents beside it link to each section on GitHub.
OpenJudge Skill
Build evaluation pipelines for LLM applications using the openjudge library.
When to Use This Skill
- User wants to evaluate LLM output quality (correctness, relevance, hallucination, etc.)
- User wants to compare two or more models and rank them
- User wants to design a scoring rubric and automate evaluation
- User wants to analyze evaluation results statistically
- User wants to build a reward model or quality filter
Sub-documents — Read When Relevant
| Topic | File | Read when… |
|---|---|---|
| Grader selection & configuration | graders.md |
User needs to pick or configure an evaluator |
| Batch evaluation pipeline | pipeline.md |
User needs to run evaluation over a dataset |
| Auto-generate graders from data | generator.md |
No rubric yet; generate from labeled examples |
| Analyze & compare results | analyzer.md |
User wants win rates, statistics, or metrics |
Read the relevant sub-document before writing any code.
Install
pip install py-openjudge
Architecture Overview
Dataset (List[dict])
│
▼
GradingRunner ← orchestrates everything
│
├─► Grader A ──► EvaluationStrategy ──► _aevaluate() ──► GraderScore / GraderRank
├─► Grader B ──► EvaluationStrategy ──► _aevaluate() ──► GraderScore / GraderRank
└─► Grader C ...
│
├─► Aggregator (optional) ← combine multiple grader scores into one
│
└─► RunnerResult ← {grader_name: [GraderScore, ...]}
│
▼
Analyzer ← statistics, win rates, validation metrics
5-Minute Quick Start
Evaluate responses for correctness using a built-in grader:
import asyncio
from openjudge.models.openai_chat_model import OpenAIChatModel
from openjudge.graders.common.correctness import CorrectnessGrader
from openjudge.runner.grading_runner import GradingRunner
# 1. Configure the judge model (OpenAI-compatible endpoint)
model = OpenAIChatModel(
model="qwen-plus",
api_key="sk-xxx",
base_url="https://dashscope.aliyuncs.com/compatible-mode/v1",
)
# 2. Instantiate a grader
grader = CorrectnessGrader(model=model)
# 3. Prepare dataset
dataset = [
{
"query": "What is the capital of France?",
"response": "Paris is the capital of France.",
"reference_response": "Paris.",
},
{
"query": "What is 2 + 2?",
"response": "The answer is five.",
"reference_response": "4.",
},
]
# 4. Run evaluation
async def main():
runner = GradingRunner(
grader_configs={"correctness": grader},
max_concurrency=8,
)
results = await runner.arun(dataset)
for i, result in enumerate(results["correctness"]):
print(f"[{i}] score={result.score} reason={result.reason}")
asyncio.run(main())
What ships with it
4 files beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 1mo ago First seen · 160 lines · 105 tokens per session scan A f413a0fac029
01-graders-and-pipeline is a skill published in the GitHub repository agentscope-ai/OpenJudge (888 stars, last pushed 5d ago), licensed Apache-2.0. It adds 105 tokens to every session and 1,289 once invoked, about $0.0004 per session on Opus 5.5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-09-12.
Other skills, from other repositories
tinker-fine-tuning
Provides guidance for fine-tuning LLMs using the Tinker cloud training API from Thinking Machines Lab. Use when running supervised fine-tuning, reinforcement learning (GRPO/PPO), or LoRA training on cloud GPUs via Tinker's managed infrastructure instead of local compute.
llm-as-judge-evaluation
Evaluate LLM outputs using frontier models as judges. Use for pairwise model comparison, quality scoring with custom rubrics, and automated evaluation pipelines. Covers position bias mitigation, statistical significance, and generating preference data for DPO/RLHF.
hugging-face-evaluation
Add and manage evaluation results in Hugging Face model cards. Supports extracting eval tables from README content, importing scores from Artificial Analysis API, and running custom model evaluations with vLLM/lighteval. Works with the model-index metadata format.
evaluating-code-models
Evaluates code generation models across HumanEval, MBPP, MultiPL-E, and 15+ benchmarks with pass@k metrics. Use when benchmarking code models, comparing coding abilities, testing multi-language support, or measuring code generation quality. Industry standard from BigCode Project used by HuggingFace leaderboards.
grpo-rl-training
Expert guidance for GRPO/RL fine-tuning with TRL for reasoning and task-specific model training.
model-evaluation
Systematic ML model evaluation — metrics, benchmarks, and error analysis.