Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add raja21068/AutoResearch --skill paper-autoratersgit clone --depth 1 https://github.com/raja21068/AutoResearchWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/raja21068/autoresearch/paper-autoraters)<a href="https://agentmods.dev/skills/raja21068/autoresearch/paper-autoraters"><img src="https://agentmods.dev/badge/skills/raja21068/autoresearch/paper-autoraters/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/raja21068/autoresearch/paper-autoraters"><img src="https://agentmods.dev/badge/skills/raja21068/autoresearch/paper-autoraters.svg" alt="Reviewed on agentmods" width="80" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00121 | $0.01773 |
| Opus 5 | $0.00060 | $0.00886 |
| Sonnet 5 | $0.00024 | $0.00355 |
| Haiku 4.5 | $0.00012 | $0.00177 |
Grade A, and why
paper-autoraters scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 8d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
This is a copy
100% identical to paper-autoraters — 0 lines differ, which has more behind it and is treated as the original. This page carries a canonical link to it rather than competing with it.
How it starts
The opening of the file, as written. The whole thing — 154 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Paper Autoraters (App. F.3)
Faithful implementation of the four LLM-as-judge autoraters used in PaperOrchestra (Song et al., 2026, arXiv:2604.05018, §5 and App. F.3).
These are the metrics the paper uses to demonstrate that PaperOrchestra beats single-agent and AI-Scientist-v2 baselines. Use them to:
- Score a generated paper against a ground-truth paper.
- Compare two paper-writing pipelines side-by-side.
- Validate your own host-agent execution of the paper-orchestra pipeline.
The four autoraters
| Autorater | What it does | Inputs | Output |
|---|---|---|---|
| Citation F1 — P0/P1 partition | Partitions reference list into P0 (must-cite) and P1 (good-to-cite) given the paper text | one paper text + its references list | JSON {ref_num: "P0"|"P1"} |
| Literature Review Quality | 6-axis 0-100 score for Intro+Related Work, with anti-inflation hard caps | one paper PDF/text + reference avg citation count | JSON with axis_scores, penalties, summary, overall_score |
| SxS Overall Paper Quality | Holistic side-by-side preference judgment | two papers (PDF or text) | JSON with winner ∈ {paper_1, paper_2, tie} |
| SxS Literature Review Quality | Side-by-side preference, Intro+Related Work only | two papers | JSON with winner ∈ {paper_1, paper_2, tie} |
The paper uses Gemini-3.1-Pro and GPT-5 as judges, set to temperature 0.0 (Gemini) or default 1.0 (GPT-5, which doesn't allow temperature adjustment). Use whatever your host LLM is.
Workflow
Citation F1 (compute Precision / Recall / F1 vs ground truth)
This is a two-step procedure:
Step 1: Partition the reference lists into P0 / P1
For both the ground-truth paper AND the generated paper, run the LLM with
references/citation-f1-prompt.md:
inputs:
paper_text: full paper LaTeX or markdown
references_str: numbered reference list (e.g., "1. Vaswani et al. (2017)
Attention Is All You Need. NeurIPS. 2. He et al. (2016)
Deep Residual Learning for Image Recognition. CVPR. ...")
output: JSON {"1": "P0", "2": "P1", "3": "P0", ...}
What ships with it
5 files beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 8d ago First seen · 154 lines · 121 tokens per session scan A 6f0f5604aa96
paper-autoraters is a skill published in the GitHub repository raja21068/AutoResearch (2 stars, last pushed 3mo ago), licensed MIT. It adds 121 tokens to every session and 1,773 once invoked, about $0.0006 per session on Opus 5. A static security scan graded it A with 0 findings. It is 100% identical to paper-autoraters, differing in 0 lines, and is treated as a copy.
Other skills, from other repositories
literature-review-agent
Step 3 of the PaperOrchestra pipeline (arXiv:2604.05018). Execute the literature search strategy from outline.json — discover candidate papers via web search, verify them through Semantic Scholar (Levenshtein > 70 fuzzy title match, temporal cutoff, dedup by paperId), cross-corroborate against Crossref + OpenAlex to…
content-refinement-agent
Step 5 of the PaperOrchestra pipeline (arXiv:2604.05018). Iteratively refine drafts/paper.tex by simulating peer review and applying targeted revisions, with strict accept/revert halt rules, deterministic 0-100 decision bands (Accept/Minor/Major/Reject) that drive a target-met early stop, and a Devil's Advocate…
agent-research-aggregator
Pre-pipeline aggregator that scans AI agent cache directories (.claude, .cursor, .antigravity, .openclaw) or any user-specified directory for experimentation logs, extracts insights and numeric results, and formats them as PaperOrchestra-ready inputs (idea.md + experimentallog.md). TRIGGER when the user says…
paper-orchestra
Orchestrate the full PaperOrchestra (Song et al., 2026, arXiv:2604.05018) five-agent pipeline to turn unstructured research materials (idea, experimental log, LaTeX template, conference guidelines, optional figures) into a submission-ready LaTeX manuscript and compiled PDF. TRIGGER when the user asks to "write a paper…
section-writing-agent
Step 4 of the PaperOrchestra pipeline (arXiv:2604.05018). ONE single multimodal LLM call that drafts the remaining paper sections (Abstract, Methodology, Experiments, Conclusion), extracts numeric values from experimentallog.md into LaTeX booktabs tables, splices the generated figures from Step 2, and merges…
outline-agent
Step 1 of the PaperOrchestra pipeline (arXiv:2604.05018). Convert (idea.md, experimentallog.md, template.tex, conferenceguidelines.md) into a strict JSON outline containing a plotting plan, literature search plan (Intro + Related Work), and section-level writing plan with citation hints. TRIGGER when the orchestrator…