reproduce-benchmark

Reproduce Benchmark

Run a tool's or paper's benchmark harness and record what could and could not be verified.

When to Use

• User asks to reproduce or verify benchmark claims for a triaged tool or paper
• Following up on a triage or analysis where self-reported metrics need independent verification
• User provides a slug, reference path, or analysis path and says "run the benchmark"

When Not to Use

• The subject has not yet been triaged — run triage-tool or triage-paper first
• A repro file already exists in benchmarks/sources/ — report it and offer to re-run instead
• The benchmark requires live external services or paid APIs that are unavailable — set outcome to unverified and document the blocker

Recommended MCP Servers

If the benchmark harness or reported figures need to be looked up from the original paper, prefer these MCPs over WebFetch:

{
  "mcpServers": {
    "semantic-scholar": {
      "type": "stdio",
      "command": "uvx",
      "args": ["semantic-scholar-fastmcp"]
    },
    "google-scholar": {
      "type": "stdio",
      "command": "uvx",
      "args": ["google_scholar_mcp_server"]
    }
  }
}

See triage-paper for full setup guidance.

Mindset

Reproduction is a verification exercise, not a sales pitch.

1. Record what you observe: quote actual output, not what the README says the output should be.
2. Mark everything reported: any figure from the README or paper must carry (as reported). Never blend reported and verified numbers without distinguishing them.
3. Partial is better than fabricated: if only some scenarios run, record those honestly. Set outcome to partially-verified.
4. Environment matters: cold-start overhead, fixture quality, and dependency versions all affect results. Document them.

Workflow

1. Identify the subject

• Confirm the slug from the user (e.g. context-mode, jiang-llmlingua).
• Locate the reference file (references/<slug>.md) and analysis file (analysis/ANALYSIS-<slug>.md) if they exist.