ToolUniverse is a collection of tools, interfaces, and supporting components for building AI systems that perform scientific work. It is for developers creating AI scientist agents that use APIs, databases, machine-learning tools, and domain-specific utilities. The catalogue includes skills, commands, an MCP server, an agent, and a hook for working with the ecosystem.
Getting it into your agent
It runs from inside its repository, so the clone comes first — what it calls does not travel with the file alone.
git clone --depth 1 https://github.com/mims-harvard/ToolUniversenpx agentmods add skills/mims-harvard/tooluniverse/devtu-benchmark-harnessWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/mims-harvard/tooluniverse/devtu-benchmark-harness)<a href="https://agentmods.dev/skills/mims-harvard/tooluniverse/devtu-benchmark-harness"><img src="https://agentmods.dev/badge/skills/mims-harvard/tooluniverse/devtu-benchmark-harness/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/mims-harvard/tooluniverse/devtu-benchmark-harness"><img src="https://agentmods.dev/badge/skills/mims-harvard/tooluniverse/devtu-benchmark-harness.svg" alt="Reviewed on agentmods" width="80" height="20"></a>- NVIDIA SkillSpector warn
SkillSpector: 3 findings, up to high
These are SkillSpector’s own severities. On a checked sample its high-severity flags on skills were ~96% false positives — a documented command, a public API, a “never do X” rule — so we show them as a caution to read, not a verdict. Why →
- high Rogue Agent · line 131 Skill modifies its own code, configuration, or behavior at runtime. Self-modification enables an agent to escalate privileges, disable safety constraints, or install persistent backdoors.Fix: Prevent the skill from modifying its own code, SKILL.md, or configuration files. Treat skill files as read-only at runtime.
- high Rogue Agent · line 134 Skill modifies its own code, configuration, or behavior at runtime. Self-modification enables an agent to escalate privileges, disable safety constraints, or install persistent backdoors.Fix: Prevent the skill from modifying its own code, SKILL.md, or configuration files. Treat skill files as read-only at runtime.
- high Rogue Agent · line 204 Skill modifies its own code, configuration, or behavior at runtime. Self-modification enables an agent to escalate privileges, disable safety constraints, or install persistent backdoors.Fix: Prevent the skill from modifying its own code, SKILL.md, or configuration files. Treat skill files as read-only at runtime.
What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00049 | $0.02574 |
| Opus 5.5 | $0.00020 | $0.01030 |
| Sonnet 5.5 | $0.00010 | $0.00515 |
| Haiku 4.5 | $0.00005 | $0.00257 |
Grade A, and why
devtu-benchmark-harness scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 1mo ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 233 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Benchmark Harness — Continuous Improvement System
A 5-step feedback loop for improving ToolUniverse tools, skills, and plugin quality.
Note: This skill is dataset-agnostic. Per-benchmark score history, known-failing question IDs, and dataset-specific investigations belong in temp_docs_and_tests/benchmark_tracking/ (gitignored workfolder), NOT in this skill directory.
The Feedback Loop
1. RUN benchmark → 2. ANALYZE results → 3. DIAGNOSE failures → 4. FIX via devtu skill → 5. RETEST → repeat
Orchestrated runner (preferred)
One command does steps 0 (memorization audit), 1 (build), 2 (run), 3 (analyze), 4 (diagnose + extract failures):
bash skills/devtu-benchmark-harness/scripts/run_harness_loop.sh --benchmark bixbench --n 20 --seed 42
# After reviewing diagnose.log and applying devtu skill fixes:
bash skills/devtu-benchmark-harness/scripts/run_harness_loop.sh --retest /path/to/failures.json
The script creates temp_docs_and_tests/benchmark_tracking/run_<TS>/ with results.json, analysis.log, diagnose.log, failures.json. Diagnose output lists each failure with the exact devtu skill to invoke — do NOT fix manually.
Anti-memorization guard
Before accepting any skill edit (from devtu-optimize-skills or manual), run:
python3 skills/devtu-benchmark-harness/scripts/check_memorization.py --all
Fails if any skill contains benchmark names, capsule UUIDs, bix-N question IDs, or known-to-be-GT specific numeric answers. This prevents overfitting the plugin to a single benchmark's answer key. Run in --strict mode to also flag specific gene names and dataset filenames (softer signal).
Step 1: RUN — Execute Benchmark
bash scripts/build-plugin.sh # rebuild plugin with latest skills
python skills/devtu-benchmark-harness/scripts/run_eval.py \
--benchmark bixbench \ # bixbench | lab-bench | custom
--mode plugin-only \ # plugin-only | baseline-only | comparison
--n 205 \ # number of questions
--timeout 1800 \ # seconds per question
--max-turns 30 # agent turns per question
What ships with it
14 files beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.
- evals/evals.json 702 B
- references/benchmark-guide.md 3.7 KB
- scripts/analyze_failures.py 7.4 KB runs code
- scripts/analyze_results.py 18 KB runs code
- scripts/check_memorization.py 6.0 KB runs code
- scripts/compute_capsule_checksums.py 1.5 KB runs code
- scripts/generate_report.py 3.6 KB runs code
- scripts/grade_answers.py 14 KB runs code
- scripts/run_eval.py 48 KB runs code
- scripts/run_harness_loop.sh 4.6 KB runs code
- scripts/run_one_by_one.sh 2.8 KB runs code
- scripts/skill_routing_test_v2.py 8.2 KB runs code
- scripts/skill_routing_test.py 5.3 KB runs code
- scripts/solve_one.sh 3.6 KB runs code
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 1mo ago First seen · 233 lines · 49 tokens per session scan A 822334816c8f
devtu-benchmark-harness is a skill published in the GitHub repository mims-harvard/ToolUniverse (1,739 stars, last pushed today), licensed Apache-2.0. It adds 49 tokens to every session and 2,574 once invoked, about $0.0002 per session on Opus 5.5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-09-03.
Other skills, from other repositories
build-and-test
How to build and test .NET projects in the Agent Framework repository. Use this when verifying or testing changes.
typescript-testing
Select and run TypeScript SDK verification for packages, examples, type checks, linting, builds, Vitest suites, Effect v4 CLI tests, and runtime E2E tests. Use when adding tests, diagnosing TypeScript CI, choosing a focused test command, validating TypeScript package changes, or writing/porting CLI tests against…
prompt-proximity-architecture
Turn an approved measurement charter, ICPs, and buyer jobs into a budget-aware prompt coverage blueprint across proximity bands, aided status, information acts, journey states, roles, locales, evidence grades, partitions, and measurement lanes. Use before prompt wording to define required, optional, and prohibited…
architecture-patterns
Architecture validation and patterns for clean architecture, backend structure enforcement, project structure validation, test standards, and context-aware sizing. Use when designing system boundaries, enforcing layered architecture, validating project structure, defining test standards, or choosing the right…
unit-test-json-serialization
Provides patterns for unit testing JSON serialization/deserialization with Jackson and @JsonTest. Validates JSON mapping, custom serializers, date formats, and polymorphic types. Use when testing JSON serialization, validating custom serializers, or writing JSON unit tests in Spring Boot applications.
unit-test-service-layer
Provides patterns for unit testing service layer with Mockito. Creates isolated tests that mock repository calls, verify method invocations, test exception scenarios, and stub external API responses. Use when testing service behaviors and business logic without database or external services.