Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx agentmods add skills/jamkris/everything-gemini-code/agent-evalnpx skills add Jamkris/everything-gemini-code --skill agent-evalgit clone --depth 1 https://github.com/Jamkris/everything-gemini-codeWhat it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5 | $0.00038 | $0.01087 |
| Opus 5 | $0.00019 | $0.00544 |
| Sonnet 5 | $0.00008 | $0.00217 |
| Haiku 4.5 | $0.00004 | $0.00109 |
Grade A, and why
agent-eval scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 2d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
This is a copy
88% identical to agent-eval — 35 lines differ, which has more behind it and is treated as the original. This page carries a canonical link to it rather than competing with it.
How it starts
The opening of the file, as written. The whole thing — 146 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Agent Eval Skill
A lightweight CLI tool for comparing coding agents head-to-head on reproducible tasks. Every "which coding agent is best?" comparison runs on vibes — this tool systematizes it.
When to Use
- Comparing coding agents (Gemini CLI, Aider, Codex, etc.) on your own codebase
- Measuring agent performance before adopting a new tool or model
- Running regression checks when an agent updates its model or tooling
- Producing data-backed agent selection decisions for a team
Installation
Note: Install agent-eval from its repository after reviewing the source.
Core Concepts
YAML Task Definitions
Define tasks declaratively. Each task specifies what to do, which files to touch, and how to judge success:
name: add-retry-logic
description: Add exponential backoff retry to the HTTP client
repo: ./my-project
files:
- src/http_client.py
prompt: |
Add retry logic with exponential backoff to all HTTP requests.
Max 3 retries. Initial delay 1s, max delay 30s.
judge:
- type: pytest
command: pytest tests/test_http_client.py -v
- type: grep
pattern: "exponential_backoff|retry"
files: src/http_client.py
commit: "abc1234" # pin to specific commit for reproducibility
Git Worktree Isolation
Each agent run gets its own git worktree — no Docker required. This provides reproducibility isolation so agents cannot interfere with each other or corrupt the base repo.
Metrics Collected
| Metric | What It Measures |
|---|---|
| Pass rate | Did the agent produce code that passes the judge? |
| Cost | API spend per task (when available) |
| Time | Wall-clock seconds to completion |
| Consistency | Pass rate across repeated runs (e.g., 3/3 = 100%) |
Workflow
1. Define Tasks
Create a tasks/ directory with YAML files, one per task:
mkdir tasks
# Write task definitions (see template above)
2. Run Agents
Execute agents against your tasks:
agent-eval run --task tasks/add-retry-logic.yaml --agent gemini-cli --agent aider --runs 3
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 2d ago First seen · 146 lines · 38 tokens per session scan A f6fa5e7d1005
agent-eval is a skill published in the GitHub repository Jamkris/everything-gemini-code (87 stars, last pushed 3mo ago), licensed MIT. It adds 38 tokens to every session and 1,087 once invoked, about $0.0002 per session on Opus 5. A static security scan graded it A with 0 findings. It is 88% identical to agent-eval, differing in 35 lines, and is treated as a copy.
Other skills, from other repositories
looker
These skills are designed for data discovery and business intelligence.
looker-dev
These skills are built for LookML developers, data engineers, and administrators who manage the backbone of Looker.
cloud-sql-postgres-health
Use these skills when you need to audit database health, identify storage bloat, find invalid indexes, analyze table statistics, and manage maintenance configurations like autovacuum.
cloud-sql-postgres-vectorassist
Use these skills to set up and optimize production-ready vector workloads by simply expressing your intent and performance requirements.
cloud-sql-postgres-data
Use these skills when you need to explore the database structure, discover schema objects like views or stored procedures, and execute custom SQL queries to interact with your data.
cloud-sql-postgres-admin
Use these skills when you need to provision new Cloud SQL instances, create databases and users, clone existing environments, and monitor the progress of long-running operations.