Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add alivirgo/Major-AI-Skills --skill llm-cost-latency-benchmarkgit clone --depth 1 https://github.com/alivirgo/Major-AI-SkillsWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/alivirgo/major-ai-skills/llm-cost-latency-benchmark)<a href="https://agentmods.dev/skills/alivirgo/major-ai-skills/llm-cost-latency-benchmark"><img src="https://agentmods.dev/badge/skills/alivirgo/major-ai-skills/llm-cost-latency-benchmark/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/alivirgo/major-ai-skills/llm-cost-latency-benchmark"><img src="https://agentmods.dev/badge/skills/alivirgo/major-ai-skills/llm-cost-latency-benchmark.svg" alt="Reviewed on agentmods" width="80" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00063 | $0.04983 |
| Opus 5.5 | $0.00025 | $0.01993 |
| Sonnet 5.5 | $0.00013 | $0.00997 |
| Haiku 4.5 | $0.00006 | $0.00498 |
Grade A, and why
llm-cost-latency-benchmark scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 22d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 454 lines — stays where its author put it; the contents beside it link to each section on GitHub.
LLM Cost & Latency Benchmark AI Skill Guide (Claude)
Overview & Engine Architecture
Cost and latency are not one number. A claim is a tuple: workload × clock boundary × cache state × concurrency/arrival process × token mix × pricing snapshot × quality floor. Mixing those variables produces vendor-slide fiction. A response that is faster but fails acceptance is not an optimization win—it is a different product.
Claude operates as a Principal LLM Performance Economist, specializing in client-observed TTFT vs server TTFT, TPOT vs ITL, cache-cold vs cache-hot regimes, billable token categories (uncached input, cache read, cache write, output including reasoning), workflow wall-clock (retrieval + tools + retries + queue), percentile reporting, and SLO-qualified goodput.
┌─────────────────────────────────────────────────────────────┐
│ Cost × Latency Benchmark Stack │
│ │
│ Workload contract │
│ ├── Case population (prod-shaped) + quality floor │
│ ├── Input/output length distributions (or fixed I/O) │
│ ├── Concurrency + arrival process (closed vs open loop) │
│ └── Cache policy: cold | warm | production-mix │
│ │
│ Timing planes (never conflate) │
│ ├── Client TTFT: send → first nonempty streamed token │
│ ├── Prefill / queue / network (diagnostic breakdown) │
│ ├── TPOT / ITL after first token │
│ └── Workflow e2e: user intent → final accepted artifact │
│ │
│ Cost planes │
│ ├── Provider usage: input, cached_read, cache_write, out │
│ ├── Non-token: tools, search, containers, retries, fails │
│ └── Pricing manifest: URL + retrieved_at + model tier │
│ │
│ Decision │
│ ├── Percentiles + sample size + goodput@SLO │
│ └── Compare only configs that meet the quality floor │
└─────────────────────────────────────────────────────────────┘
What ships with it
2 files beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 22d ago Changed · +419 lines · +33 tokens per session 20cf9ae73527
- 1mo ago First seen · 35 lines · 30 tokens per session scan A bef50ab516f1
llm-cost-latency-benchmark is a skill published in the GitHub repository alivirgo/Major-AI-Skills (2 stars, last pushed today), licensed MIT. It adds 63 tokens to every session and 4,983 once invoked, about $0.0003 per session on Opus 5.5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-09-12.
Other skills, from other repositories
llm-engineering-expert
Build reliable applications on large language models: prompt design, structured output, evaluation, guardrails, and cost and latency control. Use when the user mentions LLMs, prompts, prompt engineering, few-shot examples, structured or JSON output, function calling, hallucination, model evaluation, token costs…
LLM Cost Optimizer
Audits an AI application for unnecessary token spend and recommends prompt caching, model routing, and token reduction techniques to cut costs.
ai-engineering-toolkit
6 production-ready AI engineering workflows: prompt evaluation (8-dimension scoring), context budget planning, RAG pipeline design, agent security audit (65-point checklist), eval harness building, and product sense coaching.
production-readiness
Use this skill to verify that an AI system is ready for production before deployment. Covers model abstraction, cost/latency, retries, fallbacks, structured logging, tracing, observability, evaluation in CI/CD, monitoring, and rollback planning. Activates as a mandatory gate before production deployment of any AI/LLM…
model-routing
Pick the right model per call, not per project — route Opus/Sonnet/Haiku, GPT-5/4o/mini, Gemini Pro/Flash by task, and cut cost without losing quality. Use when the user is choosing model tiers, building a router, or debating Opus-only vs mixed-tier deployments and mentions model selection, model router, cascade…
prompt-caching
Use prompt caching correctly across Anthropic, OpenAI, Bedrock, and Gemini to cut cost and latency on hot paths. Use when the user is building a production LLM app and mentions prompt caching, cache hits, cache key, cache TTL, ephemeral cache, system-prompt caching, or asks "why is my cache hit rate low?" / "should I…