Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add vasilyu1983/AI-Agents-public --skill ai-llm-inferencegit clone --depth 1 https://github.com/vasilyu1983/AI-Agents-publicWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/vasilyu1983/ai-agents-public/ai-llm-inference)<a href="https://agentmods.dev/skills/vasilyu1983/ai-agents-public/ai-llm-inference"><img src="https://agentmods.dev/badge/skills/vasilyu1983/ai-agents-public/ai-llm-inference/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/vasilyu1983/ai-agents-public/ai-llm-inference"><img src="https://agentmods.dev/badge/skills/vasilyu1983/ai-agents-public/ai-llm-inference.svg" alt="Reviewed on agentmods" width="80" height="20"></a>- NVIDIA SkillSpector pass
What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00039 | $0.05127 |
| Opus 5 | $0.00019 | $0.02563 |
| Sonnet 5 | $0.00008 | $0.01025 |
| Haiku 4.5 | $0.00004 | $0.00513 |
Grade A, and why
ai-llm-inference scanned grade A with 1 finding against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 8d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Makes network callslowCapability
Not a fault in itself. Listed so you know the mod talks to something, and to what.
| `scripts/latency_benchmark.py` | Issue N concurrent requests against an OpenAI-compatible `/v1/chat/completions` endpoint and report p50/p95/p99 latency and throughput. stdlib-only (urllib + threading). | How it starts
The opening of the file, as written. The whole thing — 332 lines — stays where its author put it; the contents beside it link to each section on GitHub.
LLM Inference Production Skill Hub
Operational guidance for choosing and tuning modern inference stacks. Focus on runtime fit, routing, output guarantees, adapter loading, multimodal serving, and measured performance under load.
Use current primary sources for volatile facts such as versions, hardware support, benchmarks, pricing, and release status.
ASCII Flow
serving workload
|
v
intake
model + modality + context + QPS + latency SLO + hardware + output contract
|
v
serving design
engine + router + batching + cache + quantization + structured outputs
|
v
benchmark
TTFT + ITL + throughput + error rate + quality floor
|
v
production serving path
capacity plan + rollout + monitoring + rollback thresholds
When to Use This Skill
Use this skill when the user asks for:
- inference engine selection or stack comparison
- latency, TTFT, ITL, or throughput optimization
- cache-aware routing or control-plane design
- quantization strategy by runtime and hardware
- multi-GPU or multi-node serving
- structured outputs or constrained decoding at serve time
- multimodal or encoder-decoder serving patterns
- LoRA or multi-adapter serving
- cost reduction for self-hosted or API inference
- benchmarking, profiling, or capacity planning
- CPU or edge deployment with GGUF or llama.cpp, or on-device NPU (per-tensor scales, static shapes)
Scope Boundaries
- Prompting, fine-tuning, eval sets -> ai-llm
- RAG pipeline design -> ai-rag
- Deployment automation, monitoring, incident response -> ai-mlops
- Infrastructure operations -> ops-devops-platform
- Observability design -> qa-observability
Quick Reference
| Decision | Primary Question | Default Starting Point | Escalate When |
|---|---|---|---|
| Engine | Which runtime should execute tokens? | vLLM for general text serving | Need stronger KV reuse, multimodal split, or NVIDIA-specific kernels |
| Router | How should requests be placed? | Simple replica pool first | Multiple replicas, sticky prefixes, multi-LoRA, or mixed workloads |
| Output control | Must responses obey a schema? | Use native structured outputs | JSON validity or grammar constraints are part of the SLA |
| Multimodal | Is there a separate encoder path? | Keep colocated first | Encoder saturation differs from decode saturation |
| Adapters | Will many LoRAs or tenants share the base model? | Use native multi-LoRA support | Cold-load latency or adapter churn affects p95 |
| Quantization | Which precision is safe in this runtime? | Runtime-native FP8 or weight-only path | Hardware support or model quality is uncertain |
| Disaggregation | Should prefill/encoder/decode be split? | Only after colocated baseline | Queueing interference or resource asymmetry is proven |
What ships with it
41 files beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.
- agents/openai.yaml 371 B
- assets/batching/template-batching-config.md 616 B
- assets/benchmarking/template-latency-throughput-test.md 1.6 KB
- assets/caching/template-prefix-caching.md 679 B
- assets/checklists/inference-review-checklist.md 6.6 KB
- assets/inference/template-deepspeed-inference.md 1.1 KB
- assets/inference/template-tensorrtllm-config.md 819 B
- assets/inference/template-vllm-config.md 1.5 KB
- assets/quantization/template-awq.md 449 B
- assets/quantization/template-gguf.md 1.3 KB
- assets/quantization/template-gptq.md 538 B
- assets/serving/template-disaggregated-serving.md 953 B
- assets/serving/template-high-throughput-setup.md 1.1 KB
- assets/serving/template-llm-api.md 1.6 KB
- data/sources.json 29 KB
- learnings.consolidated.md 592 B
- learnings.md 573 B
- references/architecture-and-attention-serving.md 25 KB
- references/batching-and-scheduling.md 1.6 KB
- references/cost-optimization-patterns.md 2.6 KB
- references/disaggregated-inference.md 5.8 KB
- references/edge-cpu-optimization.md 15 KB
- references/gpu-node-and-cluster-tuning.md 20 KB
- references/gpu-optimization-checklists.md 2.8 KB
- references/infrastructure-tuning.md 6.2 KB
- references/kv-cache-optimization.md 20 KB
- references/moe-expert-parallelism.md 14 KB
- references/multi-model-routing.md 17 KB
- references/optimization-strategies.md 6.4 KB
- references/parallelism-patterns.md 14 KB
- references/profiling-and-capacity-planning.md 19 KB
- references/pruning-and-sparsity.md 28 KB
- references/quantization-patterns.md 17 KB
- references/queueing-theory-applied.md 41 KB
- references/reliability-theory-applied.md 51 KB
- references/resilience-ha-patterns.md 26 KB
- references/routing-and-control-planes.md 2.7 KB
- references/serving-architectures.md 4.7 KB
- references/speculative-decoding-guide.md 5.5 KB
- references/streaming-patterns.md 13 KB
- scripts/latency_benchmark.py 7.9 KB runs code
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 8d ago Changed · +13 lines fa64113a5dd2
- 12d ago First seen · 319 lines · 39 tokens per session scan A 0a3672bba6cb
ai-llm-inference is a skill published in the GitHub repository vasilyu1983/AI-Agents-public (87 stars, last pushed 9d ago), licensed MIT. It adds 39 tokens to every session and 5,127 once invoked, about $0.0002 per session on Opus 5. A static security scan graded it A with 1 finding (makes network calls). No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.
Other skills, from other repositories
spark-environment-setup
Set up a working ML training/inference environment on NVIDIA DGX Spark (GB10, aarch64, CUDA 13). Use when installing PyTorch/Unsloth/TRL/vLLM on DGX Spark, hitting libcudart or wheel-ABI errors on aarch64, or choosing between NGC containers and bare pip installs.
spark-memory-thermal-ops
Manage unified memory and thermals during long-running ML jobs on NVIDIA DGX Spark. Use when planning memory headroom for a training run on GB10, when a job OOMs on unified memory, or when monitoring temperature and power during multi-hour training.
spark-training-gotchas
Preflight and diagnose the ten known failure modes for ML training on NVIDIA DGX Spark. Use when a training run on DGX Spark fails to start, OOMs below the 128GB limit, slows down mid-run, or before any multi-hour training job on GB10.
data-quality-frameworks
Implement data quality validation with Great Expectations, dbt tests, and data contracts. Use when building data quality pipelines, implementing validation rules, or establishing data contracts.
airflow-dag-patterns
Build production Apache Airflow DAGs with best practices for operators, sensors, testing, and deployment. Use when creating data pipelines, orchestrating workflows, or scheduling batch jobs.
dbt-transformation-patterns
Master dbt (data build tool) for analytics engineering with model organization, testing, documentation, and incremental strategies. Use when building data transformations, creating data models, or implementing analytics engineering best practices.