A configuration-based benchmarking tool for deployed NVIDIA RAG Blueprint servers. It measures retrieval-stage timing, citation quality, bottlenecks, and optionally load-test results such as response time and throughput.

Use it for one-time performance benchmarks, server profiling, and optional load tests. Throughput measures how much work a system completes over time, while latency measures how long one request takes.

It helps identify slow parts of a RAG system and verify whether performance changes improve latency, capacity, or error rates.

rag-perf — Skill for Codex by PracticalSwan/agent-skills

Install

Getting it into your agent

One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.

Any agent

npx skills add PracticalSwan/agent-skills --skill rag-perf

Clone the repo

git clone --depth 1 https://github.com/PracticalSwan/agent-skills

Made for: Codex.

Wrote this? Show the measurements

A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.

README.md

[![agentmods](https://agentmods.dev/badge/skills/practicalswan/agent-skills/rag-perf/github.svg)](https://agentmods.dev/skills/practicalswan/agent-skills/rag-perf)

Your own site

<a href="https://agentmods.dev/skills/practicalswan/agent-skills/rag-perf"><img src="https://agentmods.dev/badge/skills/practicalswan/agent-skills/rag-perf/github.svg" alt="Measured on agentmods" height="20"></a>

Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.

Your own site · 80×15

<a href="https://agentmods.dev/skills/practicalswan/agent-skills/rag-perf"><img src="https://agentmods.dev/badge/skills/practicalswan/agent-skills/rag-perf.svg" alt="Reviewed on agentmods" width="80" height="20"></a>

Per session 30 Skills are progressive disclosure: only the name and description are preloaded; the body loads when the skill is used.

When invoked 4,315 The whole file, excluding the scripts and references it only reads on demand.

Security scan A 1 finding. A grade says what 26 rules found in the file — not that it is safe.

Origin 88% copy Near-identical to another mod in the catalogue.

Token cost

What it costs to keep this loaded

Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.

Model	Per session	Once invoked
Fable 5.1	$0.00030	$0.04315
Opus 5.5	$0.00012	$0.01726
Sonnet 5.5	$0.00006	$0.00863
Haiku 4.5	$0.00003	$0.00432

Measured 1mo ago against content hash be0329f84ecf, method: parsed. Prices are Anthropic first-party input rates as of 2026-10-07, from the pricing page.

Security

Grade A, and why

rag-perf scanned grade A with 1 finding against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 1mo ago.

A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.

Makes network callslowCapability

Not a fault in itself. Listed so you know the mod talks to something, and to what.

| `Citation count (mean): 0` and `Citation relevance score: N/A` for a non-empty deployment | Collection mismatch between `rag.collection_names` and what's actually ingested | Run `curl -s http://<ingestor>:8082/v1/colle

Origin

This is a copy

88% identical to rag-perf — 84 lines differ, which has more behind it and is treated as the original. This page carries a canonical link to it rather than competing with it.

rag-perf/SKILL.md · 202 lines

How it starts

The opening of the file, as written. The whole thing — 202 lines — stays where its author put it; the contents beside it link to each section on GitHub.

RAG-Perf — config-driven perf benchmark CLI

Purpose

Drive a deployed NVIDIA RAG Blueprint server with a YAML config, run a server-side profiling pass (per-stage timing, citation quality, bottleneck inference) and an optional aiperf load test (TTFT / E2E / token & request throughput / error rate), and write a unified report. The CLI is intentionally minimal: rag-perf -c <config> plus --help / --version. Behaviour is fully config-driven; field variations belong in YAML.

Scope

Accuracy / RAGAS scoring of answer quality → use the rag-eval skill.
Deploying, repairing, or configuring services (compose, helm, NIM env vars) → use the rag-blueprint skill.
Production monitoring / alerting — rag-perf is a one-shot benchmark tool.
Runtime requirement: a deployed RAG server reachable on the network.

Prerequisites

Repo cloned; run commands from the repo root (config paths in the presets are repo-root-relative).
Python 3.11+ and uv on PATH.
Install rag-perf into its own uv-managed venv: uv sync --project scripts/rag-perf.
For unit tests: install dev extras as well — uv sync --project scripts/rag-perf --extra dev (otherwise pytest-asyncio is missing and async tests error out at collection time).
A reachable RAG server (default http://localhost:8081). For the aiperf phase, the bundled nvidia_rag endpoint plugin must be installed — pip install -e ./scripts/rag-perf registers it via the aiperf.plugins entry point.
For synthetic queries: an OpenAI-compatible chat-completions endpoint reachable at synthetic.llm_url (default http://localhost:8999/v1/chat/completions).
rag-perf itself runs without NVIDIA_API_KEY (unlike rag-eval). The synthetic LLM endpoint may require its own auth — that's the deployment's concern.

Instructions

Pick a preset. The three under scripts/rag-perf/configs/ are:
- quick_profile.yaml — profile-only, ~30 s. Skips load test. For fast iteration on retrieval / reranker tuning.
- single_run.yaml — one concurrency level, profiling + aiperf, ~2 min. Regression checks.
- sweep.yaml — multi-axis sweep. load.concurrency, rag.vdb_top_k, rag.reranker_top_k are all int | list[int]; any of them as a list becomes a sweep axis (Cartesian product).

Read the full file on GitHub · 202 lines

Files

What ships with it

9 files beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.

Changes

What this file has done since we first saw it

Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.

1mo ago Changed · -1 lines be0329f84ecf
1mo ago Changed 78f310f8b26e
1mo ago First seen · 203 lines · 30 tokens per session scan A 4ddde37e583b

Subscribe to this mod's changes

rag-perf is a skill published in the GitHub repository PracticalSwan/agent-skills (21 stars, last pushed 4d ago), licensed MIT. It adds 30 tokens to every session and 4,315 once invoked, about $0.0001 per session on Opus 5.5. A static security scan graded it A with 1 finding (makes network calls). It is 88% identical to rag-perf, differing in 84 lines, and is treated as a copy.

Analytics and metrics Data and AI NVIDIA RAG and vectors latency performance profiling throughput

Performance Optimization

Full-stack performance analysis, optimization patterns, and monitoring strategies.

ArieGoldkin/ai-agent-hub · 14 tokens

evalscope

LLM evaluation & inference performance testing via the evalscope CLI. Translates natural language requests into evalscope commands for: (1) Model accuracy evaluation — runs registered benchmarks against local checkpoints or API endpoints (OpenAI-compatible, Anthropic, LiteLLM); (2) Performance stress testing — TTFT…

modelscope/evalscope · 195 tokens

rag-perf