What is atlas-rag-evaluation?

A guide for measuring the quality of a RAG search bundle, meaning the prepared data and search components an AI uses to retrieve supporting text. It calculates Precision@10 and mean reciprocal rank (MRR), which show how many top results are relevant and how high the first relevant result appears.

Use it to select a bundle, run the evaluation queries, read the scores, and compare locally built bundles with those produced by continuous integration (CI).

It helps detect search-quality regressions and compare bundles or embedding backends with repeatable measurements instead of judging a few results by eye.

How sagar-shirwalkar/servicenow-atlas configures Claude Code and Codex

Reuse

Borrowing it

Nothing to install: this file belongs to sagar-shirwalkar/servicenow-atlas. Take a copy, put it at the same path in your own repository, and replace the rules that are about this project with yours.

Copy the file

curl -O https://raw.githubusercontent.com/sagar-shirwalkar/servicenow-atlas/main/.agents/skills/atlas-rag-evaluation/SKILL.md

Clone the repo

git clone --depth 1 https://github.com/sagar-shirwalkar/servicenow-atlas

Made for: Claude Code, Codex.

Wrote this? Show the measurements

A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.

README.md

[![agentmods](https://agentmods.dev/badge/skills/sagar-shirwalkar/servicenow-atlas/atlas-rag-evaluation/github.svg)](https://agentmods.dev/skills/sagar-shirwalkar/servicenow-atlas/atlas-rag-evaluation)

Your own site

<a href="https://agentmods.dev/skills/sagar-shirwalkar/servicenow-atlas/atlas-rag-evaluation"><img src="https://agentmods.dev/badge/skills/sagar-shirwalkar/servicenow-atlas/atlas-rag-evaluation/github.svg" alt="Measured on agentmods" height="20"></a>

Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.

Your own site · 80×15

<a href="https://agentmods.dev/skills/sagar-shirwalkar/servicenow-atlas/atlas-rag-evaluation"><img src="https://agentmods.dev/badge/skills/sagar-shirwalkar/servicenow-atlas/atlas-rag-evaluation.svg" alt="Reviewed on agentmods" width="80" height="20"></a>

Per session 53 Skills are progressive disclosure: only the name and description are preloaded; the body loads when the skill is used.

When invoked 792 The whole file, excluding the scripts and references it only reads on demand.

Security scan A 0 findings. A grade says what 26 rules found in the file — not that it is safe.

Origin original No closer match found in the catalogue.

Token cost

What it costs to keep this loaded

Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.

Model	Per session	Once invoked
Fable 5.1	$0.00053	$0.00792
Opus 5	$0.00026	$0.00396
Sonnet 5	$0.00011	$0.00158
Haiku 4.5	$0.00005	$0.00079

Measured 12d ago against content hash 420147276654, method: parsed. Prices are Anthropic first-party input rates as of 2026-09-12, from the pricing page.

Security

Grade A, and why

atlas-rag-evaluation scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 12d ago.

A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.

Nothing flagged

None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.

.agents/skills/atlas-rag-evaluation/SKILL.md · 98 lines

How it starts

The opening of the file, as written. The whole thing — 98 lines — stays where its author put it; the contents beside it link to each section on GitHub.

When to use

User asks to "evaluate RAG quality", "run the evaluation script", "check search quality"
User is comparing MLX vs ONNX+CPU embedding quality
User wants to verify that a CI-built bundle meets the quality bar
User suspects a regression in search relevance after a change

Workflow

1. Select a bundle

List available bundles:

ls data/bundles/

Pick the one you want to evaluate.

2. Run the evaluation

# Point BUNDLE_DIR at the bundle and run
BUNDLE_DIR=data/bundles/<timestamp> uv run atlas-eval

If the eval script is already registered as a console script. Otherwise run directly:

BUNDLE_DIR=data/bundles/<timestamp> uv run python atlas/evaluate_rag.py

3. Read the results

The output looks like:

Query: "how to create an incident"
  P@10: 0.90    MRR: 1.0000
...
Mean P@10 over 10 queries: 0.89
Mean MRR over 10 queries:  0.94

Precision@10 (P@10): Fraction of the top-10 results that are relevant. A score of 0.89 means ~9 of 10 returned chunks are on-topic.

Mean Reciprocal Rank (MRR): How high the first relevant result appears. 1.0 means the top result is always relevant; 0.94 means the first relevant result is typically in position 1–2.

4. Benchmark bar

Metric	Minimum	Target	Excellent
Precision@10	0.75	0.85	0.95+
MRR	0.80	0.90	0.97+

A score below Minimum means the bundle is not usable — investigate chunk boundaries, model, or embeddings.
Scores between Minimum and Target may be acceptable for development but should not ship to CI.
Target or higher is required for a release build.

5. Compare CI vs local

To compare, build two bundles with the same model on CI and locally:

# Local (MLX)
uv run atlas-build --model Xenova/bge-small-en-v1.5
BUNDLE_DIR=data/bundles/<local> uv run atlas-eval

# Download CI bundle from GitHub Releases
gh release download <tag> --pattern '*.tar.zst'
tar --zstd -xf atlas-bundle-*.tar.zst -C /tmp/ci-bundle
BUNDLE_DIR=/tmp/ci-bundle uv run atlas-eval

Read the full file on GitHub · 98 lines

Changes

What this file has done since we first saw it

Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.

12d ago First seen · 98 lines · 53 tokens per session scan A 420147276654

Subscribe to this mod's changes

atlas-rag-evaluation is a skill published in the GitHub repository sagar-shirwalkar/servicenow-atlas (2 stars, last pushed 2mo ago), licensed Apache-2.0. It adds 53 tokens to every session and 792 once invoked, about $0.0003 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.

Analytics and metrics Data and AI RAG and vectors ai ai-agents mcp-server retrieval-augmented-generation servicenow servicenow-development

graph-retrieval

Exposes graph-based retrieval as a tool capability via querygraph. Reads normalized graph store files, builds a query-relevant subgraph, and returns LLM-friendly semantic triples with replayable evidence metadata.

study8677/repobrain · 46 tokens

rag-perf

Performance benchmarking for a deployed NVIDIA RAG Blueprint server: profiling pass + aiperf load test driven by a single YAML config. Not for accuracy / RAGAS scoring (use rag-eval) or for deploying / repairing services (use rag-blueprint).

NVIDIA-AI-Blueprints/rag · 56 tokens

rag-blueprint

NVIDIA RAG Blueprint — deploy, configure, troubleshoot, and manage. Handles any RAG action: deploy, install, start, enable, disable, toggle, change, configure, troubleshoot, debug, fix, shutdown, stop, or tear down any RAG feature or service (Agentic RAG, VLM, guardrails, query rewriting, models, search, ingestion…

NVIDIA-AI-Blueprints/rag · 92 tokens

rag-eval

Filesystem RAG benchmarks: corpus/, train.json, evaluaterag.py (RAGAS quality). Not for prod monitoring, latency/throughput benchmarking (use rag-perf), or evals outside this repo layout.

NVIDIA-AI-Blueprints/rag · 48 tokens

knowledge-layer

High-level deployment wrapper over RepoBrain core with graph-first knowledge injection and all-file support. Exposes refreshfilesystem and askfilesystem for building and querying the knowledge graph.

study8677/repobrain · 42 tokens

rag-evaluate-quality

Periodically measure the retrieval quality of the knowledge base using evaluateretrieval (MRR@5, Recall@5, Precision@5) plus getindexstats for health metrics. Run weekly, after significant reindex activity, or when the user reports declining answer quality. Prevents silent index rot and grounds "should we tune X"…

lyonzin/knowledge-rag · 78 tokens

servicenow-atlas: Skill for Claude Code