Borrowing it
Nothing to install: this file belongs to sagar-shirwalkar/servicenow-atlas. Take a copy, put it at the same path in your own repository, and replace the rules that are about this project with yours.
curl -O https://raw.githubusercontent.com/sagar-shirwalkar/servicenow-atlas/main/.agents/skills/atlas-rag-evaluation/SKILL.mdgit clone --depth 1 https://github.com/sagar-shirwalkar/servicenow-atlasWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/sagar-shirwalkar/servicenow-atlas/atlas-rag-evaluation)<a href="https://agentmods.dev/skills/sagar-shirwalkar/servicenow-atlas/atlas-rag-evaluation"><img src="https://agentmods.dev/badge/skills/sagar-shirwalkar/servicenow-atlas/atlas-rag-evaluation/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/sagar-shirwalkar/servicenow-atlas/atlas-rag-evaluation"><img src="https://agentmods.dev/badge/skills/sagar-shirwalkar/servicenow-atlas/atlas-rag-evaluation.svg" alt="Reviewed on agentmods" width="80" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00053 | $0.00792 |
| Opus 5.5 | $0.00021 | $0.00317 |
| Sonnet 5.5 | $0.00011 | $0.00158 |
| Haiku 4.5 | $0.00005 | $0.00079 |
Grade A, and why
atlas-rag-evaluation scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 1mo ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 98 lines — stays where its author put it; the contents beside it link to each section on GitHub.
When to use
- User asks to "evaluate RAG quality", "run the evaluation script", "check search quality"
- User is comparing MLX vs ONNX+CPU embedding quality
- User wants to verify that a CI-built bundle meets the quality bar
- User suspects a regression in search relevance after a change
Workflow
1. Select a bundle
List available bundles:
ls data/bundles/
Pick the one you want to evaluate.
2. Run the evaluation
# Point BUNDLE_DIR at the bundle and run
BUNDLE_DIR=data/bundles/<timestamp> uv run atlas-eval
If the eval script is already registered as a console script. Otherwise run directly:
BUNDLE_DIR=data/bundles/<timestamp> uv run python atlas/evaluate_rag.py
3. Read the results
The output looks like:
Query: "how to create an incident"
P@10: 0.90 MRR: 1.0000
...
Mean P@10 over 10 queries: 0.89
Mean MRR over 10 queries: 0.94
Precision@10 (P@10): Fraction of the top-10 results that are relevant. A score of 0.89 means ~9 of 10 returned chunks are on-topic.
Mean Reciprocal Rank (MRR): How high the first relevant result appears. 1.0 means the top result is always relevant; 0.94 means the first relevant result is typically in position 1–2.
4. Benchmark bar
| Metric | Minimum | Target | Excellent |
|---|---|---|---|
| Precision@10 | 0.75 | 0.85 | 0.95+ |
| MRR | 0.80 | 0.90 | 0.97+ |
- A score below Minimum means the bundle is not usable — investigate chunk boundaries, model, or embeddings.
- Scores between Minimum and Target may be acceptable for development but should not ship to CI.
- Target or higher is required for a release build.
5. Compare CI vs local
To compare, build two bundles with the same model on CI and locally:
# Local (MLX)
uv run atlas-build --model Xenova/bge-small-en-v1.5
BUNDLE_DIR=data/bundles/<local> uv run atlas-eval
# Download CI bundle from GitHub Releases
gh release download <tag> --pattern '*.tar.zst'
tar --zstd -xf atlas-bundle-*.tar.zst -C /tmp/ci-bundle
BUNDLE_DIR=/tmp/ci-bundle uv run atlas-eval
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 1mo ago First seen · 98 lines · 53 tokens per session scan A 420147276654
atlas-rag-evaluation is a skill published in the GitHub repository sagar-shirwalkar/servicenow-atlas (2 stars, last pushed 4mo ago), licensed Apache-2.0. It adds 53 tokens to every session and 792 once invoked, about $0.0002 per session on Opus 5.5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.
Other skills, from other repositories
graph-retrieval
Exposes graph-based retrieval as a tool capability via querygraph. Reads normalized graph store files, builds a query-relevant subgraph, and returns LLM-friendly semantic triples with replayable evidence metadata.
knowledge-layer
High-level deployment wrapper over RepoBrain core with graph-first knowledge injection and all-file support. Exposes refreshfilesystem and askfilesystem for building and querying the knowledge graph.
rag
Use when building grounded Q&A over your own corpus — chunk, retrieve hybrid, rerank, ground, cite chunk ids, refuse when the sources fall short — or when the right document is retrieved but the answer is still wrong, invented, or unmeasured. NOT operating the store itself — collection schema, HNSW efsearch…
rag-index-decisions
After making a non-obvious architectural decision, solving a novel bug, agreeing on a coding standard, or reaching a conclusion worth remembering, index it back into the knowledge base so the next occurrence is one search away. Uses adddocument or addfromurl. Closes the feedback loop that makes a RAG-backed team…
rag-code-review
When performing code review on a PR, diff, snippet, or "look at this change" request, first consult the corpus for related ADRs, coding standards, prior patterns, and similar files. Grounds review comments in the team's actual decisions instead of generic best practices. Trigger on any review-style request — "review"…
rag-check-first
Before answering any technical question, code request, architecture decision, or factual claim, call searchknowledge to check the local corpus. Trigger on any query that could be answered with prior work, indexed docs, ADRs, runbooks, or team context. Prevents hallucination and forces reliance on the indexed knowledge…