Getting it into your agent
It runs from inside its repository, so the clone comes first — what it calls does not travel with the file alone.
git clone --depth 1 https://github.com/softspark/ai-toolkitnpx agentmods add skills/softspark/ai-toolkit/evaluateWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/softspark/ai-toolkit/evaluate)<a href="https://agentmods.dev/skills/softspark/ai-toolkit/evaluate"><img src="https://agentmods.dev/badge/skills/softspark/ai-toolkit/evaluate.svg" alt="Measured on agentmods" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00042 | $0.01140 |
| Opus 5 | $0.00021 | $0.00570 |
| Sonnet 5 | $0.00008 | $0.00228 |
| Haiku 4.5 | $0.00004 | $0.00114 |
Grade A, and why
evaluate scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 3d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 154 lines — stays where its author put it; the contents beside it link to each section on GitHub.
RAG Evaluation
Evaluate RAG quality using LLM-as-a-Judge methodology.
Usage
/evaluate [--threshold 0.7]
Execution
Direct Execution (recommended for most projects)
# Run RAG evaluation
python3 scripts/evaluate_rag.py
# With custom thresholds
python3 scripts/evaluate_rag.py \
--faithfulness 0.7 \
--relevancy 0.7 \
--context 0.6
# Detect knowledge gaps
python3 scripts/knowledge_gaps.py --detect
# Generate gap report
python3 scripts/knowledge_gaps.py --report
Docker Execution (containerized projects)
# Replace {api-container} with your API server container name
docker exec {api-container} python3 scripts/evaluate_rag.py
# With custom thresholds
docker exec {api-container} python3 scripts/evaluate_rag.py \
--faithfulness 0.7 \
--relevancy 0.7 \
--context 0.6
# Detect knowledge gaps
docker exec {api-container} python3 scripts/knowledge_gaps.py --detect
# Generate gap report
docker exec {api-container} python3 scripts/knowledge_gaps.py --report
Metrics
| Metric | Description | Target |
|---|---|---|
| Faithfulness | Is answer based on context? | >70% |
| Relevancy | Does answer address question? | >70% |
| Context Precision | Is found context accurate? | >60% |
Evaluation Process
- Generate test queries from golden dataset
- Execute RAG pipeline for each query
- LLM judges each response on metrics
- Report aggregate scores
Golden Dataset
Located at: scripts/golden_dataset.json (or project-specific path)
{
"queries": [
{
"query": "How to configure rate limiting?",
"expected_topics": ["nginx", "rate-limiting"],
"expected_sources": ["kb/nginx/howto/rate-limiting.md"]
}
]
}
Output Example
RAG Evaluation Results
======================
Total Queries: 50
Average Faithfulness: 0.82
Average Relevancy: 0.78
Average Context Precision: 0.71
Quality: GOOD
Failed Queries (faithfulness < 0.7):
- Query: "How to backup PostgreSQL?"
Score: 0.45
Issue: No relevant documents found
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 3d ago First seen · 154 lines · 42 tokens per session scan A 09c45695469d
evaluate is a skill published in the GitHub repository softspark/ai-toolkit (170 stars, last pushed 2d ago), licensed Apache-2.0. It adds 42 tokens to every session and 1,140 once invoked, about $0.0002 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-09-03.
Other skills, from other repositories
ai-portable-setup
Erstellt einen portablen KI-Arbeitsbereich auf einem USB-Stick oder beliebigen Laufwerk. RAG-Pipeline mit lokalen LLM-Modellen (Ollama), Vektordatenbank (ChromaDB) und vorkonfigurierten Prompts.
document-chunker
Split documents into overlapping token chunks for RAG pipelines and LLM context windows. Zero dependencies.
continue-gemini-explicit
Continue's Gemini provider doesn't use the cachedContents API at all. Add explicit caching for sessions over the minimum token threshold.
roo-fix-volatile-msg
Ladder-aware Roo Code Anthropic caching — verify the rolling read/write ladder on the wire, then close the real gaps (Vertex 4-block budget, MiniMax path).
roo-bedrock-custom-arn
Roo Code's Bedrock provider silently disables caching for custom ARNs. Populate cachableFields to fix.
langchain
Apply when building LangChain pipelines, LCEL chains, agents, or retrieval-augmented generation systems.