Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx agentmods add skills/saitarrun/devforge-ai/llm-evalsnpx skills add saitarrun/Devforge-ai --skill llm-evalsgit clone --depth 1 https://github.com/saitarrun/Devforge-aiWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/saitarrun/devforge-ai/llm-evals)<a href="https://agentmods.dev/skills/saitarrun/devforge-ai/llm-evals"><img src="https://agentmods.dev/badge/skills/saitarrun/devforge-ai/llm-evals.svg" alt="Measured on agentmods" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5 | $0.00031 | $0.00455 |
| Opus 5 | $0.00015 | $0.00228 |
| Sonnet 5 | $0.00006 | $0.00091 |
| Haiku 4.5 | $0.00003 | $0.00046 |
Grade A, and why
llm-evals scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 5d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
What it actually says
LLM & RAG Application Engineering Skill
This skill defines industry standards for developing, evaluating, and operating LLM, RAG, and AI Agent features cleanly.
Core Architectural Pillars
1. Evaluation & Benchmarking (Evals)
- Deterministic Evals: Assert output schema validity (Zod/Pydantic validation).
- Model-Graded Evals: Evaluate accuracy, grounding (faithfulness), hallucination rate, and context relevance using LLM-as-a-Judge.
- RAG Triad:
- Context Relevance (Retrieval quality)
- Groundedness (LLM stays within retrieved context)
- Answer Relevance (Output addresses user prompt directly)
2. Token & Cost Budget Guardrails
- Max Token Limits: Hard limit max response tokens per prompt.
- Circuit Breakers: Halt downstream requests if daily/hourly token expenditure exceeds set threshold.
- Semantic Caching: Store prompt/embedding responses in Redis/pgvector to eliminate redundant LLM calls.
3. AI Security & Safety
- Prompt Injection Defense: Sanitize user inputs; separate system instructions from untrusted user content.
- PII Scrubbing: Redact secrets, emails, SSNs, and credit card numbers before sending payloads to LLM APIs.
- OWASP Top 10 for LLMs: Guard against Insecure Output Handling, Excessive Agency, and Data Poisoning.
Checklist for Feature Slices with has_llm: true
- Schema validation for structured output (JSON mode / Tool Call response parsing)
- Fallback model strategy (e.g., fallback from primary model to secondary model on rate limit)
- OpenTelemetry LLM tracing (LangSmith / Helicone / Phoenix / OTel instrumentation)
- Token usage logging and latency tracking
- Eval dataset created with at least 10 gold-standard ground truth examples
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 5d ago First seen · 37 lines · 31 tokens per session scan A 9e804011bafa
llm-evals is a skill published in the GitHub repository saitarrun/Devforge-ai (5 stars, last pushed 22d ago), licensed Apache-2.0. It adds 31 tokens to every session and 455 once invoked, about $0.0002 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.
Other skills, from other repositories
llamacloud-index
Use this skill whenever a question should be answered from documents stored in a LlamaCloud Index v2 knowledge base — retrieving passages, locating indexed files, or searching indexed content through the LlamaParse MCP Index v2 tools (retrieveFromIndex, findFilesInIndex, grepFileFromIndex...). Teaches agentic…
aws-bedrock-ai
WORKFLOW SKILL — Amazon Bedrock and AWS AI design: foundation model selection, knowledge bases (RAG), agents for bedrock, guardrails, provisioned throughput, batch inference, fine-tuning, KMS, VPC endpoints, regional GA, and per-provider licensing.
AgentDB Performance Optimization
Optimize AgentDB performance with quantization (4-32x memory reduction), HNSW indexing (150x faster search), caching, and batch operations. Use when optimizing memory usage, improving search speed, or scaling to millions of vectors.
AgentDB Memory Patterns
Implement persistent memory patterns for AI agents using AgentDB. Includes session memory, long-term storage, pattern learning, and context management. Use when building stateful agents, chat systems, or intelligent assistants.
AgentDB Vector Search
Implement semantic vector search with AgentDB for intelligent document retrieval, similarity matching, and context-aware querying. Use when building RAG systems, semantic search engines, or intelligent knowledge bases.
V3 Memory Unification
Unify 6+ memory systems into AgentDB with HNSW indexing for 150x-12,500x search improvements. Implements ADR-006 (Unified Memory Service) and ADR-009 (Hybrid Memory Backend).