Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add saski/arnesto --skill promptfoo-evalsgit clone --depth 1 https://github.com/saski/arnestoWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/saski/arnesto/promptfoo-evals)<a href="https://agentmods.dev/skills/saski/arnesto/promptfoo-evals"><img src="https://agentmods.dev/badge/skills/saski/arnesto/promptfoo-evals/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/saski/arnesto/promptfoo-evals"><img src="https://agentmods.dev/badge/skills/saski/arnesto/promptfoo-evals.svg" alt="Reviewed on agentmods" width="80" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00047 | $0.00700 |
| Opus 5.5 | $0.00019 | $0.00280 |
| Sonnet 5.5 | $0.00009 | $0.00140 |
| Haiku 4.5 | $0.00005 | $0.00070 |
Grade A, and why
promptfoo-evals scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 1mo ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 75 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Promptfoo Evaluations
Use promptfoo when the question concerns observable LLM behavior across a repeatable set of inputs. A deterministic function or configuration invariant usually belongs in the project's ordinary test suite instead.
Define what is being evaluated
State the behavior, available evidence, and unacceptable outcome before writing the case. Choose the evaluation boundary explicitly:
- Prompt only: send the candidate prompt and controlled context to a model.
- Configured client: invoke the real agent entry point with its instructions, tools, and runtime configuration. Attribute the result to that combination.
A client passing a case does not prove the harness caused the behavior. To test that claim, compare with and without the relevant instructions while keeping the cases, model, and runtime settings stable. Preserve a user's requested provider; report unavailable models rather than silently replacing them.
Build a useful case and grader
Start with a representative failure mode from actual use. Include enough evidence to determine the expected behavior without putting the expected answer in the evaluated prompt. Add ambiguity or pressure only when it models a real risk. Expand coverage from observed gaps, not arbitrary case counts.
Prefer deterministic assertions for structure, exact facts, and prohibited actions. Check the substantive claim as well as the format: valid JSON alone is not success. Avoid exact prose matching when equivalent wording is valid. For tool-use claims, inspect execution traces rather than the agent's self-report.
Exercise the grader with a known correct response and plausible incorrect responses before trusting it. Use an LLM judge only for properties that need semantic judgment; record its model and rubric and check its judgments against examples reviewed by a person. Do not weaken assertions just to obtain green runs.
Run and interpret
Use the project's existing installation, configuration, and execution targets. For new integrations, pin the evaluator and commit the dependency lockfile. Check current official documentation when adding unfamiliar provider or assertion configuration; avoid copying guessed API syntax.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 1mo ago First seen · 75 lines · 47 tokens per session scan A ccf78eb024b5
promptfoo-evals is a skill published in the GitHub repository saski/arnesto (5 stars, last pushed yesterday), licensed Unlicense. It adds 47 tokens to every session and 700 once invoked, about $0.0002 per session on Opus 5.5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-09-17.
Other skills, from other repositories
ai-observability-promptfoo
Testing and evaluation framework for LLM prompts and applications -- promptfooconfig.yaml, assertions, model-graded evals, red teaming, CI/CD integration, custom providers, and comparative evaluation.
performance:hill-climb
Use when asked to make software faster or leaner (runtime, startup, throughput, memory, bundle size) through repeated measured changes. For prompts or skills scored by an eval suite, use prompting:hill-climb.
prompt-optimization-loop
Prompt-engineering expertise for designing test cases, edge cases, adversarial inputs, and iterating on prompts based on eval results.
prompt-evaluation-runner
Use when evaluating prompts, LLM outputs, red-team suites, or model behavior with local eval configs and safe provider/cost controls.
llm-tester
You are the LLM Tester, specializing in systematic prompt evaluation, red-teaming, and LLM output quality assurance. You replace "vibes-based" AI evaluation with rigorous, automated, and repeatable verification suites.
promptfoo-evaluation
Configure and run Promptfoo evaluations, assertions, rubrics, and datasets when testing LLM prompts or comparing model behavior.