Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add PostHog/posthog-foss --skill creating-online-evaluationsgit clone --depth 1 https://github.com/PostHog/posthog-fossWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/posthog/posthog-foss/creating-online-evaluations)<a href="https://agentmods.dev/skills/posthog/posthog-foss/creating-online-evaluations"><img src="https://agentmods.dev/badge/skills/posthog/posthog-foss/creating-online-evaluations/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/posthog/posthog-foss/creating-online-evaluations"><img src="https://agentmods.dev/badge/skills/posthog/posthog-foss/creating-online-evaluations.svg" alt="Reviewed on agentmods" width="80" height="20"></a>- NVIDIA SkillSpector warn
SkillSpector: 3 findings, up to medium
These are SkillSpector’s own severities. On a checked sample its high-severity flags on skills were ~96% false positives — a documented command, a public API, a “never do X” rule — so we show them as a caution to read, not a verdict. Why →
- medium Prompt Injection · line 132 Large whitespace padding was detected (a block of blank lines or a long run of spaces). This can push injected instructions below or to the right of the visible area so a human reviewer never sees them while the agent still reads them. Manual review of the hidden content is recommended.Fix: Remove the large whitespace padding (blank-line blocks or long space runs) and review any content hidden below or to the right of it. Keep skill files compact and reviewable so no instructions can be
- medium Prompt Injection · line 142 Large whitespace padding was detected (a block of blank lines or a long run of spaces). This can push injected instructions below or to the right of the visible area so a human reviewer never sees them while the agent still reads them. Manual review of the hidden content is recommended.Fix: Remove the large whitespace padding (blank-line blocks or long space runs) and review any content hidden below or to the right of it. Keep skill files compact and reviewable so no instructions can be
- medium Prompt Injection · line 192 Large whitespace padding was detected (a block of blank lines or a long run of spaces). This can push injected instructions below or to the right of the visible area so a human reviewer never sees them while the agent still reads them. Manual review of the hidden content is recommended.Fix: Remove the large whitespace padding (blank-line blocks or long space runs) and review any content hidden below or to the right of it. Keep skill files compact and reviewable so no instructions can be
What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00199 | $0.05561 |
| Opus 5 | $0.00100 | $0.02780 |
| Sonnet 5 | $0.00040 | $0.01112 |
| Haiku 4.5 | $0.00020 | $0.00556 |
Grade A, and why
creating-online-evaluations scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 4d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
Copies of this mod
1 near-identical copy found in the catalogue:
- creating-online-evaluations — 100% identical, 0 lines differ
How it starts
The opening of the file, as written. The whole thing — 375 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Creating online evaluations
An online evaluation automatically scores either each matching $ai_generation or the whole trace
containing it, until disabled. A good eval comes from a real failure mode you've found in production traffic,
not from a guess or a generic metric like "hallucination" or "helpfulness". This skill starts once those
failure modes are identified and turns them into scoped, continuously-running evals.
One eval per failure mode, and as many evals as the data justifies. How many to create is a judgment call you make from what the traces actually showed — sometimes one, often three or four. Never assume the answer is one, and never bundle several modes into one evaluator.
Propose before you create. Bring the user a short list of candidate evals and let them pick which ones they want (Phase 1.1). Creating evals they didn't ask for costs them money and noise.
First, know what you're evaluating. Finding and ranking the failure modes worth catching is a
separate job. If the user doesn't specify what they want to evaluate, ask them. If they are still vague
about it and don't refer to a specific failure mode, run exploring-ai-failures to scope a use case,
find failing traces, and produce a ranked list of failure modes.
For the mechanics of writing and iterating an evaluator (Hog source vs LLM-judge prompt, dry-running,
debugging a live eval), defer to exploring-llm-evaluations.
Tools
| Tool | Purpose |
|---|---|
posthog:llma-evaluation-config-get |
Check the active provider key used by unpinned judges |
posthog:llma-provider-key-list |
Find a usable (ok state) provider key to pin |
posthog:llma-evaluation-judge-models |
List valid provider+model combos |
posthog:llma-evaluation-directory-list |
List directories available for organizing the evaluation |
posthog:llma-evaluation-directory-create |
Create a directory when the user asks for a new one |
posthog:llma-evaluation-test-hog |
Dry-run Hog source against recent generations before creating |
posthog:llma-evaluation-create |
Create the evaluation (always enabled: false first) |
posthog:llma-evaluation-run |
Spot-run a draft eval against one generation |
posthog:llma-evaluation-update |
Iterate config, then flip enabled: true |
posthog:execute-sql |
Verify a condition matches the events and volume you expect |
posthog:generate-app-url |
Build a region- and project-qualified deep link to the eval |
What ships with it
1 file beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 4d ago Changed · +7 lines · -4 tokens per session fcef11b03bbc
- 5d ago First seen · 368 lines · 203 tokens per session scan A 2e05bdc18714
creating-online-evaluations is a skill published in the GitHub repository PostHog/posthog-foss (714 stars, last pushed today), licensed MIT. It adds 199 tokens to every session and 5,561 once invoked, about $0.0010 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-09-03.
Other skills, from other repositories
cli-eval
Create and run evaluation suites, watch live benchmark progress, view scorecards, compare model performance, and integrate eval runs with CI workflows from the CLI.
model-merging
Merge multiple fine-tuned models using mergekit to combine capabilities without retraining. Use when creating specialized models by blending domain-specific expertise (math + coding + chat), improving performance beyond single models, or experimenting rapidly with model variants. Covers SLERP, TIES-Merging, DARE, Task…
darwinian-evolver
Evolve prompts/regex/SQL/code with Imbue's evolution loop.
validate
Validate Semantica pipelines, extraction quality, graph schemas, and ontology consistency. Returns structured error/warning checklists. Uses PipelineValidator, PipelineBuilder.validatepipeline(), GraphValidator, and OntologyValidator. Sub-commands: pipeline, step, dependencies, extraction, graph, ontology, performance.
launching-evals
Run, monitor, analyze, and debug LLM evaluations via nemo-evaluator-launcher. Covers running evaluations, checking status and live progress, debugging failed runs, exporting artifacts and logs, and analyzing results. ALWAYS triggers on mentions of running evaluations, checking progress, debugging failed evals…
nemo-automodel-recipe-development
Create and modify NeMo AutoModel training and evaluation recipes, including YAML structure, builders, and execution flow.