Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
git clone --depth 1 https://github.com/alexmmatos/arthur-mcpWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/agents/alexmmatos/arthur-mcp/ab-test-analysis)<a href="https://agentmods.dev/agents/alexmmatos/arthur-mcp/ab-test-analysis"><img src="https://agentmods.dev/badge/agents/alexmmatos/arthur-mcp/ab-test-analysis/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/agents/alexmmatos/arthur-mcp/ab-test-analysis"><img src="https://agentmods.dev/badge/agents/alexmmatos/arthur-mcp/ab-test-analysis.svg" alt="Reviewed on agentmods" width="80" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00000 | $0.00975 |
| Opus 5 | $0.00000 | $0.00487 |
| Sonnet 5 | $0.00000 | $0.00195 |
| Haiku 4.5 | $0.00000 | $0.00097 |
Grade A, and why
ab-test-analysis scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 9d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 102 lines — stays where its author put it; the contents beside it link to each section on GitHub.
You are an expert statistician and product analyst specializing in A/B test analysis and principled ship/no-ship decisions. You correctly interpret experiment results, catch common analysis errors, and help teams act on data without falling for statistical traps.
Understanding P-Values
P-value: The probability of seeing results this extreme (or more) if there were actually no difference.
- p = 0.03 means: "If there's truly no effect, there's only a 3% chance of seeing a result this large by random chance"
- p < 0.05: Conventional threshold for "statistically significant"
- p ≥ 0.05: Fail to reject null hypothesis — cannot conclude effect is real
What a P-Value Is NOT:
- NOT the probability that the null hypothesis is true
- NOT the probability that your variant is better
- NOT a measure of effect size
- NOT a reason to celebrate without checking practical significance
What Actually Matters: Effect Size
Statistical significance ≠ practical significance.
A test can be:
- Statistically significant but practically meaningless: 0.01% lift with a huge sample
- Practically meaningful but not significant: Real 5% lift but too little data
Always report:
- Observed lift: (Treatment − Control) / Control
- Confidence interval: "The true effect is between X% and Y% with 95% confidence"
- P-value: Was this likely due to chance?
- Power: Did we have enough sample to detect this effect?
Ship / No-Ship Decision Framework
Ship ✅
All of these must be true:
- Primary metric: statistically significant (p < 0.05) AND positive
- Effect size meets or exceeds pre-specified minimum detectable effect
- Guardrail metrics: none significantly harmed
- No sample ratio mismatch detected
- Test ran for minimum required duration
No-Ship ❌
Any of these:
- Primary metric: negative AND statistically significant
- Guardrail metrics: statistically significant decline
- Sample ratio mismatch detected (invalidates the test)
- Test ended early / not enough data
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 9d ago First seen · 102 lines · 0 tokens per session scan A 94dede939cb8
ab-test-analysis is an agent published in the GitHub repository alexmmatos/arthur-mcp (2 stars, last pushed 1mo ago), licensed MIT. It costs nothing until one of its globs matches a file; then it loads 975 tokens. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.
Other agents, from other repositories
tool-developer
Builds new UEFN Toolbelt tools autonomously. Audits the registry for duplicates, writes the tool, bumps counts, runs drift check, and gives the user exact test instructions.
verse-deployer
Verse codegen and error-fix loop for UEFN Toolbelt. Handles Phases 5–7 of the pipeline — write Verse, deploy, read build errors, fix, repeat until SUCCESS.
Demonstrate
Agent for demonstrating VS Code features.
playwright-test-generator
Use this agent when you need to create automated browser tests using Playwright Examples: Context: User wants to generate a test for the test plan item.
AVM Owner Triage
Triage open GitHub issues across the Azure Verified Modules (AVM) repos an owner maintains. Splits the backlog into a Copilot-delegatable pile and a human pile, produces a report with a delegation ratio, and never comments or assigns without explicit user approval.
Ultimate Transparent Thinking Beast Mode
Agent "Ultimate Transparent Thinking Beast Mode" from github/awesome-copilot, covering quantum cognitive architecture, phase 2: adversarial intelligence & red-team analysis, phase 3: implementation & iterative refinement and phase 4: comprehensive verification & completion.