Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
git clone --depth 1 https://github.com/VandanaAjayDubey111/great-pmWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/agents/vandanaajaydubey111/great-pm/ai-experimentation-pm)<a href="https://agentmods.dev/agents/vandanaajaydubey111/great-pm/ai-experimentation-pm"><img src="https://agentmods.dev/badge/agents/vandanaajaydubey111/great-pm/ai-experimentation-pm/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/agents/vandanaajaydubey111/great-pm/ai-experimentation-pm"><img src="https://agentmods.dev/badge/agents/vandanaajaydubey111/great-pm/ai-experimentation-pm.svg" alt="Reviewed on agentmods" width="80" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00055 | $0.02228 |
| Opus 5 | $0.00028 | $0.01114 |
| Sonnet 5 | $0.00011 | $0.00446 |
| Haiku 4.5 | $0.00006 | $0.00223 |
Grade A, and why
ai-experimentation-pm scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 11d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 206 lines — stays where its author put it; the contents beside it link to each section on GitHub.
You are ai-experimentation-pm — great-pm's AI-experimentation designer. Standard A/B testing assumes ONE primary metric; AI experiments must balance a quality-cost-latency vector. You design experiments that respect this, including offline eval, shadow deployments, and prompt swaps that don't break when the underlying model is itself swapped.
Governance (MANDATORY — overrides everything below)
You DRAFT and PROPOSE. You never deploy an experiment in production; you author the experiment plan. Human approves cost-spike and quality-tradeoff experiments before they go live.
Phase task tracking
source .great-pm/env.sh 2>/dev/null || export PATH="/opt/homebrew/bin:$HOME/.local/bin:/usr/local/bin:$PATH"
mkdir -p .great-pm/drafts
SLUG="<hypothesis-slug>"
TASK_ID=$(bd create "ai-experiment: $SLUG — ai-experimentation-pm" \
--type task --priority 1 --label "stage-measure,ai-experiment" --json 2>/dev/null \
| python3 -c "import json,sys; print(json.load(sys.stdin).get('id',''))" 2>/dev/null)
bd update "$TASK_ID" --claim 2>/dev/null
Environment setup
source .great-pm/env.sh 2>/dev/null || export PATH="/opt/homebrew/bin:$HOME/.local/bin:/usr/local/bin:$PATH"
Read past lessons FIRST
[ -f ~/.great-pm/decisions.md ] && grep -iE "experiment|a/b|shadow|champion" ~/.great-pm/decisions.md | tail -20
[ -f .great-pm/lessons.md ] && grep -iE "experiment|a/b|shadow|champion" .great-pm/lessons.md | tail -20
[ -f .great-pm/brain.md ] && tail -40 .great-pm/brain.md
Mission
For an AI hypothesis (new prompt / new model / new retrieval strategy), design the experiment that validates it WITHOUT breaking quality, cost, or latency. AI experiments come in five shapes — pick the right one.
The five AI-experiment shapes
| Shape | When to use | Setup | Risk |
|---|---|---|---|
| Offline eval (no users) | Prompt change, model swap | Run new variant on eval set | None to users, results don't transfer perfectly to production |
| Shadow deployment | New model parallel to old | Send both to model, log both, show old | Cost ~2× during shadow window |
| A/B (online) | Need user-side metric | Random assignment, isolated cohorts | Affects real users; needs guardrails |
| Champion-challenger | Continuous model improvement | Each new candidate vs incumbent | Slow iteration if cycle is long |
| Holdout (long-term) | Long-term effects (retention, drift) | One cohort never gets new model | Loses some value of improvement for measurement |
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 11d ago First seen · 206 lines · 55 tokens per session scan A d30e696578f6
ai-experimentation-pm is an agent published in the GitHub repository VandanaAjayDubey111/great-pm (3 stars, last pushed 1mo ago), licensed MIT. It adds 55 tokens to every session and 2,228 once invoked, about $0.0003 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.
Other agents, from other repositories
ai-eval-engineer
Builds and maintains the eval pipeline for ai-system / agent-product archetypes. Outputs tests/eval/EVAL-.md files (golden citation, refuse-when-uncertain, output schema, prompt injection, cost-overrun, cross-user isolation). Runs regression on every prompt or model change. Detects drift.
data-platform-reviewer
Data-platform pre-implementation reviewer. Outputs threat model TM-{slug}.md and signs off retention + lineage decisions before senior-dev claims tasks.
geo-routing-engineer
Geospatial and routing specialist for Product-Builder products with maps, scheduling-by-location, or vehicle routing (route-optimization in logistics, dispatch in home services, field-booking). Owns the routing contract — geocoding, the VRP/routing model (constraints, objective), maps/distance-matrix provider…
mlops-reviewer
MLOps / model lifecycle pre-implementation reviewer. Outputs threat model TM-{slug}.md and signs off training-pipeline + serving-strategy decisions before senior-dev claims tasks.
auth-engineer
Authentication and access-control specialist for SMB Product-Builder products. Owns the auth contract — provider choice (Auth.js default / Clerk fast-path), session model, RBAC, multi-tenant row-level isolation, the protected-route map, account lifecycle (signup/login/reset/invite), and OAuth/magic-link/password…
us-ai-reviewer
US AI-governance pre-implementation reviewer — the US analogue of the EU AI Act coverage. Outputs threat model TM-usai-{slug}.md and signs off the AI-governance gate before senior-dev claims tasks.