An automatic check for Sample Ratio Mismatch, or SRM, in experiment data. SRM means the treatment and control groups in an A/B test do not have the expected split, which can make the comparison unreliable.

Use it before analyzing A/B tests or other experiments to check treatment and control counts and block analysis when the split is compromised.

It catches broken randomization before treatment results are calculated. This prevents analysis from proceeding when the experiment groups are not formed as expected.

srm-check — Skill for Claude Code by ai-analyst-lab

Install

Getting it into your agent

One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.

Any agent

npx skills add ai-analyst-lab/ai-analyst --skill srm-check

Clone the repo

git clone --depth 1 https://github.com/ai-analyst-lab/ai-analyst

Made for: Claude Code.

Wrote this? Show the measurements

A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.

README.md

[![agentmods](https://agentmods.dev/badge/skills/ai-analyst-lab/ai-analyst/srm-check/github.svg)](https://agentmods.dev/skills/ai-analyst-lab/ai-analyst/srm-check)

Your own site

<a href="https://agentmods.dev/skills/ai-analyst-lab/ai-analyst/srm-check"><img src="https://agentmods.dev/badge/skills/ai-analyst-lab/ai-analyst/srm-check/github.svg" alt="Measured on agentmods" height="20"></a>

Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.

Your own site · 80×15

<a href="https://agentmods.dev/skills/ai-analyst-lab/ai-analyst/srm-check"><img src="https://agentmods.dev/badge/skills/ai-analyst-lab/ai-analyst/srm-check.svg" alt="Reviewed on agentmods" width="80" height="20"></a>

Per session 320 Skills are progressive disclosure: only the name and description are preloaded; the body loads when the skill is used.

When invoked 2,157 The whole file, excluding the scripts and references it only reads on demand.

Security scan A 0 findings. A grade says what 26 rules found in the file — not that it is safe.

Origin original No closer match found in the catalogue.

Token cost

What it costs to keep this loaded

Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.

Model	Per session	Once invoked
Fable 5.1	$0.00320	$0.02157
Opus 5	$0.00160	$0.01078
Sonnet 5	$0.00064	$0.00431
Haiku 4.5	$0.00032	$0.00216

Measured 2d ago against content hash 8cc05c00dec4, method: parsed. Prices are Anthropic first-party input rates as of 2026-09-13, from the pricing page.

Security

Grade A, and why

srm-check scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 2d ago.

A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.

Nothing flagged

None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.

Origin

Copies of this mod

1 near-identical copy found in the catalogue:

srm-check — 88% identical, 30 lines differ

.claude/skills/srm-check/SKILL.md · 176 lines

How it starts

The opening of the file, as written. The whole thing — 176 lines — stays where its author put it; the contents beside it link to each section on GitHub.

Skill: SRM Check (Sample Ratio Mismatch)

Purpose

Automatically detect Sample Ratio Mismatch in experiment data before any analysis proceeds. SRM is a randomization integrity check — if the treatment/control split deviates significantly from the expected ratio, the experiment is compromised and results cannot be trusted. This skill acts as a safety gate that blocks analysis when randomization is broken.

When to Use

Apply this skill when:

Loading any experiment or A/B test dataset — auto-fire on detection of treatment/control columns (e.g., variant, group, treatment, arm, experiment_group)
Before any treatment effect calculation — SRM must pass before comparing outcomes
When the Experiment Analyzer agent starts — first step of any experiment analysis workflow

This skill auto-fires on experiment data detection. Do NOT wait to be asked.

Instructions

What Is SRM?

Sample Ratio Mismatch (SRM) occurs when the observed ratio of users in treatment vs. control deviates significantly from the expected ratio. For a 50/50 experiment with 10,000 users, you expect ~5,000 in each group. If you see 5,500 vs. 4,500, something is wrong with randomization.

SRM CHECK
━━━━━━━━━━
Expected ratio:  50/50 (or whatever was designed)
Observed ratio:  [actual counts]
Test:            Chi-squared goodness-of-fit
Decision:        PASS (proceed) or BLOCK (halt analysis)

Why SRM matters: If randomization is broken, treatment and control groups are NOT comparable. Any observed difference in outcomes could be caused by the broken randomization, not the treatment. SRM is the single most important validity check in experimentation.

Detection Logic

Step 1: Identify the experiment column

Scan the dataset for columns that indicate experiment assignment. Look for:

Column names: variant, group, treatment, control, arm, experiment_group, test_group, bucket, condition
Column values: binary (0/1, control/treatment, A/B), or small number of distinct values (< 10)
User language: phrases like "A/B test", "experiment", "treatment vs control", "randomization"
Metadata: check for experiment config files in .knowledge/experiments/

Read the full file on GitHub · 176 lines

Changes

What this file has done since we first saw it

Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.

2d ago First seen · 176 lines · 320 tokens per session scan A 8cc05c00dec4

Subscribe to this mod's changes

srm-check is a skill published in the GitHub repository ai-analyst-lab/ai-analyst (298 stars, last pushed 3d ago), licensed MIT. It adds 320 tokens to every session and 2,157 once invoked, about $0.0016 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-09-12.

Analytics and metrics Testing

research-engineer

An uncompromising Academic Research Engineer. Operates with absolute scientific rigor, objective criticism, and zero flair. Focuses on theoretical correctness, formal verification, and optimal implementation across any required technology.

davila7/claude-code-templates · 43 tokens

tika-eval-compare

Compare extracts from two Tika builds over a corpus to detect regressions in content, encoding, exceptions, and embedded-document handling. Use for "compare before/after extracts", "eval this change against the corpus".

apache/tika · 50 tokens

neuron-evaluation-engineer

Create and run AI evaluations with datasets, assertions, and output drivers in Neuron AI. Use this skill whenever the user mentions evaluation, testing AI systems, creating evaluators, dataset-driven testing, assertion-based validation, or wants to measure AI system performance. Also trigger for tasks involving…

neuron-core/neuron-ai · 77 tokens

srm-check