Agent Substrate is a runtime system for running many isolated applications or AI agents on shared computing infrastructure. It manages agent sandboxes throughout their lifecycle, places them on available workers, and routes traffic to them while using technologies such as Kubernetes, microVMs, and gVisor. Its catalogue skills and instructions support operating and developing workflows around this agent runtime.
Borrowing it
Nothing to install: this file belongs to agent-substrate/substrate. Take a copy, put it at the same path in your own repository, and replace the rules that are about this project with yours.
curl -O https://raw.githubusercontent.com/agent-substrate/substrate/main/.agents/skills/detect-flaky-tests/SKILL.mdgit clone --depth 1 https://github.com/agent-substrate/substrateWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/agent-substrate/substrate/detect-flaky-tests)<a href="https://agentmods.dev/skills/agent-substrate/substrate/detect-flaky-tests"><img src="https://agentmods.dev/badge/skills/agent-substrate/substrate/detect-flaky-tests/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/agent-substrate/substrate/detect-flaky-tests"><img src="https://agentmods.dev/badge/skills/agent-substrate/substrate/detect-flaky-tests.svg" alt="Reviewed on agentmods" width="80" height="20"></a>- NVIDIA SkillSpector pass
What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00102 | $0.03468 |
| Opus 5.5 | $0.00041 | $0.01387 |
| Sonnet 5.5 | $0.00020 | $0.00694 |
| Haiku 4.5 | $0.00010 | $0.00347 |
Grade A, and why
detect-flaky-tests scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 6d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 347 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Detect Flaky Tests
A test is flaky when it produces both PASS and FAIL outcomes across multiple independent CI runs in the last 7 days, with no code change to that test's package explaining the inconsistency. Cross-PR analysis provides the strongest signal: if the same test fails on PR-A but passes on PR-B, that inconsistency is almost certainly non-determinism, not a legitimate regression.
Flakiness threshold (keeps false-positive rate low)
A test is flagged only when all three conditions hold in the 7-day window:
| Condition | Rationale |
|---|---|
fail_count >= 2 |
One failure could be infra noise |
pass_count >= 2 |
One pass could be a pre-fix lucky run |
0.05 < fail_rate < 0.95 |
Outside this band it is either reliably broken or reliably passing |
Step 1 — Collect workflow run IDs (last 7 days)
SINCE=$(date -u -v-7d +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || date -u -d '7 days ago' +%Y-%m-%dT%H:%M:%SZ)
gh api --paginate \
"repos/agent-substrate/substrate/actions/workflows/pr-workflow.yaml/runs?status=completed&per_page=100&created=>=$SINCE" \
--jq '.workflow_runs[] | {id: .id, conclusion: .conclusion, head_sha: .head_sha, created_at: .created_at}'
--paginate is required: a typical week has several hundred completed runs (600+ as of
August 2026), far more than one page of 100.
Collect all run IDs. Process both successful and failed runs — both contain test output.
Step 2 — Download and parse logs: two jobs, three lanes per run
For each run, you need logs from two jobs:
| Job name | Coverage |
|---|---|
run-tests |
Unit + integration tests (go test -race -v ./...) |
e2e-test, e2e-test (envoy, preview) |
E2E suite, both sandbox classes — two sequential steps in each job |
# List all jobs for a run
gh api "repos/agent-substrate/substrate/actions/runs/<RUN_ID>/jobs" \
--jq '.jobs[] | {id: .id, name: .name, conclusion: .conclusion}'
# Download log for a specific job
gh api "repos/agent-substrate/substrate/actions/jobs/<JOB_ID>/logs" > /tmp/job_<JOB_ID>.log
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 6d ago Changed c3159fbb7f59
- 14d ago Changed · +8 lines f671bc311625
- 2mo ago First seen · 339 lines · 102 tokens per session scan A 4d8244a1d969
detect-flaky-tests is a skill published in the GitHub repository agent-substrate/substrate (5,957 stars, last pushed today), licensed Apache-2.0. It adds 102 tokens to every session and 3,468 once invoked, about $0.0004 per session on Opus 5.5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.
Other skills, from other repositories
testing-patterns
Redirect — testing-patterns was split into 5 focused sub-skills. Use when looking for testing-patterns, writing tests, or test automation. Redirects to testing-unit, testing-e2e, testing-integration, testing-llm, or testing-perf.
loom-test-strategy
Test strategy guidance covering pyramid design, coverage, test categorization, flaky tests, infrastructure, and risk-based prioritization.
loom-testing
Test implementation across unit, integration, e2e, security, infrastructure, data pipeline, and ML domains.
langchain4j-testing-strategies
Provides unit test, integration test, and mock AI patterns for LangChain4j applications. Creates mock LLM responses, tests retrieval chains, validates RAG workflows, and implements Testcontainers-based integration tests for Java AI services. Use when unit testing AI services, integration testing LangChain4j…
testing-python
Stratégie de Tests Python 3.14+. Use when writing tests, reviewing test coverage, or setting up testing.
testing-symfony
Stratégie de Tests Symfony 8.1 / PHP 8.5. Use when writing tests, reviewing test coverage, or setting up testing.