test-suite-generator

test-suite-generator — EARS spec → full test pyramid

You are invoked to turn an upstream acceptance-spec output (a specs/<feature>/ directory containing spec.md and done_when.yaml) into the actual test files that an independent agent can execute to verify implementation.

Say once at the start, then start working:

"I'm using the test-suite-generator skill. I'll walk five sub-steps (existence → unit → integration → e2e → mutation), one batch at a time. Each generated test traces back to a REQ-ID via based_on:."

Do not narrate further — just walk the sub-steps.

Iron rules (re-read before every sub-step)

1. Verifiable beats judgeable. Programmatic checks (assertion-based tests, AST inspection, mutation kill rate) beat LLM-as-judge on speed, cost, consistency. Per HTML v2 §3 principle I (and §3.5 corollary): most claims that feel like they need an LLM judge can be re-designed as programmatic checks — "README quickstart works" → really run it; "agent can call API from docs alone" → spin a clean session; "types are correct" → run tsc/mypy. The ~10% genuinely-unautomatable cases are routed to /pm-reviewer's requires_human_verification verdict, not to a fake LLM rubric here. This skill does NOT emit fitness rubrics in v1.0.0 (former 4-F was retired).
2. One batch at a time. Generate → write to disk → run → iterate. Then proceed. Never generate all six batches in one pass — empirically, LLM test generation quality collapses past a few dozen tests in a single prompt. The doc that motivates this skill names this as Pitfall #2; do not be the next person to step on it.
3. PBT property type is recovered from the test name. Under schema v1 (Appendix C of done-when-pipeline.md), every leaf entry under behavior.*.property_based / behavior.*.example_based / e2e_tests is a bare string — there is no property_type: sub-field on the entry. Parse the test name suffix to infer the property archetype (one of: invariant / idempotent / reversible / boundary / monotonic / state_machine). E.g. test_cancel_is_idempotent → idempotent pattern. Read references/pbt-property-types.md to know which pattern to emit. If a property-based name does not encode a recognisable archetype, the requirement is not a good PBT candidate — emit an example-based test instead and tell the user the name should be revised upstream.
4. No mocks for integration tests. Use testcontainers (or equivalent) to spin up real Postgres / Redis / Kafka / etc. Mocks hide interface drift, which is exactly the failure mode that catches AI-generated code most often. If the language lacks testcontainers, fall back to docker-compose; do not fall back to mocks.
5. Mutation testing is mandatory, not optional. A done_when that only enforces coverage ≥ 80% will incentivize the implementer to write assert True to inflate the number. Mutation kill rate ≥ 70% is the gate that closes that loop. Always emit a mutation.config file (4-E).
6. Every generated test carries based_on:. Either as a comment header in the test file (# based_on: REQ-001, REQ-003), or as part of the test name when the language convention favors it. Tests without traceability cannot be culled when a REQ is dropped, and cannot be explained when they fail.
7. No inventing requirements. If a property occurs to you that isn't in any REQ, do not add a test for it. Either push back upstream ("REQ-NNN seems to imply X — should that be a separate REQ?") or skip it. Tests must derive from the spec, not from your own intuition about what a good system does.
8. Pyramid ratio rebalanced for AI-coding. Per the source doc, AI-written code is more likely to fail at module boundaries (interface mismatches, protocol confusions). Target ~50% unit / ~35% integration / ~15% e2e — not the traditional 70/20/10. Push slightly more weight into integration than you might be used to.
9. Verbatim test names from done_when.yaml. The string the framework records as the test identifier must equal the YAML entry character-for-character (prefix, underscores, suffix). Python def test_* is naturally fine; TS/JS test('…') / it('…') is the trap because it accepts arbitrary strings. Human-readable descriptions go in comments, describe() blocks, or annotations — never in the test() first argument. See §4-D for the full rule + forbidden-pattern examples; this exists because downstream Step 5-6 review skills (/qa-reviewer, /pm-reviewer) grep contract names against produced files, and any paraphrase breaks traceability.
10. Existence script is fail-fast (set -e), not count-all. The script in §4-A MUST exit on the first failing check. Wrapping checks in if … then PASS=… else FAIL=… fi defeats errexit — that pattern is forbidden. See §4-A for the required idiom and the explicit list of forbidden patterns.
11. Test-only API endpoints must be registered as existence claims. Any endpoint the test suite calls but that is not part of the user-facing contract (/api/_test/*, /api/_internal/*, hidden test-hook query/body fields like _force_surface_outcomes) MUST be added to done_when.yaml.existence: as a route: entry — otherwise the test suite implicitly demands a bigger contract than the contract makes explicit (spec drift). If done_when.yaml was emitted by /acceptance-spec and does not list these endpoints, either (a) push back upstream ("the contract is missing test-hook endpoints X, Y, Z — regenerate via /acceptance-spec to include them") or (b) add the entries here and note the augmentation in the manifest you emit at the end of the skill. Never ship a test suite that calls undeclared endpoints. See §4-A for the existence list and §4-C for the test-hook conventions.
12. Self-report version comes from this SKILL.md frontmatter, not a hardcoded literal. Every emitted file's "Generated by test-suite-generator/X.Y.Z" header MUST be filled in by reading the version: value from this SKILL.md's YAML frontmatter at generation time. Do NOT paste a version literal lifted from references/sub-modules/*.md templates (those use a <skill-version> placeholder for this exact reason). The downstream audit signal is: artifact self-report mismatch with plugin.json ⇒ skill source bug. See §4-B "Version-string substitution rule" for the canonical reminder.