hydraflow: Skill for Codex

Test Audit

Run a comprehensive test quality audit across the entire repo. Analyzes test naming, structure, factory usage, anti-patterns, coverage gaps, and flaky patterns. Creates GitHub issues for findings so HydraFlow can process them.

Instructions

1. Resolve configuration before doing anything else:

   • Run echo "$HYDRAFLOW_GITHUB_REPO" — if set, use it as the target repo (e.g., owner/repo). If empty, run git remote get-url origin and extract the owner/repo slug (strip https://github.com/ prefix and .git suffix).
   • Run echo "$HYDRAFLOW_GITHUB_ASSIGNEE" — if set, use it as the issue assignee. If empty, extract the owner from the repo slug (the part before /).
   • Use hydraflow-plan as the label for created issues.
   • Store resolved values as $REPO, $ASSIGNEE, $LABEL.

2. Discover project structure:

   • Use Glob to find all test files: **/test_*.py, **/tests/conftest.py, **/tests/helpers.py
   • Exclude .venv/, venv/, __pycache__/, node_modules/
   • Also find all UI test files: ui/src/**/*.test.jsx, ui/src/**/*.test.js
   • Count total test files and identify the test helper infrastructure

3. Launch agents in parallel using Task with run_in_background: true and subagent_type: "general-purpose":

   • Agent 1: Test naming & structure — Checks naming conventions, 3As structure, single responsibility, and organization.
   • Agent 2: Anti-patterns & flaky tests — Detects over-mocking, weak assertions, flaky patterns, and test isolation issues.
   • Agent 3: Factory/fixture gaps & coverage — Finds missing factories, repeated setup, coverage gaps, and missing edge case tests.

4. Wait for all agents to complete.

5. After all finish, run gh issue list --repo $REPO --label $LABEL --state open --search "test quality" --limit 200 to show the user a final summary of all issues created.

Agent 1: Test Naming & Structure

You are a test quality auditor focused on naming and structure for the project at {repo_root}.

## Configuration
- GitHub repo: {REPO}
- Assignee: {ASSIGNEE}
- Label: {LABEL}

## Steps

### Phase 1: Read All Test Files
1. Use Glob to find all test files: tests/test_*.py, ui/src/**/*.test.jsx
2. Read each test file

### Phase 2: Audit Test Naming
3. Check every test function/method name against the convention:
   - **Pattern**: `test_<method/feature>_<scenario>[_<expected_result>]`
   - **Flag**: names < 3 words (e.g., `test_init`, `test_run`)
   - **Flag**: generic names (test_1, test_something, test_basic)
   - **Flag**: redundant "test" in name (test_user_test)
   - **Flag**: names that don't describe what's being tested
   For each violation, note: file path, line number, current name, suggested better name

### Phase 3: Audit 3As Structure (Arrange-Act-Assert)
4. For each test function, check:
   - Is there clear separation of setup, execution, and verification?
   - Is all arrange code before the act?
   - Are all assertions after the act?
   - Are phases mixed? (e.g., assertions interleaved with setup)
   - Does setup dominate the test? (> 60% of lines are setup — push into factories/fixtures)

### Phase 4: Audit Single Responsibility
5. For each test, count assertions:
   - Flag tests with > 3 assertions testing **different** attributes (related assertions on the same result are OK)
   - Suggest splitting into focused tests
   - Note tests with zero assertions (test does nothing useful)

### Phase 5: Audit Test Organization
6. Check file-level organization:
   - Are test classes used to group related tests?
   - Are tests organized to mirror source file structure?
   - Are there test files > 500 lines that should be split?
   - Are there test files with < 3 tests (too granular)?

### Phase 6: Create GitHub Issues
7. Check for duplicate GH issues first:
   gh issue list --repo {REPO} --label {LABEL} --state open --search "<key terms>"
8. Create GH issues for NEW findings only, grouped by theme:
   gh issue create --repo {REPO} --assignee {ASSIGNEE} --label {LABEL} --title "Test Quality: <theme>" --body "<details>"

## Issue Body Format
```markdown