test-skip-discipline

Test-Skip Discipline

The distinguishing rule: skip if the test cannot execute (missing credentials, wrong platform, feature flag off); fail if the test executes but the system is broken.

Every test.skip(...) / pytest.skip(...) / #[ignore] is one of two things. One is a legitimate "we cannot run this test here" gate. The other is a silent mask over a broken system — the test ran, the upstream was degraded, the skip absorbed the signal, and the runner reports green for an unknown period. The second class is BLOCKED.

The Problem (Grounded In The STP Incident)

Downstream STP's Playwright spec tests/e2e/stp-chatbot-scope.spec.ts contained test.skip(chatStatus >= 500, ...) which was the only reason the spec wasn't red despite POST /api/chat returning 503 on every run. The skip masked the Bedrock gap for an unknown period; nobody noticed until a human filed issue #52.

This is not Bedrock-specific. Any test that skips on the system-under-test's runtime behaviour silently absorbs every non-functional AI path — OpenAI auth failures, Anthropic rate-limits, Google SSL handshakes, local Ollama not running, any upstream degradation. In a multi-downstream template ecosystem, every USE repo that copies the pattern inherits the masking.

Origin: workspaces/use-feedback-triage/journal/0003-GAP-test-skip-masks-ai-failures.md.

Acceptable Skip Patterns

Skip gates that check whether the test can execute at all — before the system-under-test is invoked. The check is on the environment, not on the response.

1. credential_absent — Tier 2 Integration Tests Requiring Real Cloud Credentials

// DO — skip when the credential isn't present; the test cannot execute without it
test("bedrock claude returns a completion", async ({ page }) => {
  test.skip(
    !process.env.AWS_BEARER_TOKEN_BEDROCK,
    "requires AWS_BEARER_TOKEN_BEDROCK",
  );
  // ... real upstream call; 5xx here is a FAILURE, not a skip
});

# DO — pytest credential gate
@pytest.mark.skipif(
    not os.getenv("AWS_BEARER_TOKEN_BEDROCK"),
    reason="requires AWS_BEARER_TOKEN_BEDROCK for live Bedrock call",
)
def test_bedrock_claude_completion():
    ...