debug-order-sensitive-pytest

Debug Order-Sensitive Pytest Failures

Find and fix tests that pass on their own but fail when pytest runs in a different order. The suite is green again only after every leaked-global-state polluter is fixed and both the shuffled and the normal runs pass.

Quick quality checklist

• Repro order is pinned with a fixed seed and printed so any failure is reproducible
• Always bisect with the failing target test appended to every slice (a failing window can contain multiple polluters)
• Instrumented runs are only hints — always confirm on clean runs
• No private data, paths, or project-specific test names in this skill

When to use

• A test fails only in the full suite but passes standalone with pytest <file> or pytest <nodeid>
• A CI job that randomizes test order (pytest-randomly, custom shuffle) is flaky
• You suspect a test mutates sys.modules, reloads a module, or mutates a shared module-level singleton without restoring it

Required tools / APIs

• Python 3.8+ and pytest (no external API)
• Optional: a shuffling runner (pytest-randomly, or a local seeded-shuffle plugin, described below)

Steps

1. Reproduce with a fixed-seed shuffle

Pin the order so every bisection uses the exact same failing sequence. Use pytest-randomly:

pip install pytest-randomly
python -m pytest -q --randomly-seed=42 -p no:cacheprovider --tb=no | tee order.log

Or, for a repo that already has a seeded-shuffle runner (e.g. tests/run_order_report.py --seed 42), use it. If neither exists, add a tiny local plugin that shuffles collection with a fixed seed:

# shuffle42.py  (load with: python -m pytest -p shuffle42 ...)
import random

def pytest_collection_modifyitems(items):
    random.Random(42).shuffle(items)

Record the failing nodeids from the log. The seed must be printed/stored so the order is reproducible.

2. Dump the shuffled order to a file

Every bisection slice must come from the same shuffled order. The shuffle must be re-applied deterministically to the same collected list, so dump the exact order once: