worktree-historical-test-replay-missing-dirs

Worktree historical test replay — missing dirs

Problem

You're replaying historical bugs by checking out each pre-fix SHA in a temp worktree and running the project's canonical test command. At HEAD the command works (e.g. 159 tests collect). At older SHAs you get:

ERROR: file or directory not found: scripts/overnight/tests/

no tests ran in 0.01s

Exit code 4, duration 1-2 seconds. Pytest aborted before collection because one of the test directories you passed didn't exist yet at that commit.

This silently corrupts a historical-replay audit: you can't tell apart "the suite passed and missed the bug" from "the suite never ran." Both look like pytest failures from the outside.

Context / Trigger Conditions

• You're running historical incident replay, mutation testing, git bisect, or any other "check out an old SHA and run the suite" workflow
• The project's test layout grew over time (test dirs added, removed, reorganized in subsequent commits)
• The test command embeds explicit directory paths: pytest scripts/overnight/tests/ <analytics_pkg>/tests/ ...
• Pytest exits 4 (USAGE_ERROR) in <5 seconds with no test execution
• The SAME test command works at HEAD but fails at pre-fix SHAs older than the test layout's current shape

Solution

Before invoking the test command at each SHA, filter the test directory list to only those that exist in the worktree at that SHA:

# Inside the worktree at the target SHA:
TEST_DIRS=()
for d in scripts/overnight/tests <analytics_pkg>/tests <analytics_pkg>/cloudrun/<dashboard_app>/tests; do
  [ -d "$d" ] && TEST_DIRS+=("$d")
done

if [ ${#TEST_DIRS[@]} -eq 0 ]; then
  # No test dirs existed at this SHA — classify as 'unrunnable', skip.
  echo "STATUS=unrunnable_no_test_dirs_at_sha"
  exit 0
fi

PYTHONPATH=. pytest "${TEST_DIRS[@]}" --tb=no -q

In Python:

test_dirs_at_head = ["scripts/overnight/tests", "<analytics_pkg>/tests"]
existing = [d for d in test_dirs_at_head if (worktree_path / d).is_dir()]
if not existing:
    return {"status": "unrunnable", "reason": "no test dirs existed at this SHA"}
subprocess.run(["pytest", *existing, "--tb=no", "-q"], cwd=worktree_path)