bisect-aware-instrumentation

Bisect-Aware Instrumentation

Overview

Instrument code to support efficient git bisect operations by producing deterministic pass/fail signals and concise runtime summaries. This skill helps create robust test scripts that work reliably with git bisect run, handling edge cases like flaky tests, build failures, and non-deterministic behavior.

Core Workflow

1. Understand the Regression

Before instrumenting, clarify:

• What behavior changed? (bug introduced, performance regression, test failure)
• What is the "good" commit? (known working state)
• What is the "bad" commit? (known broken state)
• How to reproduce the issue? (test command, manual steps)

2. Create Bisect Test Script

Generate a test script that returns proper exit codes for git bisect:

Exit Code Convention:

• 0: Good commit (test passes)
• 1-124, 126-127: Bad commit (test fails)
• 125: Skip commit (cannot test - build failure, missing dependencies)

Template:

#!/bin/bash
# bisect_test.sh - Test script for git bisect run

set -e  # Exit on error

# Build/setup phase
if ! make build 2>/dev/null; then
    echo "SKIP: Build failed"
    exit 125
fi

# Run test with timeout
timeout 30s ./run_test || TEST_RESULT=$?

# Interpret results
if [ $TEST_RESULT -eq 0 ]; then
    echo "GOOD: Test passed"
    exit 0
elif [ $TEST_RESULT -eq 124 ]; then
    echo "SKIP: Test timeout"
    exit 125
else
    echo "BAD: Test failed with code $TEST_RESULT"
    exit 1
fi

3. Add Determinism Safeguards

Handle non-deterministic behavior:

Retry Logic for Flaky Tests:

# Run test multiple times to confirm
PASS_COUNT=0
for i in {1..3}; do
    if ./run_test; then
        ((PASS_COUNT++))
    fi
done

if [ $PASS_COUNT -eq 3 ]; then
    echo "GOOD: All 3 runs passed"
    exit 0
elif [ $PASS_COUNT -eq 0 ]; then
    echo "BAD: All 3 runs failed"
    exit 1
else
    echo "SKIP: Flaky test ($PASS_COUNT/3 passed)"
    exit 125
fi

Environment Isolation:

# Clean state before each test
rm -rf /tmp/test_cache
export RANDOM_SEED=42
export TZ=UTC