failure-oriented-instrumentation

Failure-Oriented Instrumentation

Strategically instrument code to capture high-signal runtime data for debugging failures, focusing only on suspicious regions rather than comprehensive instrumentation.

Workflow

1. Analyze the Failure

Gather and analyze failure information:

• Error message and exception type
• Stack trace showing call chain
• Failure location (file, line, function)
• Reproduction steps or test case
• Expected vs actual behavior

Identify suspicious code regions:

• Functions in the stack trace
• Code paths leading to the failure
• Variables involved in the error
• Conditional branches that may affect the outcome

2. Determine Instrumentation Strategy

Based on the failure type, choose instrumentation targets:

For crashes/exceptions:

• Function entry/exit in stack trace
• Variable values before the crash
• Conditional branches leading to error path
• Exception handling blocks

For incorrect results:

• Variable values at key computation points
• Conditional branch decisions
• Loop iterations and state changes
• Function return values

For performance issues:

• Timing information for slow operations
• Loop iteration counts
• Resource allocation/deallocation
• Function call frequency

For intermittent failures:

• State variables that may cause non-determinism
• Thread/concurrency information
• External dependencies (I/O, network, time)
• Retry logic and error recovery paths

3. Select Instrumentation Patterns

Choose appropriate patterns based on language and context. See language-specific references:

• Python: references/python.md
• JavaScript/TypeScript: references/javascript.md
• Java: references/java.md
• C/C++: references/c-cpp.md

Common patterns:

• Function entry/exit logging
• Variable value tracking
• Conditional branch tracking
• Loop iteration monitoring
• Timing measurements
• Assertions for invariants