rr-debug

rr Reverse Debugging Workflow

Use this skill when the user wants to debug a crash, segfault, test failure, or any bug where the cause isn't obvious from reading the code.

Prerequisites

• rr and gdb must be installed and on PATH
• Linux x86-64 only
• perf_event_paranoid must be <= 1: sysctl kernel.perf_event_paranoid
• The program should be compiled with debug symbols (-g, preferably -O0 or -Og)

Workflow

1. Record the Failure

Generate a random trace directory name under the project directory to keep traces local and avoid cluttering ~/.local/share/rr/:

rr_record(command=["./failing_test"], trace_dir="<project>/rr-trace-<random>")

Pass working_directory if the command needs to run from a specific directory. Pass env for additional environment variables (e.g., {"MALLOC_CHECK_": "3"}).

2. Identify the Right Process (Multi-Process Traces)

If the recorded command spawns child processes (test harnesses, build systems, etc.), the default replay targets the root process, which is usually NOT what you want.

rr_ps(trace_dir="<trace>")

Look for the actual program binary in the command column. Use exit codes to identify the crashing process: negative codes indicate signals (-11 = SIGSEGV, -6 = SIGABRT).

3. Start Replay

rr_replay_start(trace_dir="<trace>", pid=<pid>)

Always pass pid for multi-process recordings.

4. Navigate to the Bug

• For crashes/signals: rr_continue() stops automatically at the signal
• For logic bugs: set breakpoints first with rr_breakpoint_set(), then rr_continue()
• For assertion failures: rr_breakpoint_set("abort") or rr_breakpoint_set("__assert_fail"), then rr_continue()

5. Examine State

• rr_backtrace() to see the call stack
• rr_locals() to see local variables
• rr_evaluate("expr") to evaluate expressions
• rr_select_frame(N) to inspect caller frames without stepping
• rr_source_lines() to see source code at the current position