stop-chasing-the-optimizer-reduce-instead

Stop chasing the optimizer; reduce the repro instead

When to use

You have a hot function that passes on one toolchain (typically MSVC) but miscompiles or crashes under GCC / Clang at -O2 (or higher), and you're on attempt three of sprinkling anti-optimization decorations:

• volatile on a local / a parameter
• __attribute__((noinline)) on the callee
• __attribute__((optimize("O0"))) on the callee
• -fno-strict-aliasing on the TU
• asm volatile("" ::: "memory") barriers

Each one "almost works" — the symptom moves by a few bytes or into a different test case — but never fully goes away. That's the signal to stop.

Problem

Adding anti-optimization pragmas is a local fix applied to a global belief: "the optimizer is wrong." Three things go wrong in practice:

1. The decorations change the bug, not the cause. noinline in one spot shifts the inlining decision elsewhere; the miscompile now lives in a different frame. You spend another iteration finding it.
2. Some decorations are themselves traps. __attribute__((optimize("O0"))) on a single function inside an -O2 TU is known to produce ABI mismatches between the -O0 callee and the -O2 caller on GCC (frame pointer, red-zone, stack alignment). The "fix" introduces a new SIGSEGV on the first call.
3. You never learn whether it's a compiler bug, your UB, or your aliasing. All three present identically — "works on MSVC, breaks on GCC -O2" — and have very different fixes. Patching blindly leaves the root cause unknown, so the same bug returns the next time someone touches the file.

The heuristic: after two anti-optimization patches in a row have failed to fully fix the symptom, the next step is not a third patch. It's reduction.

Solution

Switch from "patch" mode to "reduce" mode:

1. Extract a standalone repro. Rip the suspect function out of the project into a single .cpp of ≤ 200 lines that links with nothing but libc. Must reproduce the divergence between MSVC and GCC/Clang -O2. If it doesn't reproduce standalone, the bug is in how the function is called, not the function itself — go up a frame.
2. Diff the codegen. g++ -O2 -S -masm=intel repro.cpp vs clang++ -O2 -S -masm=intel repro.cpp vs MSVC /FAs. Look for loads from offsets you never wrote, or stores that the compiler elided. This usually tells you within minutes whether it's UB (compiler is within its rights) or a real miscompile.
3. Shrink with creduce / cvise. Feed the standalone repro plus a predicate script (g++ -O2 x.cpp && ./a.out; [ $? -ne 0 ]) to cvise. 200 lines typically collapses to 20.
4. Decide once, fix once.
   • UB in your code → fix the UB (e.g. replace x >> 64 with a branch, use memcpy instead of pointer-punning, add an explicit bounds check). No volatile needed.
   • Real compiler bug → file upstream with the reduced repro, then put one narrow workaround (guarded by compiler-version macros) with a link to the bug report.
   • Aliasing assumption → use memcpy or __attribute__((may_alias)) at the type, not -fno-strict-aliasing on the whole TU.