core-dumps

Core Dumps

Purpose

Guide agents through enabling, collecting, and analysing core dumps for post-mortem crash investigation without rerunning the buggy program.

Triggers

• "My program crashed in production — how do I analyse the core?"
• "How do I enable core dumps on Linux?"
• "I have a core file but no symbols / source"
• "How do I use debuginfod to get symbols for a core?"
• "coredumpctl show me the crash"

Workflow

1. Enable core dumps (Linux)

# Per-session (lost on logout)
ulimit -c unlimited

# Persistent (add to /etc/security/limits.conf)
*   soft   core   unlimited
*   hard   core   unlimited

# Check current limit
ulimit -c

# Set core pattern (where and how cores are named)
# Default: 'core' in CWD — often not useful
sudo sysctl -w kernel.core_pattern=/tmp/core-%e-%p-%t
# %e = executable, %p = PID, %t = timestamp

# Persistent (add to /etc/sysctl.d/99-core.conf)
kernel.core_pattern=/tmp/core-%e-%p-%t
kernel.core_uses_pid=1

2. systemd/coredumpctl (modern Linux)

If systemd manages core dumps (common on Ubuntu 20+, Fedora, Arch):

# List recent crashes
coredumpctl list

# Show details of the latest crash
coredumpctl info

# Load latest crash in GDB
coredumpctl gdb

# Load specific PID crash
coredumpctl gdb 12345

# Export core file
coredumpctl dump -o myapp.core PID

Core storage location: /var/lib/systemd/coredump/.

3. Enable core dumps (macOS)

# macOS uses /cores by default (must be root-writable)
ulimit -c unlimited

# Check
ls /cores/

# launchd-launched services: set in plist
# <key>HardResourceLimits</key>
# <dict><key>Core</key><integer>9223372036854775807</integer></dict>

4. Analyse a core with GDB

# Load binary and core
gdb ./prog core.12345

# If the binary was stripped, provide the unstripped copy
gdb ./prog-with-symbols core.12345

# Essential first commands
(gdb) bt                    # call stack
(gdb) bt full               # stack + locals
(gdb) info registers        # CPU state at crash
(gdb) frame 2               # jump to interesting frame
(gdb) info locals           # local variables in frame
(gdb) print ptr             # inspect a pointer

# All threads (multi-threaded crash)
(gdb) thread apply all bt full