incident-response

Incident Response

A practical IR runbook for small teams and solo operators. Built around the SANS PICERL phases: Preparation → Identification → Containment → Eradication → Recovery → Lessons Learned.

When to invoke

You are in incident territory if any of these are true:

• A site you operate is defaced, redirecting, or serving content you did not publish
• A webshell or backdoor file has been found
• An admin/root account appeared that nobody on the team created
• A hosting provider, registrar, or upstream sent an abuse notice
• A credential you control turned up in a public dump
• Traffic is being dropped from your IP by major email/DNS providers (sign of being on a blocklist)

If you only suspect something is wrong, treat it as a real incident until you have evidence otherwise. False positives are cheap; missed compromises are expensive.

Phase 1 — Preparation (before anything happens)

Have these ready before the first incident:

• Out-of-band contact channel (Signal/Telegram/phone) — your primary work tool may itself be compromised
• Backup of last-known-good for every production system, verified restorable
• Inventory of who has access to what (hosting, DNS, registrar, repo, secrets store)
• A pre-written holding statement for customers ("we are investigating an issue affecting X, more soon")
• A way to reach your registrar fast — domain hijacks are catastrophic and the registrar is often slowest in real time

Phase 2 — Identification

Goal: confirm a real incident and scope it.

# Webserver: who is hitting unusual paths?
sudo tail -n 10000 /var/log/nginx/access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -30

# Auth log: failed + successful root/SSH activity
sudo grep -E 'Failed password|Accepted' /var/log/auth.log | tail -100

# Currently logged-in users + open sessions
who -a
last -n 30

# Listening processes — anything new?
sudo ss -tlnp

# Crontabs at every level
for u in $(cut -d: -f1 /etc/passwd); do sudo crontab -u "$u" -l 2>/dev/null | sed "s|^|[$u] |"; done
sudo ls -la /etc/cron.* /var/spool/cron/

# Recently modified files in sensitive trees
sudo find /var/www /etc /usr/local/bin -mtime -30 -type f 2>/dev/null