liveops-incident-response

Liveops Incident Response

Overview

Create a calm, evidence-based incident record and decision structure while preserving explicit human authority over production changes.

When to use

Use for outages, severe latency, failed deployments, exploit waves, economy abuse, data integrity risk, authentication failures, or player-impacting service degradation.

When NOT to use

Do not use for routine bug triage, speculative incidents without observable impact, or autonomous production commands.

Required inputs and context discovery

Collect incident start time, affected services and regions, player impact, dashboards, alerts, deployment history, build and configuration identity, security or privacy indicators, incident commander, communication owner, authorized runbooks, rollback state, and protected credentials.

Safety and risk level

High-risk production context. Read-only diagnosis may proceed, but service control, traffic changes, rollback, database writes, bans, credential rotation, and public communication require explicit authority for the exact action. Otherwise mark them BLOCKED.

Workflow

1. Establish incident command, severity, scope, communication channel, and decision log. Completion criterion: one incident commander and current impact statement are explicit.
2. Preserve a timestamped timeline of alerts, observations, hypotheses, and actions. Completion criterion: facts are separated from assumptions and every action names its actor.
3. Stabilize by proposing the smallest reversible mitigations from approved runbooks. Completion criterion: unapproved production actions remain BLOCKED with risk and owner.
4. Validate recovery against player-facing and system health indicators. Completion criterion: recovery is not declared from one metric or absence of alerts alone.
5. Hand off monitoring, follow-up defects, security review, and post-incident analysis. Completion criterion: owners, deadlines, evidence paths, and residual risks are recorded.