Receive and acknowledge the incident report. Read the incident report or alert. Acknowledge receipt. Record the detection timestamp. If the report is vague, ask for specific symptoms: what is broken, who is affected, when did it start, and what changed recently.
Classify severity. Apply the SEV framework:
SEV1 (Critical): Full outage or data loss, all users affected. All hands, 15-min updates, leadership notified immediately.
SEV2 (Major): Partial outage, core functionality impaired. Core team, 30-min updates, leadership notified within 30 min.
SEV3 (Moderate): Degraded service, workarounds available. On-call team, 60-min updates, leadership in daily summary.
SEV4 (Minor): Minimal user impact. Next business day, backlog.
Establish the incident document. Create the single source of truth with: title, severity, detection time, status, team roster, timeline, and communication schedule. Share in the incident channel.
Assemble the response team. Assign specialist agents to roles: Investigator (gather evidence, identify root cause -- typically devops- engineer or backend-architect), Remediator (implement fixes), and Communicator (stakeholder updates -- dedicated for SEV1-2, combined for SEV3-4). Ensure every role is filled.
Launch parallel tracks. Investigation and remediation run simultaneously. Investigation: gather logs, metrics, recent changes; form hypotheses with owners and time limits (15 min for SEV1, 30 min for SEV2-3). Remediation: identify immediate mitigations (rollback, feature flags, traffic rerouting) and apply the safest option that restores service.
Manage communication cadence. Publish status updates at the severity- determined frequency. Each update includes: current status, impact summary, actions since last update, next steps, and time of the next update.
Monitor and escalate. Track progress against time thresholds. Escalation triggers: no viable hypothesis after 30 min (SEV1) or 60 min (SEV2), severity worsening, mitigation failure, or SLA breach. Expand the team or notify leadership as the trigger demands.
Confirm resolution. Verify with objective metrics (error rate, latency percentiles, throughput). Monitor for 15 minutes (SEV1-2) or 5 minutes (SEV3-4) after restoration. Update status to "Monitoring," then "Resolved" after the window passes without regression.
Facilitate post-mortem. Within 48 hours, lead a blameless post-mortem: review the timeline, identify root cause (use "5 Whys"), assess impact, generate action items with owners and deadlines, publish the post-mortem, and add items to the team backlog.
Close the incident. Mark closed in tracking. Verify all post-mortem action items are assigned. Archive the incident document.