Cascading failure across multiple services: If the incident affects multiple services (Service A's failure causes Service B to degrade, which causes Service C to fail), prioritize the root service (Service A) during investigation. Do not attempt to fix downstream services until the root cause service is restored. The downstream failures will resolve automatically once the root cause is fixed.
Incident during another incident: If a second incident occurs while the first is still active, assess whether they are related (common cause, cascading effect) or independent. If related, treat them as a single incident with a broader scope. If independent, assign a separate incident commander and response team for the second incident.
Incident caused by a third-party dependency: If the root cause is a third-party service outage (payment provider, email service, CDN), the fix options are limited: enable circuit breakers to gracefully degrade, switch to a backup provider if available, or communicate the third-party dependency to users. The postmortem should evaluate whether the dependency can be made redundant or whether the degradation behavior can be improved.
Incident discovered during off-hours: If the incident is detected by automated alerting outside business hours, the on-call engineer handles Steps 1-4. If the severity is P3 and the impact is tolerable, it may be acceptable to acknowledge the incident, apply a temporary mitigation (increase timeouts, scale up), and defer full investigation to business hours. P1 and P2 incidents require immediate full response regardless of time.
Incident with no clear root cause: If after 2 hours of investigation (Step 2) no root cause is identified, the options are: (1) escalate to specialized engineering (database team, infrastructure team, vendor support), (2) apply broad mitigations (restart affected services, scale up, clear caches) and monitor for recurrence, (3) add instrumentation (more logging, more metrics) to capture data if the incident recurs. Document the unknown root cause in the postmortem with a plan for improving observability.
Incident affecting data integrity: If the incident caused incorrect data to be written (wrong calculations, duplicate records, corrupted references), the resolution in Step 4 must include a data repair plan in addition to the code fix. Identify the affected data scope (which records, which time window), write a repair script that corrects the data, test the script on a staging copy of the production database, and execute it as a tracked migration. Notify affected users if the data corruption was visible to them. The postmortem must include a data audit confirming all affected records were corrected.
Recurring incident (same failure mode, different trigger): If the postmortem in Step 5 reveals this incident is similar to a previous one (same service, same failure pattern, different root cause), the action items must address the systemic vulnerability, not just the latest trigger. For example, if the payments service has had three incidents from three different query bugs, the systemic fix is automated query performance testing in CI, not patching each individual query.
Incident requiring customer data access for diagnosis: If debugging requires querying production databases containing customer data, follow data access protocols: use read-only replicas when possible, log all queries executed during investigation, limit result sets to the minimum needed for diagnosis, and redact personally identifiable information in any screenshots or log excerpts shared in the incident channel. The postmortem should note what customer data was accessed and why it was necessary.
Incident with customer-visible data corruption: If users received incorrect data (wrong prices, wrong account balances, misattributed content), the resolution must include both a technical fix and a customer remediation plan. After deploying the fix in Step 4, generate a list of affected users and the specific data they saw. Work with customer support to send proactive notifications explaining the error, what was corrected, and any compensation offered. The postmortem in Step 5 must include the customer impact timeline and the remediation actions taken.