NewThe Learning module is now liveSee what's new
Beginner
Postmortems and Learning from Incidents
30 min readChapter 5Core
1Postmortems and learning from incidentsChapter 1

"How do you make sure the same outage doesn't happen twice?" That question closes most incident-management interview rounds, and a postmortem is the answer. It is a written review of an incident that records what happened, how users were hurt, which conditions let it happen and made it last, and which changes will follow. A blameless postmortem does this without naming a person as the cause. Our example is a photo-sharing app whose uploads failed for 27 minutes on a Tuesday evening after a deploy added a slow database query.

Treat a mistake as a symptom. Ask what made it easy to make and costly once it happened, find every condition that let the outage start or last, and give each one a change with an owner and a deadline, tracked until it is done. Write a full postmortem when an incident burns more than a fifth of the error budget, loses data, needs manual intervention, or a stakeholder asks for one. Skip it for a blip that burned almost none of the budget or a known bug already fixed, since the incident log covers those.

Expect 5 to 10 minutes on this right after the incident response question. Mid-level candidates describe a timeline, the impact, the causes, and action items with owners. Senior candidates explain why "human error" is not a root cause, why one chain of whys misses most of the conditions, and how items are tracked until a repeat would be noticed. The chapters follow the order of those decisions.

Your progressLogin to save progress
Spot an issue or have a suggestion?
Overload and Cascading Failures  |

Discussion

Log in to join the discussion.