NewThe Learning module is now liveSee what's new
Beginner
Incident Response and On-Call
30 min readChapter 4Core
1Incident response and on-callChapter 1

You're on call for a photo-sharing app, your phone buzzes, and uploads are failing. The interviewer asks what you do in the first 15 minutes. The answer is incident response, the agreed way a team handles an outage: someone declares an incident, a few people take named jobs, and the first goal is to stop users from being hurt, not to find out why.

The principle is three moves in order. Rate the incident by what users feel, name one incident commander (the single person who decides while others do the work), and undo the most likely recent change before the cause is understood. Reach for it when users are failing right now and more than one person is needed to help. Skip the ceremony for a contained problem one person can fix in minutes, and for a slow issue with no user impact today. A ticket is enough for both.

The example throughout is the photo-sharing app from SLIs, SLOs, and Error Budgets. It handles about 60M uploads a month, about 8k a minute at the evening peak, and its 99.9% upload SLO (service level objective) over a rolling 30 days allows 60k failed uploads.

This is a standard scenario question in site reliability engineering (SRE) and production-engineering loops, 5 to 15 minutes with follow-ups, and it shows up as the "how would you operate it?" segment of a design round. Mid-level candidates give the steps in a sensible order. Senior candidates defend mitigating before diagnosing with a count of users hurt, and say who decides, how people are told, and when the incident ends. The chapters follow the order those decisions arrive.

Your progressLogin to save progress
Spot an issue or have a suggestion?
Postmortems and Learning from Incidents  |

Discussion

Log in to join the discussion.