NewThe Learning module is now liveSee what's new
Beginner
Alerting That Pages for the Right Reasons
30 min readChapter 2Core
1Alerting that pages for the right reasonsChapter 1

On-call for the photo-sharing app's upload service gets 40 pages a week, and most of them clear on their own. The interviewer asks how you would redesign alerting. An alert is an automatic message that asks a person to act. A page interrupts someone right now, a ticket waits in a queue for working hours, and a dashboard panel asks for nothing until someone looks.

The basic idea is to page only when users are hurt fast enough that waiting would cost error budget and a person can act. Burn rate, how quickly the budget is being spent, turns "fast enough" into a number. Reach for it when pages outnumber what people can act on, or when you're handed an SLO (service level objective) and asked how to alert on it. Skip it for a service with no SLO yet (define one first) or a once-a-day batch job (alert when the run misses its deadline).

The upload SLO is 99.9% of uploads succeeding over a rolling 30 days, a budget of 60k failed uploads, defined in SLIs, SLOs, and Error Budgets. In SRE (site reliability engineering) interviews it follows the SLO question, and the whole answer fits in under 2 minutes. Mid-level candidates sort alerts and explain burn rate. Senior candidates also design the windows and tiers and score them. The chapters follow the order those questions arrive.

Your progressLogin to save progress
Spot an issue or have a suggestion?
Debugging Production with Telemetry  |

Discussion

Log in to join the discussion.