1. Safe releases and rollback
The interviewer says most outages are caused by changes and asks how you would make deploys safe. For the photo-sharing upload service, a safe release ships a change, a new build or a config value, to a small share of users first. It compares that share automatically against everyone else and can be undone in minutes. A bad change then harms few users for a short time, instead of every user until someone notices.
The basic principle is blast radius: the share of users a bad change reaches, times how long it takes to notice and undo it. Staging shrinks the share, while automatic comparison and fast rollback shrink the time. Use staged releases for every production change, including config values and feature flags (settings that turn code on for some users). Skip them for an internal tool with a handful of users, or for a service too quiet to show failures in a small slice, such as a nightly batch job, where tests and a quick rollback or rerun are enough.
This topic carries a lot of weight in site reliability engineering (SRE) and platform loops, and gets a few minutes at the end of a backend design round, so aim to say the whole answer in 2 minutes. Mid-level candidates name a canary (a small slice of the fleet that runs the new version first), flags, and rollback with one reason each. Senior candidates also put a number on blast radius, defend how a canary is judged, and handle config and schema changes.
The chapters follow the order you would give the answer in. SLIs, SLOs, and Error Budgets covers the 99.9% upload service level objective (SLO) and its error budget, the failure allowance that target implies. The release gate in the second-to-last chapter uses that budget.
Discussion
Upgrade to premium to read and join the discussion.