Reliability: SLOs, incidents and postmortems

Every service fails sometimes. The question is not how to reach 100% uptime (impossible and very expensive) but how reliable you need to be, how you measure it, and how you respond when it breaks. Site Reliability Engineering (SRE) is the practice, started at Google, of treating operations as an engineering problem with numbers and habits.

This lesson covers the vocabulary every DevOps or SRE interview uses: SLI, SLO, SLA, error budgets, incident response and blameless postmortems.

Three short words that people confuse:

The SLI is the thermometer, the SLO is the temperature you aim for, and the SLA is the promise you make to a customer. Your SLO should be stricter than your SLA so you notice problems before you owe anyone money.

A good SLI measures what users feel: successful requests and fast enough responses, not CPU. A simple formula is: SLI = good events / valid events.

Reliability targets are often described as 'nines'. Each extra nine is about ten times stricter and costs much more. Here is how much downtime each allows in a 30-day month (30 days is 43,200 minutes):

For the Brew & Bean ordering app, 99.99% would be silly: the shop is open 14 hours a day and a five-minute glitch at 3 am hurts no one. 99.5% might be right. Ask: what do users really need, and what would it cost to do better? A target nobody can meet leads to constant firefighting; a target that is too loose lets users suffer.

If your SLO is 99.9%, you are allowed to be bad 0.1% of the time. That allowance is the error budget. In a 30-day month with 1,000,000 requests, a 99.9% SLO allows 1,000 failed requests.

The budget turns an argument into arithmetic. Product teams want to ship features fast; operations wants stability. With an error budget the rule is simple: while there is budget left, release and take risks. When the budget is spent, slow down and spend time on reliability until it recovers.

A worked example: your SLO is 99.9% of requests succeed. This month you served 2,000,000 requests and 1,400 failed. The budget is 0.1% of 2,000,000, which is 2,000 failures, so you have used 70% of it. If one bad deployment can burn 700 failures, you can afford one more risky release, not two.

A simple alert on 'errors above 5%' fires on blips and misses slow leaks. Better is burn rate: how fast you are spending the budget. A burn rate of 1 means you will use exactly the whole budget by the end of the window. A burn rate of 10 means you will use it in a tenth of the time. Typical practice is a fast alert (a high burn rate over a short window, such as the last hour, which pages someone) and a slow one (a low burn rate over several hours or days, which opens a ticket). The SRE Workbook, published free online by Google, explains the exact multi-window recipes.

An incident is an unplanned event that harms users. The goals of response are, in order: stop the harm, then understand the cause, then prevent a repeat. Resist the urge to find the root cause first. If a bad release caused errors, rolling back now is better than reading code for an hour.

A postmortem (also called a retrospective or incident review) is a written account of an incident. Its purpose is learning, not punishment. 'Blameless' means we assume people acted sensibly with the information they had, and we ask what in the system allowed a human mistake to cause harm. If people fear blame, they hide mistakes and the same problem returns.

Compare two sentences. Blaming: 'Ravi deployed without testing.' Blameless: 'The pipeline allowed a deploy that skipped the integration tests, and there was no automatic rollback.' The second leads to fixes.

# Postmortem: Orders failing for 38 minutes (example format)

Date: 2026-03-14        Severity: SEV2        Author: [name]
Status: Complete        Time to detect: 4 min   Time to mitigate: 38 min

## Summary
One or two sentences: what broke, who was affected and for how long.

## Impact
- 38 minutes of failed order requests (about 62% of attempts)
- Estimated 140 orders not placed

## Timeline (IST)
- 18:02  Release v1.8.0 deployed
- 18:06  Alert HighErrorRate fired; on-call acknowledged
- 18:15  Database connection errors found in logs
- 18:40  Rolled back to v1.7.9; errors stopped

## Root causes and contributing factors
- v1.8.0 opened a new database connection per request, exhausting the pool
- Load tests did not run in the pipeline
- Rollback needed manual steps

## What went well / what went badly / where we were lucky
- ...

## Action items (each with an owner and a due date)
- [ ] Add connection pool limit test to CI - owner, date
- [ ] One-command rollback script - owner, date

This is a template you can reuse. The action items are the most important part: a postmortem without owned, dated actions is just a story.

SRE teams measure toil: repetitive, manual work that grows with the service and has no lasting value, such as restarting a service by hand every week. The goal is to automate toil away. A common guideline is to keep toil below about half of an engineer's time so the rest goes into engineering improvements.

On-call means being the person who responds when an alert fires. Healthy on-call has a rota (people take turns), clear runbooks, alerts that are worth waking for, and time off after a bad night. If on-call is miserable, fix the alerts and the system, not the people.

Most incidents are caused by changes, so the safest reliability habit is to make changes small and reversible: deploy often in small pieces; use canary releases (send 1% to 5% of traffic to the new version first and watch the metrics); use feature flags to switch features off without a deploy; keep a one-command rollback; and test restores of your backups, because a backup you have never restored is a hope, not a plan. *Observability* in the previous lesson gives you the signals to judge a canary.

Be ready to answer: What is the difference between SLI, SLO and SLA? What is an error budget and how do you use it? Walk me through an incident you handled (or how you would). What makes a postmortem blameless? How would you reduce alert fatigue? A strong answer to 'a deployment broke production' is: mitigate first by rolling back, communicate, then investigate, then write a blameless postmortem with owned action items.

You are an experienced SRE helping me write a blameless postmortem.

Here are my raw notes from an incident (timeline, commands, chat messages):

<notes>
[paste your notes here]
</notes>

Produce a postmortem using these sections: Summary, Impact, Timeline, Root causes and contributing factors, What went well, What went badly, Action items. Use blameless language (describe systems and decisions, not people's faults). Only use facts from my notes. If something important is missing, list it as questions at the end instead of inventing it.

What to expect: A good draft sticks to your facts, uses neutral wording, and ends with questions about gaps. Review every statement before sharing it.