$ tail -f runbook.log
What happens from the moment something goes wrong, written as a log. Timings are relative, and the italic lines explain each step for humans.
## stage: detect
T+00:00ALERTcheck failed: homepage returned an errorEither monitoring notices, or you email us. Both start the same process.
T+00:01INFOticket logged with time and reporterEverything gets a record so nothing relies on memory.
## stage: triage
T+triageINFOhuman acknowledges, confirms scopeIs it everyone or one user? One page or all of them?
T+triageSEVseverity set: down | degraded | annoying | requestSeverity decides order of work, and is shared with you.
T+triageNOTIFYyou receive: what we know, what we are doing next
## stage: contain
T+containACTroll back last change OR restore from backup if neededGetting you working again comes before finding the perfect fix.
T+containVERIFYservice checks pass
## stage: repair
T+repairACTroot cause identified, fix written, tested off liveFixes are tried somewhere safe before they touch your real site.
T+repairDEPLOYfix released, monitored closely
## stage: close
T+closeREPORTwrite-up sent: cause, change, follow-upsShort and readable. If there is a lesson, it is written down.
T+closeOKticket closed with your agreement
## routine (no incident)
weeklyCRONupdates reviewed, low-risk ones applied
monthlyCRONbackup restore tested, health note sent
> something broken now? /ticket