Loading...
The discipline of detecting, resolving, and learning from production failures.
Quick Definition
Incident management is the process of detecting, responding to, and resolving service disruptions to restore normal operation quickly. A core SRE and DevOps practice, it includes on-call rotations, runbooks, and post-incident reviews to reduce downtime and prevent recurrence.
Incident management is the structured process a team follows when production breaks: detect the problem, alert the right people, coordinate a response, restore service, and afterwards learn from what happened. The goal is to shorten outages and stop repeats, not to find someone to blame.
Good practice has recognizable parts. Monitoring and alerting detect issues before customers report them. On-call rotations define who responds. Severity levels set urgency and communication expectations. And the blameless postmortem turns each incident into fixes: missing alerts, fragile dependencies, runbooks that did not exist.
Example. At 2:14 a.m. an alert fires for rising checkout errors. The on-call engineer acknowledges in three minutes, rolls back the previous deploy, and service recovers by 2:31. The postmortem adds a canary check that would have caught the bug before full rollout.
Cost incidents deserve the same machinery: a runaway job burning thousands per hour is an outage for the budget, and anomaly detection is its alerting layer. The application monitoring tools roundup covers the detection stack.
It examines systems and processes rather than individuals. People share honestly only when honesty is safe.
Time to detect, time to resolve, and repeat rate. Improving detection usually delivers the fastest gains.
Yes. A severe cost anomaly gets an owner, a response, and a postmortem, exactly like downtime.