Incident response cheatsheet
A one-page reference for incident response and Linux SRE debugging. For the full mindset, roadmap, and interview scenarios, see the complete guide.
📖 Full guide: Incident Response →Severity & roles
| Concept | One-liner |
|---|---|
| SEV1/2/3/4 | full outage → major degradation → minor/limited → cosmetic |
| IC | owns decisions |
| Comms Lead | owns messaging |
| Scribe | owns the timeline |
| SMEs | fix — never overlap these roles under pressure |
Incident lifecycle
Detect → Triage/Declare → Mitigate → Resolve → Postmortem
Comms cadence: SEV1 every 15-30min, SEV2 every 30-60min, SEV3 at milestones only.
On-call & paging
- Primary paged first, Secondary is the safety net.
- Escalation policy defines the timeout chain (who/when).
- Alert routing = which service; dedup collapses noise into one incident.
Stabilize first (0-5 min)
- Ack the page — stop the noise.
- Confirm real user impact (not just an alert).
- Declare severity — don't investigate silently.
- Say out loud what you're checking next.
Process commands
ps aux --sort=-%cpu | head
top # P=sort CPU, M=sort mem, k=kill
pstree -p
kill -15 PID # graceful
kill -9 PID # force
pgrep -fl java
nice -n 10 myscript.sh
renice 5 -p 1234
/proc — real gold for SREs
cat /proc/<pid>/status # mem, state, threads
cat /proc/<pid>/limits # ulimits in effect
ls -la /proc/<pid>/fd # open file descriptors
cat /proc/loadavg
cat /proc/meminfo
Filesystem & disk
df -h # disk usage
du -sh */ | sort -h # biggest dirs
lsof +D /path # who has files open here
lsof | grep deleted # deleted-but-held disk space
Inodes exhausted ≠ disk full — check df -i separately.
Follow the request path
- Client → LB → is the LB healthy / routing correctly?
- LB → app → is the process alive and accepting connections?
- App → DB/cache → is the dependency the actual bottleneck?
- Check logs at each hop, not just the top of the stack.
Postmortem
- Blameless. Timestamped timeline. 5-whys root cause.
- Contributing factors listed separately from root cause.
- Action items have owners and dates — no names attached to blame.
Runbooks
Write for a stressed reader at 3am: numbered steps, exact commands, explicit decision points — not prose. If a step has a judgment call, say what to check to make it.
See: 90-Day Linux SRE Roadmap for a full ramp-up plan