Skip to main content

DevOps/SRE Interview Scenarios

Table of Contents​

  1. Core DevOps/SRE Scenarios
  2. AWS + Linux Combined Incident Simulations
  3. Linux Troubleshooting Scenario Q&A
  4. Additional Incident Pattern Catalog
  5. What Interviewers Look For
🧭 Mental modelFour practice domains, one evaluation signalInterview prep spans Linux troubleshooting, combined AWS and Linux incidents, Kubernetes scenarios, and a 15-pattern incident catalog, all ultimately judged by the same signal: structured, calm, evidence-driven troubleshooting.Linux Scenarioscore troubleshootingAWS + Linuxcombined incidentsKubernetespod / cluster failuresIncident Pattern15-pattern catalogWhat Interviewers Look Forstabilize, evidence, priority, communication

The four practice domains here — Linux, AWS+Linux, Kubernetes, and the broader incident-pattern catalog — are just different surfaces for testing the same underlying signal: can you stabilize first, reason from evidence, prioritize correctly, and communicate clearly under pressure.

🚀 DevOps / SRE Scenario-Based Interview Q&A​

1️⃣ Production API Latency Suddenly Increased​

Scenario: latency jumped from 50ms to 800ms after a new deployment.

Answer:

  1. Stabilize first — roll back if customer impact is high; reduce blast radius.
  2. Check metrics — CPU, memory, I/O, DB latency, external API calls, error rate.
  3. Compare before vs after deploy — new queries? new dependencies? increased payload size?
  4. Check logs and tracing — slow endpoints? time spent in DB vs app?

👉 Priority = restore service first, then root cause.

2️⃣ Kubernetes Pods Keep Restarting​

Scenario: pods restart every few minutes.

Answer:

kubectl get pods
kubectl describe pod <name>
kubectl logs <pod>

Common causes: OOMKilled, CrashLoopBackOff, liveness probe failing, misconfigured environment variables.

If OOM: increase memory limit, fix memory leak, adjust resource requests/limits.

👉 In Kubernetes, restarts usually indicate resource or health-probe issues.

3️⃣ Database CPU at 100%​

Answer:

  1. Identify heavy queries; check slow query logs.
  2. Check: missing indexes? full table scans? sudden traffic spike?
  3. Look at connection count.
  4. If needed: scale vertically, add read replicas, cache results (Redis).

👉 Databases fail due to bad queries more often than hardware.

4️⃣ Memory Leak in Production​

Answer:

  1. Confirm the pattern via monitoring.
  2. Check heap usage (JVM/Node/Python).
  3. Take a memory dump; analyze for unreleased objects / growing cache.
  4. Short-term fix: restart with rolling deployment.
  5. Long-term: fix the code leak.

👉 SRE focuses on mitigation + permanent fix.

5️⃣ Service is Up but Users See Errors​

Scenario: monitoring shows healthy, but users get 500 errors.

Answer: health checks may only test a basic endpoint, not the full dependency chain. Possible causes: downstream service failing, DB connection pool exhausted, timeout misconfiguration.

Steps: check logs, check dependency health, trace a full request path.

👉 Health checks must test real functionality, not just "process running."

6️⃣ Deployment Causes Partial Outage​

Scenario: after deployment, 30% of users get errors.

Answer: common reasons — rolling deployment with incompatible versions, DB schema mismatch, cache inconsistency, one AZ unhealthy, feature flag misconfiguration.

Fix: roll back; use backward-compatible migrations; use canary deployments.

👉 SRE principle: deploy safely and gradually.

7️⃣ Traffic Suddenly Spikes 10x​

Answer:

  1. Check: is it real traffic or DDoS?
  2. If real: is auto-scaling working? scale horizontally.
  3. If DDoS: enable rate limiting, activate WAF, use CDN protection.
  4. Monitor: CPU, DB connections, queue depth.

👉 Resilience design matters before traffic spikes.

8️⃣ High Error Rate But No Infra Issues​

Answer: likely dependency failure, expired certificate, feature flag issue, or external API change. Check recent deployments, external API status, TLS cert expiry, config changes.

👉 Not all outages are resource-related.

9️⃣ Alert Fatigue in Monitoring​

Answer:

  1. Remove low-value alerts.
  2. Use SLO-based alerting.
  3. Alert on symptoms, not causes.
  4. Create severity levels.
  5. Use aggregation and deduplication.

👉 Alert only when user impact exists.

🔟 Incident: Entire Region Goes Down​

Answer: architecture should have multi-region deployment, active-active or active-passive failover, global load balancer, replicated database, automated failover. After recovery: run postmortem, identify weak spots, improve disaster recovery plan.

1️⃣1️⃣ How Do You Design for Reliability? (Senior-Level)​

Answer:

  1. Define SLOs (Service Level Objectives).
  2. Measure SLIs (latency, availability).
  3. Implement error budgets.
  4. Use auto-scaling, circuit breakers, retries with backoff, observability (metrics, logs, traces).
  5. Run chaos testing.
  6. Do postmortems.

👉 Reliability is engineered, not hoped for.

AWS + Linux Combined Incident Simulations​

Test whether you can connect cloud infra + OS debugging.

1️⃣ EC2 reachable but application down​

AWS layer: instance status = running. Linux layer: service crashed. Investigate: SSH → check process, logs, ports; compare deployment/config change. Root cause pattern: app crash, bad startup config, port binding failure. Key tools: EC2 instance console logs, ps, journalctl, /proc

2️⃣ High latency only on one EC2 instance​

AWS layer: load balancer healthy. Linux layer: node overloaded. Investigate: compare CPU/memory across nodes; check thread count, FD usage; look for hot shard / uneven load. Root cause pattern: instance-level resource exhaustion. Key services: Elastic Load Balancing, Linux process + limits debugging

3️⃣ Application works locally but fails after scaling​

AWS layer: new instances fail health check. Linux layer: missing dependency. Investigate: compare AMI, environment variables; validate startup scripts. Root cause pattern: config drift between instances. Key services: Auto Scaling, startup logs + /proc

4️⃣ CPU low but response time high​

AWS layer: infrastructure stable. Linux layer: waiting on external dependency. Investigate: process state → I/O wait; check network calls / DB connections. Root cause pattern: downstream service slowness. Key services: Amazon RDS latency, Linux process state D

5️⃣ Instance crashes during traffic spike​

AWS layer: auto-restart observed. Linux layer: resource exhaustion. Investigate: kernel logs; memory usage trend; limits and FD count. Root cause pattern: OOM killer, insufficient memory sizing. Key services: CloudWatch metrics, /proc/<pid>/limits

6️⃣ Users intermittently cannot connect​

AWS layer: load balancer reports 502. Linux layer: service unstable. Investigate: health check endpoint; port listening status; restart frequency. Root cause pattern: crash loop or timeout mismatch. Key services: Application Load Balancer, Linux service lifecycle

7️⃣ Deployment causes partial outage​

AWS layer: some instances OK, others failing. Linux layer: inconsistent config. Investigate: compare environment variables; check running version; validate startup scripts. Root cause pattern: configuration drift. Key services: AWS Systems Manager, process inspection

8️⃣ Disk usage suddenly full on EC2​

AWS layer: instance healthy. Linux layer: disk exhaustion. Investigate: identify largest files; check log rotation; inspect open deleted files. Root cause pattern: log growth or temp file leak. Key services: Amazon EBS, Linux file descriptors

9️⃣ Traffic drop but instances healthy​

AWS layer: requests not reaching service. Linux layer: no issue locally. Investigate: DNS resolution; load balancer routing; security rules. Root cause pattern: routing or DNS misconfiguration. Key services: Route 53, network checks on Linux

🔟 Sudden increase in error rate after scaling​

AWS layer: new instances failing under load. Linux layer: resource limits too low. Investigate: open file limit; thread limit; connection pool. Root cause pattern: default limits insufficient for scale. Key services: VPC connectivity + limits, /proc/<pid>/limits

🎯 Interview Answer Framework (AWS + Linux)​

Step 1 — Infra health: instance state, load balancer, scaling events, metrics Step 2 — Node health: process running? CPU/memory/FD, logs + limits Step 3 — Dependencies: DB, network, config drift Step 4 — Mitigate then root cause: restart, scale, reroute traffic → fix underlying issue

🔥 Scenario-Based Linux Troubleshooting Q&A​

1️⃣ Server is Slow – High CPU Usage​

top
htop
uptime

Identify which process is consuming CPU, and whether it's user vs system CPU. If system CPU is high → possibly kernel, I/O wait, or interrupts. If one process misbehaves → restart service, investigate logs. If many processes → possible scaling issue. If iowait is high → investigate disk bottlenecks.

2️⃣ High Load Average but Low CPU Usage​

Indicates processes are waiting — most likely for I/O.

top   # check %wa
iostat -x 1

If disk utilization is high → disk bottleneck, possibly database workload or slow storage.

👉 High load ≠ high CPU. It includes processes waiting for I/O.

3️⃣ System Out of Memory (OOM Killer Triggered)​

dmesg | grep -i oom
free -m
top

Possible causes: memory leak, too many processes, no swap configured. Fix: add swap, increase RAM, fix memory leak, adjust vm.overcommit_memory.

4️⃣ Disk is Full but du Doesn't Show Large Files​

Likely a deleted file still open by a running process.

lsof | grep deleted

If found, restart the service holding the file — space will be released.

👉 Linux doesn't free space until file descriptors are closed.

5️⃣ Service Cannot Bind to Port 80​

ss -tulnp | grep :80
netstat -tulnp

Kill the conflicting process or change the port.

👉 Only one process can bind to a port at a time (unless using special socket options).

6️⃣ SSH is Slow to Connect​

Common causes: DNS reverse lookup delay, GSSAPI authentication enabled, network latency. Fix in /etc/ssh/sshd_config:

UseDNS no
GSSAPIAuthentication no

Restart SSH service.

7️⃣ High Memory Usage but Free Memory is Low​

Not a problem — Linux uses free memory for buffer/cache.

free -m   # look at "available", not just "free"

Linux frees cache automatically if needed.

👉 "Free memory is wasted memory" in Linux.

8️⃣ One Process Consuming All Memory​

top
ps aux --sort=-%mem

Check: is it expected workload? memory leak? Check JVM heap settings. Restart service if necessary; monitor over time.

9️⃣ Server Randomly Reboots​

journalctl -xb -1
dmesg

Look for kernel panic, OOM, hardware errors, power failure. Possible causes: hardware issue, faulty RAM, kernel bug, power supply problem.

🔟 Too Many Open Files Error​

ulimit -n
cat /proc/sys/fs/file-max

Fix: increase limits in /etc/security/limits.conf; restart service.

1️⃣1️⃣ High Interrupt Usage (Advanced)​

Scenario: system CPU is high, but no process is using it. top shows high %si (software interrupt).

cat /proc/interrupts

Likely causes: network flood, faulty driver, hardware issue. May need NIC tuning, driver update, IRQ balancing.

🎯 Interview Tip for Scenario Questions​

When answering: 1) identify the symptom, 2) mention diagnostic commands, 3) explain likely root causes, 4) suggest a fix, 5) mention prevention if possible. This shows structured troubleshooting thinking.

🚨 Additional Incident Pattern Catalog​

Fifteen additional real production-style incident patterns for rapid-fire practice — symptom, check, and root-cause pattern only (no commands), useful for verbal mock-interview drills.

#ScenarioSymptomsCheckRoot Cause Pattern
1API latency spike after deploymentResponse time ↑, error rate stableCPU, threads, DB connections, config diffInefficient code path, connection pool exhaustion
2Service healthy but users cannot connectProcess running, no responseListening ports, firewall, socket statePort conflict, network ACL change
3CPU suddenly at 100% on one nodeLoad imbalanceTop processes, thread usage, recent trafficHot shard, infinite loop, retry storm
4Memory keeps increasing slowlyWorks fine initially → crashes laterMemory usage trend, OOM logsMemory leak
5"Too many open files" errorService stops accepting requestsOpen FDs, limits, connection handlingFile/socket leak
6Service not stopping during restartkill ignoredProcess stateStuck in I/O wait or kernel lock
7Sudden spike in load averageSystem slow, CPU moderateLoad vs CPU cores, blocked processesDisk bottleneck or lock contention
8Intermittent 502 / gateway errorsRandom failuresUpstream health, timeout settingsDependency slowness
9Node crashes under traffic spikeService restarts automaticallyResource limits, OOM killerInsufficient memory or FD limit
10Logs stop updating but process aliveNo new activityThread state, blocked syscallsDeadlock or external dependency hang
11High response time but CPU lowUsers complain, system idleI/O wait, network, DB latencyExternal service bottleneck
12Many zombie processes accumulateProcess table fills slowlyParent process behaviorChild exit not handled
13Deployment works on some nodes onlyPartial outageConfig drift, environment variablesInconsistent infrastructure state
14Traffic drop but service healthyNo errors but usage downDNS, load balancer, routingTraffic not reaching service
15Monitoring alerts but system seems fineFalse positivesAlert thresholds, metrics lagMisconfigured monitoring

🧠 How SRE3 Answers in Interview​

For ANY scenario, structure the response like this:

1️⃣ Identify process / service health 2️⃣ Check CPU, memory, I/O, limits 3️⃣ Inspect dependencies 4️⃣ Verify configuration changes 5️⃣ Mitigate → then find root cause

That structured, calm thinking is the signal interviewers look for.

🎯 What Interviewers Look For​

  • Structured thinking
  • Calm incident response
  • Understanding of distributed systems
  • Tradeoff decisions
  • Prevention mindset (not just fixing)

Summary​

  • 💡 Every answer should follow the same shape: identify → check resources/dependencies → verify recent changes → mitigate → root-cause → prevent.
  • 🔥 "Health check passing" and "service actually working" are different claims — probes must exercise the real dependency chain, not just a liveness ping.
  • ⚠️ At scale, default resource limits (open files, threads, connection pools) become the bottleneck even when code and infra are unchanged.
  • ✅ For any region/AZ-level failure, the answer is always architectural: multi-region, automated failover, replicated data — not a runbook step.

See Also​