Incident Response Playbook for SREs

Managing production incidents effectively requires preparation, clear processes, and calm execution. This playbook outlines battle-tested strategies for handling incidents from detection to resolution.

Critical Principle: The goal of incident response is to restore service as quickly as possible while minimizing impact to users.

Incident Severity Levels

Understanding severity helps prioritize response and allocate resources appropriately.

Severity	Impact	Response Time	Examples
P0 - Critical	Complete service outage	< 15 minutes	Database down, site unreachable
P1 - High	Major feature unavailable	< 1 hour	Payment processing failing
P2 - Medium	Degraded performance	< 4 hours	Slow page loads, partial failures
P3 - Low	Minor issues	< 24 hours	Cosmetic bugs, non-critical errors

Phase 1: Detection and Alert

Monitoring Setup

Implement comprehensive monitoring across all layers:

# Example Prometheus alert rules
groups:
  - name: website_alerts
    interval: 30s
    rules:
      - alert: HighErrorRate
        expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High error rate detected"
          description: "Error rate is {{ $value }}% over 5 minutes"

      - alert: HighResponseTime
        expr: http_request_duration_seconds{quantile="0.95"} > 2
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "High response time detected"
          description: "95th percentile is {{ $value }}s"

      - alert: DatabaseConnectionPoolExhausted
        expr: db_connections_active / db_connections_max > 0.9
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Database connection pool near capacity"

Alert Channels

Configure multiple notification channels:

PagerDuty/OpsGenie: For critical alerts (P0, P1)
Slack: For all alerts with severity filtering
Email: For digest and low-priority alerts
SMS: For P0 incidents only

Phase 2: Initial Response

Incident Commander Checklist

When you’re the first responder:

Acknowledge the alert immediately
Assess the scope and severity
Create incident channel (#incident-YYYYMMDD-HHmm)
Start incident timeline document
Notify stakeholders
Gather initial data

Quick Assessment Script

#!/bin/bash
# Quick health check for initial triage

echo "=== System Health Check ==="
echo "Timestamp: $(date)"
echo ""

# Check system resources
echo "--- CPU & Memory ---"
top -b -n 1 | head -5

echo ""
echo "--- Disk Usage ---"
df -h | grep -v tmpfs

echo ""
echo "--- Active Connections ---"
netstat -an | grep ESTABLISHED | wc -l

echo ""
echo "--- Recent Errors (last 100 lines) ---"
tail -100 /var/log/application/error.log | grep ERROR | tail -10

echo ""
echo "--- Database Status ---"
mysql -e "SHOW PROCESSLIST;" | wc -l
echo "Active MySQL connections"

echo ""
echo "--- Container Status ---"
docker ps --filter "status=running" --format "table {{.Names}}\t{{.Status}}"

Phase 3: Diagnosis

Systematic Troubleshooting

Follow the RED method for microservices:

Rate: Request volume
Errors: Error rate
Duration: Response time

# Quick metrics collection
# Request Rate
echo "Requests per minute:"
awk '{print $4}' /var/log/nginx/access.log | cut -d: -f1,2 | \
  uniq -c | tail -10

# Error Rate
echo "Errors per minute:"
grep " 500 " /var/log/nginx/access.log | \
  awk '{print $4}' | cut -d: -f1,2 | uniq -c | tail -10

# Response Time (95th percentile)
echo "95th percentile response time:"
awk '{print $NF}' /var/log/nginx/access.log | \
  sort -n | awk '{a[NR]=$1} END {print a[int(NR*0.95)]}'

Common Investigation Paths

Database Issues

# Check slow queries
mysql -e "SHOW FULL PROCESSLIST;" | grep -v Sleep

# Check table locks
mysql -e "SHOW OPEN TABLES WHERE In_use > 0;"

# Check connection count
mysql -e "SHOW STATUS LIKE 'Threads_connected';"
mysql -e "SHOW VARIABLES LIKE 'max_connections';"

Memory Issues

# Memory usage by process
ps aux --sort=-%mem | head -10

# Check for OOM kills
dmesg | grep -i "out of memory"
grep -i "out of memory" /var/log/syslog

# Available memory
free -m

Network Issues

# Check connection states
netstat -ant | awk '{print $6}' | sort | uniq -c | sort -n

# Check network errors
netstat -i

# DNS resolution test
dig example.com

Phase 4: Mitigation

Immediate Actions

Choose the fastest path to restore service:

Warning: Document all changes made during mitigation for post-mortem analysis!

Quick Wins

Restart service (if safe)

systemctl restart application
# Monitor logs immediately
journalctl -u application -f

Scale horizontally (if capacity issue)

docker-compose up -d --scale web=5
# or with Kubernetes
kubectl scale deployment web --replicas=10

Enable maintenance mode (if critical bug)

# Nginx maintenance page
cp /etc/nginx/sites-available/maintenance \
   /etc/nginx/sites-enabled/default
nginx -t && systemctl reload nginx

Rollback deployment

# Docker
docker service update --rollback web

# Kubernetes
kubectl rollout undo deployment/web

Circuit Breaker Pattern

Protect downstream services:

# Example circuit breaker implementation
from circuitbreaker import circuit

@circuit(failure_threshold=5, recovery_timeout=60)
def call_external_api():
    response = requests.get('https://api.example.com/data')
    response.raise_for_status()
    return response.json()

try:
    data = call_external_api()
except CircuitBreakerError:
    # Use cached data or return degraded response
    data = get_cached_data()

Phase 5: Communication

Status Updates

Provide regular updates to stakeholders:

Initial Notification (within 15 minutes):

🔴 INCIDENT DETECTED [P0]

Issue: Website responding with 503 errors
Impact: All users unable to access the site
Started: 14:23 UTC
Status: Investigating

Next update: 14:45 UTC

Progress Updates (every 15-30 minutes):

🟡 UPDATE #2 [P0]

Root cause identified: Database connection pool exhausted
Action taken: Increased connection pool size
Status: Service partially restored (80% traffic)

ETA for full resolution: 15:30 UTC
Next update: 15:15 UTC

Resolution Notice:

🟢 RESOLVED [P0]

Issue: Database connection pool exhaustion
Resolution: Pool size increased from 100 to 300
Impact: 47 minutes total downtime
Recovery: 100% traffic restored

Post-mortem: Will be published within 72 hours

Phase 6: Post-Mortem

Blameless Post-Mortem Template

# Incident Post-Mortem: [Title]

**Date**: 2024-01-15
**Duration**: 47 minutes
**Severity**: P0
**Impact**: Complete service outage

## Summary

Brief description of what happened and the impact on users.

## Timeline (All times UTC)

- **14:23** - Alert triggered: High error rate
- **14:25** - Incident commander joined
- **14:30** - Root cause identified
- **14:35** - Mitigation deployed
- **14:45** - Service partially restored (80%)
- **15:10** - Full service restored

## Root Cause

The database connection pool was configured with a maximum of 100
connections. Traffic spike from marketing campaign exceeded this limit,
causing connection timeouts and 503 errors.

## Resolution

1. Increased connection pool from 100 to 300
2. Added monitoring for connection pool usage
3. Configured auto-scaling for application tier

## Action Items

| Action | Owner | Due Date | Status |
|--------|-------|----------|--------|
| Implement connection pool monitoring | @ops | 2024-01-20 | ✅ |
| Add capacity planning runbook | @sre | 2024-01-25 | 🔄 |
| Review all pool configurations | @dev | 2024-02-01 | ⏳ |
| Load test new configuration | @qa | 2024-01-22 | ✅ |

## Lessons Learned

### What Went Well
- Fast detection (< 2 minutes)
- Clear communication
- Quick root cause identification

### What Could Be Improved
- Better capacity planning
- Proactive monitoring
- Load testing before campaigns

Prevention Strategies

Chaos Engineering

Test your systems proactively:

# Example chaos experiments

# 1. Kill random container
docker kill $(docker ps -q | shuf -n 1)

# 2. Inject network latency
tc qdisc add dev eth0 root netem delay 100ms 20ms

# 3. Fill disk space
dd if=/dev/zero of=/tmp/fill bs=1M count=1000

# 4. CPU stress
stress --cpu 8 --timeout 60s

# 5. Memory pressure
stress --vm 2 --vm-bytes 1G --timeout 30s

Automated Remediation

Implement self-healing systems:

# Kubernetes example with liveness probes
apiVersion: v1
kind: Pod
metadata:
  name: web-app
spec:
  containers:
  - name: app
    image: myapp:latest
    livenessProbe:
      httpGet:
        path: /health
        port: 8080
      initialDelaySeconds: 10
      periodSeconds: 5
      failureThreshold: 3
    readinessProbe:
      httpGet:
        path: /ready
        port: 8080
      initialDelaySeconds: 5
      periodSeconds: 3

Conclusion

Effective incident response combines preparation, process, and practice. Key principles:

Prepare: Set up monitoring, alerts, and runbooks before incidents occur
Process: Follow established procedures to reduce cognitive load
Practice: Run game days and chaos experiments regularly
Learn: Conduct blameless post-mortems and implement improvements

Remember: Every incident is a learning opportunity. Focus on system improvement, not individual blame.

Quick Reference Card

Save this for quick access during incidents:

INCIDENT RESPONSE CHECKLIST
===========================
☐ Acknowledge alert
☐ Create incident channel
☐ Assess severity (P0-P3)
☐ Start timeline doc
☐ Notify stakeholders
☐ Begin diagnosis
☐ Implement mitigation
☐ Send status updates (every 15-30min)
☐ Verify resolution
☐ Send resolved notification
☐ Schedule post-mortem (within 72h)

Written by Dzubayyan Ahmad | System Administrator & SRE