How to resolve disk full — automated runbook

Trigger Conditions

What the Agent Does

Example Incident Log

Symptoms

• Disk usage at 93% on prod-web-03 (/dev/sda1)
• CRITICAL: Volume /data is full on api-server-1
• Datadog: system.disk.in_use > 0.9 for >5m on host prod-db-3
• PagerDuty: Disk capacity exceeded on api-server-1 — CRITICAL
• Prometheus: node_filesystem_avail_bytes < 2GB on /var/lib
• CloudWatch: RDS FreeStorageSpace < 5GB on prod-pg-1

Root cause diagnostic tree

1. df -h — which filesystem is full?
   ├── /var/log partition → rotated logs not trimmed
   ├── /tmp partition → app temp files not cleaned
   └── root volume with /var/lib/docker → container overlay bloat

2. du -shx /* | sort -hr | head -10
   └── Locate the top consumer directory

3. Check file recency:
   ├── find /var/log -mtime +7 -ls
   └── If > 5 GB of >7d logs → logrotate failed

4. Capacity relief:
   ├── Compress with gzip + remove >7d logs
   ├── Prune Docker: docker system prune -af --filter "until=72h"
   └── Last resort: expand the EBS volume + resize2fs

Manual remediation steps

# 1. Identify what's eating the disk
df -h
sudo du -shx /* 2>/dev/null | sort -hr | head -10

# 2. Vacuum rotated journals
sudo journalctl --vacuum-time=7d

# 3. Compress and prune stale app logs
sudo find /var/log -type f -name "*.log" -mtime +1 -exec gzip {} \;
sudo find /var/log -type f -name "*.gz" -mtime +7 -delete

# 4. Trim Docker overlay layers
sudo docker system prune -af --filter "until=72h"

# 5. Clear apt / yum caches
sudo apt-get clean
sudo yum clean all

# 6. Verify usage is back below 80%
df -h /

How OnCallReady's agent handles this

Related runbooks