Skip to main content

Filesystem & Storage Playbook

From an SRE perspective, filesystem & storage knowledge is not theoretical. It directly impacts:

  • Availability (disk full → outage)
  • Performance (I/O bottlenecks)
  • Security (wrong permissions → breach)
  • Scalability (poor volume planning → incidents)
  • Recovery (RAID/LVM mismanagement → data loss)

Table of Contents​

  1. Filesystem Hierarchy Standard
  2. Filesystem Commands
  3. Inodes — Silent Production Killer
  4. LVM
  5. RAID Concepts
  6. File Permissions
  7. Special Bits
  8. umask
  9. ACLs
  10. Real SRE Incident Scenarios
  11. Storage Incident Debugging Playbook
  12. Senior SRE Storage Checklist
🧭 Mental modelFive ways filesystem knowledge shows up as SRE riskFilesystem and storage knowledge fans out into five production concerns: availability, performance, security, scalability, and recovery.Filesystem & Storagenot theoretical — production riskAvailabilitydisk full →outagePerformanceI/ObottlenecksSecuritywrong perms →breachScalabilitypoor volumeplanningRecoveryRAID/LVMmismanaged

Filesystem and storage decisions are not theoretical for an SRE — the same underlying knowledge, from inodes to LVM to permission bits, is what determines availability, performance, security, scalability, and recovery outcomes in production.

1. Filesystem Hierarchy Standard (FHS) — Why SREs Care​

FHS defines where things live in Linux. As an SRE, this helps you debug systems fast.

🔹 /etc → Configuration​

Contains system & application configs: /etc/passwd, /etc/fstab, /etc/nginx/nginx.conf

🔥 SRE Impact: misconfigured /etc/fstab → system won't boot. Broken service config → production outage. Configuration drift across servers → inconsistency.

🛠 Monitor: track config changes (GitOps, Ansible), audit sensitive file changes, backup critical configs.

🔹 /var → Logs & Changing Data​

Contains /var/log (system & app logs), /var/lib (databases, app state), /var/spool (queues).

🔥 Classic SRE Incident:

Logs grow → /var fills → system crash
Database can't write → corruption
Kubernetes node → NotReady

🛠 Monitor: disk usage specifically on /var, log rotation health, alert at 70–80% usage.

🔹 /proc → Kernel & Process Info (Virtual FS)​

/proc/cpuinfo
/proc/meminfo
/proc/<pid>/

Used heavily in observability tools. SRE usage: debug memory leaks, check open file descriptors, monitor process limits.

🔹 /sys → Kernel & Device Interface​

Interface to kernel and hardware. Used for tuning performance and inspecting device state. Advanced SRE work (low-level tuning, containers, cgroups).

🔹 /dev → Devices​

Represents disks: /dev/sda, /dev/nvme0n1

Production relevance: mounting the wrong disk = catastrophic data loss.

🔹 /run → Runtime State​

Stores PID files, sockets, lock files. Temporary — cleared on reboot.

2. Filesystem Commands (SRE Operational Toolkit)​

df -h → Disk Free​

Shows disk usage by mounted filesystem.

🚨 Incident pattern: filesystem 100% → application crashes. SRE tip: always check inode usage too (see below).

du -sh​

Finds what's consuming space.

du -sh /var/*

Used during disk pressure incidents.

mount, umount​

Mount filesystems. Critical in: NFS failures, EBS volumes in cloud, Kubernetes persistent volumes.

lsblk​

Shows block devices hierarchy. Useful for debugging new volume attachment and verifying LVM layout.

blkid​

Shows filesystem type + UUID. Important for /etc/fstab troubleshooting and recovery after reboot failure.

findmnt​

Better way to see the mount tree.

3. Inodes — Silent Production Killer​

🔹 What is an inode?​

An inode stores file metadata: permissions, owner, block pointers. It is NOT the filename — just metadata.

🔥 How Inode Exhaustion Breaks Systems​

You can have 0% disk usage but 100% inode usage — e.g. millions of tiny log files.

Result: cannot create files, applications crash, Kubernetes pods fail, email servers stop.

Check with:

df -i

SRE Monitoring: monitor inode usage %, especially for log directories, temp directories, and container systems.

4. LVM (Logical Volume Manager)​

Used heavily in production. Why SREs love LVM: resize disks without downtime, flexible storage allocation, snapshot capability.

🔹 Key Concepts​

PV → Physical Volume (actual disk)

pvcreate /dev/sdb

VG → Volume Group (pool of storage)

vgcreate myvg /dev/sdb

LV → Logical Volume (usable partition)

lvcreate -L 10G -n mylv myvg

🔹 Extending Volume Online (Real Incident Case)​

Disk full → extend without reboot:

lvextend -L +10G /dev/myvg/mylv
resize2fs /dev/myvg/mylv

⚠️ If you forget resize2fs, the filesystem won't grow.

SRE Best Practice: use LVM for databases, keep root small, keep data volumes separate, monitor VG free space.

5. RAID Concepts (Redundancy & Reliability)​

RAID = data redundancy or performance improvement.

🔹 RAID Levels​

LevelPurposeSRE View
RAID 0Performance❌ No redundancy
RAID 1Mirroring✔ High safety
RAID 5Striping + parityBalanced
RAID 10Mirror + stripe🔥 Production favorite

🔹 /proc/mdstat​

Check RAID health:

cat /proc/mdstat

Shows degraded arrays and rebuild progress.

🔥 Real SRE Failure: disk fails in RAID5 → not replaced → second disk fails → total data loss.

Monitor: RAID state, rebuild events, disk SMART health.

6. File Permissions — Security & Stability​

🔹 rwx Model​

User    Group    Others
rwx     rwx      rwx

Example: -rwxr-xr--

chmod​

chmod 755 file

Meaning: owner full, others read/execute.

chown​

Change ownership. Critical for web servers, databases, Kubernetes volumes.

7. Special Bits (Advanced SRE Knowledge)​

🔹 SUID (4000)​

Executable runs as the file owner. Example: /usr/bin/passwd. ⚠️ Security risk if misused.

🔹 SGID (2000)​

In directories: new files inherit the group. Used in shared dev directories.

🔹 Sticky Bit (1000)​

Common on /tmp. Prevents users from deleting others' files.

8. umask​

Default permission mask.

umask 022

Prevents overly permissive files. SRE concern: bad umask → security vulnerability.

9. ACLs (Access Control Lists)​

When rwx isn't enough.

getfacl file
setfacl -m u:user:rw file

Used in enterprise systems, shared storage, complex permission models.

🔥 Real SRE Incident Scenarios​

  1. Disk Full → Production Down — Cause: log rotation misconfigured.
  2. Inode Exhaustion — Cause: millions of temp files.
  3. RAID Degraded → Ignored Alert → Data Loss
  4. Wrong chmod → Application Can't Start
  5. Mounted Wrong Volume → Overwritten Data

📊 What SRE Should Monitor​

MetricWhy
Disk usage %Prevent outages
Inode usage %Silent killer
IOPSPerformance
Disk latencyDB health
RAID stateData protection
LVM free spaceCapacity planning
Log growth rateForecasting

🧠 Senior-Level SRE Thinking​

Always ask: What happens if this disk fills? What happens if this mount disappears? What happens if this node reboots? Can I recover this volume?

🚨 SRE Storage Incident Debugging Playbook​

Structured like a real production incident: Detection → Triage → Stabilization → Root Cause → Prevention.

🔴 SCENARIO 1: Disk Full (Most Common Production Outage)​

Symptoms: app returning 500 errors, database crash, "No space left on device", Kubernetes pods restarting, node NotReady.

Step 1 — Confirm

df -h
df -i    # also check inode exhaustion

Step 2 — Identify what's growing

du -sh /* 2>/dev/null | sort -h
du -sh /var/* | sort -h

Common culprits: /var/log, /var/lib/docker, /tmp, app logs, core dumps.

Real root causes seen in production: log rotation broken, debug logging accidentally enabled, infinite loop writing files, container image layer explosion, backup script writing locally instead of S3, audit logs filling /var.

Step 3 — Emergency mitigation

> large.log   # safely truncate a large log file

⚠️ Never delete active DB files.

Step 4 — Deep diagnosis: Why did the alert not trigger earlier? Was growth sudden or gradual? Is this predictable growth? Check historical disk growth rate and log rate.

Step 5 — Prevention: alerts at 70%, 80%, 90%; log rotation validation; capacity forecasting; separate log partition; auto-scaling or auto-expansion.

🔴 SCENARIO 2: Inode Exhaustion (Silent Killer)​

Symptoms: disk shows free space, but app says "No space left"; file creation fails.

df -i    # confirm
for i in /*; do echo $i; find $i | wc -l; done
find /var -xdev -type f | wc -l

Common causes: millions of tiny cache files, email queues, Kubernetes emptyDir abuse, temp file leak, log sharding misconfiguration.

Fix: delete unnecessary small files, restart the service creating them, consider reformatting with more inodes (long-term).

🔴 SCENARIO 3: High Disk I/O / Latency (Performance Incident)​

Symptoms: DB slow, API latency spike, CPU iowait high, pods timing out.

top          # look at %wa
iostat -x 1  # look at await, svctm, %util

Common causes: heavy backup job, unindexed DB query, log burst, swap thrashing, RAID rebuild.

Real production example: backup ran during peak traffic → saturated disk → checkout failures.

Mitigation: stop backup, throttle IO, move heavy jobs to off-peak, add faster disk, use a separate volume for DB logs.

🔴 SCENARIO 4: RAID Degraded​

cat /proc/mdstat

Look for [UU] (healthy) vs [U_] (degraded).

If degraded: identify failed disk → replace disk → rebuild RAID → monitor rebuild. ⚠️ Risk: if a second disk fails in RAID5 → full data loss.

🔴 SCENARIO 5: Filesystem Corruption​

Symptoms: system won't boot, "UNEXPECTED INCONSISTENCY", files missing, read-only filesystem.

dmesg | grep -i error
fsck /dev/sdX

⚠️ Never run fsck on a mounted production filesystem unless read-only. Downtime required.

🔴 SCENARIO 6: Mount Disappeared (Cloud / NFS)​

Symptoms: app freeze, high load, threads stuck in D state, NFS timeout.

mount | grep nfs
findmnt

Real cloud incident pattern: network glitch, EBS detaches, Kubernetes node stuck, pod cannot write.

Mitigation: remount, restart service, possibly reboot node.

🔴 SCENARIO 7: Permission Denied After Deployment​

Symptoms: app fails after release, logs show "Permission denied".

ls -l   # check owner, group, mode

Common real cause: CI/CD changed ownership, or a Docker volume was mapped incorrectly.

📊 Senior SRE Storage Checklist​

When paged for a storage issue:

1️⃣ What is failing? App? DB? Node?

2️⃣ Is it: Capacity? Performance? Corruption? Mount? Permissions?

3️⃣ What changed? Deployment? Backup? Traffic spike? New log level?

4️⃣ Is data at risk?

Advanced Production Design Principles​

Always design for: separate OS and data disks; separate logs from DB; RAID + backups; snapshot strategy; alerting before impact; load-testing storage.

Observability Metrics to Always Have​

MetricCritical
Disk usage %Yes
Inode %Yes
IOPSYes
LatencyYes
ThroughputYes
iowaitYes
Mount availabilityYes

Summary​

  • 💡 "If it fills, you fail." Treat disk like memory — monitor it before it becomes an outage.
  • 🔥 Disk-full with df -h showing free space almost always means either a deleted-but-open file (lsof | grep deleted) or inode exhaustion (df -i), not a du measurement error.
  • ⚠️ Storage incidents cascade fast and can corrupt data — they are more dangerous than CPU/memory incidents and are often under-monitored in capacity planning.
  • ✅ Always run resize2fs after lvextend — extending the LV without growing the filesystem is a common silent mistake.

See Also​