Storage Drives, RAID, and Database Performance

Storage design is a set of tradeoffs between latency, throughput, durability, rebuild risk, capacity, and operational recovery. A fast disk layout that loses the only copy of data is not resilient. A redundant disk layout that hides latency until a database stalls is not healthy. Databases make these tradeoffs visible because they issue sync writes, random reads, scans, compactions, checkpoints, and recovery reads under real user pressure.

Command Examples

lsblk -o NAME,TYPE,MODEL,SERIAL,ROTA,SIZE,FSTYPE,MOUNTPOINTS
cat /proc/mdstat
mdadm --detail /dev/md0
iostat -xz 1
smartctl -a /dev/sda
nvme smart-log /dev/nvme0

Example output and meaning:

Drive Types

lsblk exposes ROTA: 1 usually means rotational media, while 0 usually means SSD or virtualized storage. Treat it as a clue, not a full performance model. Cloud volumes, SAN LUNs, and virtual disks may hide the real media.

RAID Vocabulary

RAID is not backup. RAID can survive some device failures. It does not protect against deletion, corruption, ransomware, bad migrations, wrong rm, filesystem bugs, or application-level logical damage.

RAID 0 Through 10

RAID 10 is usually the conservative default for write-heavy local storage when you need both redundancy and performance. RAID 0 is a performance tool, not a resilience tool. RAID 5 and RAID 6 can be reasonable for capacity, but parity updates and long rebuilds are poor fits for latency-sensitive write workloads.

RAID level details:

• RAID 0 stripes data across all members. Large reads and writes can use all disks, but the array’s failure probability rises with every added member because any single disk loss loses the whole volume.
• RAID 1 mirrors data. Reads can be balanced across copies, but each write must reach every mirror image that participates in durability. Rebuild reads from a surviving copy and writes the replacement.
• RAID 5 stripes data plus one parity block per stripe. A small overwrite often becomes read old data, read old parity, write new data, write new parity. That read-modify-write path is why small random writes hurt.
• RAID 6 is like RAID 5 with two parity calculations. It tolerates two failed members, but small writes have more parity work and rebuilds still stress every surviving disk.
• RAID 10 stripes across mirrors. It gives mirror-style rebuild behavior and stripe-style aggregate throughput. It can survive multiple disk failures only when at least one member in each mirror set survives.
• RAID 0+1 is a mirror of striped sets. RAID 10 is a stripe of mirrors. RAID 10 is usually preferred because one failed disk only degrades one mirror pair; RAID 0+1 can lose a whole stripe side after one member fails.

RAID Performance Patterns

RAID performance is workload-shaped. A layout that looks fast in sequential benchmarks may be bad for sync commits, compactions, or degraded rebuilds.

Important knobs and caveats:

• Small random parity writes are expensive because the array must preserve parity consistency. Full-stripe writes are better because parity can be calculated from the new data without reading old data first.
• Chunk size and stripe width should match the workload only after measurement. Too small can amplify metadata and seek overhead; too large can underuse disks for medium I/O.
• Write-back cache is useful only when it is power-loss protected. A cache that acknowledges durable writes before they are safe can break PostgreSQL and filesystem assumptions.
• Rebuilds compete with application I/O. Throttling rebuild too hard extends the risk window; running it unbounded can stall the database.
• SSD and NVMe arrays still need endurance planning. RAID does not remove write amplification, garbage collection, thermal throttling, or firmware failure risks.
• Discard/TRIM must pass through every layer that should receive it: filesystem, LVM, dm-crypt, md RAID, multipath, hypervisor, and physical device.

Disk Failure Recovery

The first goal during a disk failure is evidence and containment, not heroic repair.

1. Stop unnecessary writes and reduce rebuild pressure if the service allows it.
2. Identify whether the failure is physical media, controller, cable/path, filesystem, RAID member, cloud volume, or application corruption.
3. Capture evidence before changing state: lsblk, /proc/mdstat, mdadm --detail, dmesg, journalctl -k, SMART/NVMe health, and application errors.
4. Verify backups, snapshots, replicas, and restore points before replacing or rebuilding.
5. If the array is degraded but serving, decide whether to fail over the application before rebuilding.
6. Replace the failed member by stable identity: serial, by-id path, enclosure slot, cloud disk ID, or md member UUID.
7. Start the rebuild and monitor read errors, speed, remaining time, temperature, and latency.
8. After rebuild, run filesystem checks only when the lower block layer is stable and the filesystem requires it.
9. Validate application-level consistency: PostgreSQL recovery status, Elasticsearch cluster health, checksums, sample reads, and backups.

Do not run fsck, xfs_repair, or database repair tools on top of a device that is still returning I/O errors. Lower-layer instability can turn recoverable metadata damage into unrecoverable damage.

Common Linux md recovery checks:

cat /proc/mdstat
mdadm --detail /dev/md0
mdadm --examine /dev/sdX1
journalctl -k | grep -Ei 'md|raid|I/O error|medium error|reset|nvme|scsi'

Typical replacement shape:

mdadm /dev/md0 --fail /dev/sdX1
mdadm /dev/md0 --remove /dev/sdX1
mdadm /dev/md0 --add /dev/sdY1
watch -n 5 cat /proc/mdstat

Use those commands only after mapping the correct device. Removing the wrong member from a degraded array can destroy the last good redundancy.

RAID Failure Modes

Disk failures are only one class of RAID incident.

Recovery by level:

• RAID 0: there is no degraded recovery. Restore from backup, rebuild from replicas, or recreate the scratch/cache volume.
• RAID 1: identify the failed member by serial or by-id path, fail/remove it if needed, add the replacement, and watch mirror resync.
• RAID 5: verify backups before rebuild, replace one failed member, monitor surviving disks for read errors, and consider application failover during rebuild.
• RAID 6: tolerate up to two member failures, but still avoid casual rebuilds. Replace one failed device at a time unless the platform explicitly supports a planned multi-device operation.
• RAID 10: map mirror pairs before replacing hardware. The dangerous case is losing both members of the same mirror pair, not simply losing any two disks.

Do not force-assemble an array, run filesystem repair, or start database-level repair until you know which members contain the newest writes. If the array holds production data and metadata is ambiguous, clone members or take block-level images before invasive recovery.

Rebuild Risk

Rebuilds are dangerous because they read the surviving members heavily while writing the replacement. On large HDD arrays, rebuilds can last hours or days. During that window:

• latency rises for applications,
• a second disk can fail,
• an unreadable sector can block reconstruction,
• controller or cable problems can appear as disk failures,
• parity RAID has less margin than mirrors or RAID 10.

Mitigations:

• keep current backups and test restores,
• prefer RAID 10 or distributed replication for write-heavy critical data,
• replace suspect drives before they fail hard,
• avoid mixing very old and new disks in one failure domain,
• monitor SMART/NVMe wear and media errors,
• schedule rebuilds or failovers with application impact in mind.

LVM With RAID

LVM and RAID can be layered in several valid ways. The important part is knowing which layer owns redundancy and which layer owns allocation.

flowchart TB
  Disks[Physical disks / cloud volumes] --> Redundancy[RAID or replicated storage]
  Redundancy --> Encryption[Optional dm-crypt / LUKS]
  Encryption --> Allocation[LVM volume group and logical volumes]
  Allocation --> Filesystem[Filesystem or database volume]
  Filesystem --> Database[Database files, WAL, indexes, temp]

RAID vs LVM vs Filesystem

Common layouts:

Native LVM RAID supports RAID LV types such as raid0, raid1, raid5, raid6, and raid10. LVM uses device mapper for visible LVs and Linux md logic for RAID placement. It also creates hidden sub-LVs for RAID images and metadata, so use lvs -a when troubleshooting.

Useful checks:

lvs -a -o lv_name,segtype,attr,devices,lv_health_status,sync_percent
lvchange --syncaction check /dev/vg_data/lv_db
lvs -o lv_name,raid_sync_action,raid_mismatch_count
vgcfgbackup vg_data

Operational rules:

• If md RAID is below LVM, fix md health before changing LVs or filesystems.
• If LVM native RAID is used, monitor lv_health_status, sync_percent, raid_mismatch_count, and hidden _rimage_ or _rmeta_ sub-LVs.
• Keep vgcfgbackup output with system backups. LVM metadata recovery can restore mappings, but it does not recover overwritten data.
• Be careful with pvmove during degraded RAID or disk errors. It creates extra I/O and can turn weak media into a hard failure.
• Document the stack order in the runbook. During an incident, lsblk and dmsetup ls --tree should explain the same layer order the design document claims.

PostgreSQL Storage Mapping

PostgreSQL cares about durable sync writes, random reads, sequential scans, checkpoints, temporary files, and recovery speed.

Storage-related PostgreSQL settings and choices:

• random_page_cost should reflect the relative cost of random reads. SSD/NVMe usually justify a lower value than slow HDDs after measurement.
• effective_io_concurrency helps PostgreSQL decide how much concurrent I/O to issue for some access paths, especially bitmap heap scans on storage that handles parallel I/O well.
• effective_cache_size should reflect likely OS and database cache available to the planner.
• checkpoint_timeout, max_wal_size, and checkpoint pacing affect bursts of dirty-page writeback.
• wal_compression can trade CPU for lower WAL volume.
• temp_tablespaces can move temporary-file I/O to different storage when that is actually faster and safe.

RAID guidance for PostgreSQL:

• RAID 10 on SSD/NVMe is usually a strong local default for mixed OLTP workloads.
• RAID 1 can work well for smaller systems or dedicated WAL volumes.
• RAID 0 is only reasonable when the database has another full durability layer, such as tested replication plus backups, and the node can be lost.
• Parity RAID can be acceptable for mostly-read data, but write-heavy OLTP often exposes the parity write penalty.
• Battery-backed or power-loss-protected write cache matters; lying write caches can violate durability assumptions.

PostgreSQL examples:

• OLTP primary on four NVMe drives: use RAID 10 for the data directory and pg_wal unless the platform already provides durable replicated block storage. Tune random_page_cost and effective_io_concurrency after measuring real latency.
• Dedicated WAL mirror: place pg_wal on RAID 1 with power-loss-protected SSDs when WAL latency is the bottleneck and data files live on a separate RAID 10 volume.
• Reporting database on large mostly-read data: RAID 6 can be acceptable when query latency targets tolerate parity overhead and rebuild windows, but benchmark checkpoints, index builds, and refresh jobs.
• Temporary-file volume: RAID 0 or ephemeral NVMe may be reasonable for temp_tablespaces only when losing it does not lose committed data and the database can restart cleanly.
• Standby or disposable read replica: RAID 0 can be acceptable if promotion is not required, base backups are current, WAL retention is sufficient, and rebuild time is part of the SLO.

During PostgreSQL storage failure, prefer database-aware recovery over filesystem heroics. Check pg_wal durability, crash recovery logs, pg_isready, replication status, backup manifests, and point-in-time recovery options before deciding that the block device repair was enough.

Elasticsearch Storage Mapping

Elasticsearch stores Lucene segment files in shard directories. Its resilience usually comes from shard replicas on other nodes, snapshots, and cluster allocation, not from one local RAID controller alone.

Storage-sensitive Elasticsearch behaviors:

• indexing writes new segments and later merges them,
• merges create heavy sequential reads and writes,
• searches benefit from filesystem cache,
• shard relocation and recovery can saturate disks,
• snapshots protect against node or cluster loss when tested and retained.

Elasticsearch storage guidance:

• Prefer fast local SSD/NVMe for hot data nodes when low latency matters.
• Size shard counts and replicas so a disk or node failure can recover without overwhelming the cluster.
• RAID 0 can improve one node’s local throughput but makes that node’s data dependent on every member disk; replicas and snapshots must be healthy.
• RAID 1 or RAID 10 reduces single-node disk-loss blast radius but does not replace Elasticsearch replicas.
• Avoid relying on deprecated multiple path.data striping behavior; use a filesystem spanning disks through RAID/LVM or add nodes.
• Watch disk watermarks, merge backlog, recovery traffic, and filesystem cache pressure.

For Elasticsearch, the operational question is not “does this disk have RAID?” It is “can the cluster relocate or restore shards fast enough while serving traffic if this node or volume disappears?”

Elasticsearch examples:

• Hot tier with several data nodes: local NVMe RAID 0 can be reasonable if every index has replicas on other nodes, snapshots are current, and losing one node does not violate capacity or recovery targets.
• Hot tier with fewer nodes or expensive recovery: RAID 10 lowers the chance that one disk failure removes an entire node, while still keeping good merge and search performance.
• Warm tier on capacity disks: RAID 6 may fit mostly-read data with slower recovery expectations, but shard relocation and segment merges can still punish parity arrays.
• Single-node lab or edge deployment: RAID 1 or RAID 10 can reduce local disk-failure downtime, but it is still not a substitute for snapshots because there is no replica elsewhere.
• Kubernetes Elasticsearch: prefer clear node-local or PVC failure domains. A replicated Elasticsearch cluster on top of opaque replicated storage can work, but it complicates performance diagnosis and recovery ownership.

During Elasticsearch storage failure, first decide whether to keep serving from replicas, exclude the node, or stop the node. Check cluster health, unassigned shards, allocation explanations, disk watermarks, snapshot status, and corruption logs. Do not rely on local RAID rebuild alone if shard copies or snapshots are already safer recovery paths.

Choosing a Layout

Study Cards

References

• Linux md RAID documentation
• mdadm(8)
• lvmraid(7)
• PostgreSQL WAL Reliability
• PostgreSQL Resource Consumption
• PostgreSQL Query Planning Configuration
• PostgreSQL WAL Configuration
• Elasticsearch path settings
• Elasticsearch data corruption troubleshooting