Disk failure and RAID recovery

Spotting the failure: read /proc/mdstat and mdadm —detail.

# Immediate check when alert fires
cat /proc/mdstat
# Personalities : [raid5]
# md0 : active raid5 sde[3] sdc[1] sdb[0]
#       3145728 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/3] [UUU_]
#
# Decode:
# [4/3]   = expected 4 disks, only 3 active
# [UUU_]  = positions 0,1,3 up; position 2 (sdd) is missing — it is the failed disk
# Note: sdd is absent from the device list entirely

# More detail
sudo mdadm --detail /dev/md0
# State : clean, degraded          <-- still serving data, but no redundancy
# Active Devices : 3
# Failed Devices : 1
#
# Number   Major   Minor   RaidDevice State
#    0     8:16      0      active sync   /dev/sdb
#    1     8:32      1      active sync   /dev/sdc
#    -     0:0       2      removed       <-- slot 2 is the missing disk
#    3     8:80      3      active sync   /dev/sde

# Also check kernel logs for the failure event and any I/O errors
sudo journalctl -b | grep -i 'md0\|raid\|sdd\|I/O error' | tail -30
# Look for: "md/raid:md0: Disk failure on sdd, disabling device"
# or: "blk_update_request: I/O error, dev sdd"

Mark the failed disk as faulty (if not already done by the kernel).

The kernel often auto-fails a disk when it gets repeated I/O errors. If the disk shows as removed in mdadm --detail the kernel already did this. If it shows as faulty you must explicitly remove it. If it is neither — e.g. the disk is physically missing after a hot-swap — the slot is already clean.

# Check the disk's state in the array
sudo mdadm --detail /dev/md0 | grep sdd
# (if it still appears with state 'faulty')
#    2     8:48      2      faulty   /dev/sdd

# Explicitly mark as failed (required before --remove if kernel did not auto-fail it)
sudo mdadm /dev/md0 --fail /dev/sdd
# mdadm: set /dev/sdd faulty in /dev/md0

# Now remove it from the array (the slot stays — just the disk leaves)
sudo mdadm /dev/md0 --remove /dev/sdd
# mdadm: hot removed /dev/sdd from /dev/md0

# Confirm it is gone
sudo mdadm --detail /dev/md0 | grep -E 'State|Devices|RaidDevice'
# State : clean, degraded
# Active Devices : 3
# Failed Devices : 0   <-- cleaned up
# Removed Devices : 1

Hot-add the replacement disk and start the rebuild.

# Physical step: insert the new disk. In a hot-swap bay this is live.
# In a VM/cloud: attach the new block device.
# Verify the OS sees it:
lsblk | grep -v dm
# sdb    8:16   0  1T  0 disk
# sdc    8:32   0  1T  0 disk
# sde    8:80   0  1T  0 disk
# sdf    8:96   0  1T  0 disk   <-- new disk

# The new disk must be at least as large as the failed disk
# (mdadm uses the smallest disk's size for the array)

# Add it to the array — rebuild starts automatically
sudo mdadm /dev/md0 --add /dev/sdf
# mdadm: added /dev/sdf

# Watch the rebuild
watch -n5 cat /proc/mdstat
# md0 : active raid5 sdf[4] sde[3] sdc[1] sdb[0]
#       3145728 blocks super 1.2 level 5 [4/3] [UUU_]
#       [=>...................]  recovery = 7.3% (...)
#       finish=142.3min speed=198400K/sec
#
# Progress: the _ position fills as recovery proceeds
# Once complete:
# md0 : active raid5 sdf[4] sde[3] sdc[1] sdb[0]
#       3145728 blocks super 1.2 level 5 [4/4] [UUUU]
# [4/4] [UUUU] = fully rebuilt, all disks healthy

The rebuild window: why this is the highest-risk moment.

During rebuild, the array must read every block on every surviving disk to reconstruct the missing disk’s data. This is maximum sustained I/O on disks that may already be aged. Two risks converge:

# Risk 1: Unrecoverable Read Error (URE)
# Modern enterprise drives: URE rate ~1 in 10^15 bits read
# Consumer drives: URE rate ~1 in 10^14 bits read
#
# RAID 5 with 4 x 4 TB drives: total data = 12 TB = ~10^14 bits
# On consumer drives: roughly 1 URE expected per rebuild of this size
# A URE during rebuild = data that cannot be reconstructed = partial corruption
#
# This is why large RAID 5 arrays on consumer drives are not recommended.
# RAID 6 tolerates a URE during rebuild (second parity covers it).

# Risk 2: Second disk failure during rebuild
# The surviving disks are under maximum read load for hours.
# Disks from the same batch often fail at similar ages.
# A second failure during rebuild = total data loss on RAID 5.

# What to do during the rebuild window:
# 1. Do NOT add more I/O load than necessary — pause non-critical jobs
# 2. Monitor /proc/mdstat every few minutes
# 3. Watch kernel logs for any new I/O errors on surviving disks:
sudo journalctl -f | grep -i 'I/O error\|error.*sd'
# 4. Have your backup restore procedure ready — just in case

# You can throttle rebuild speed to reduce load on surviving disks:
# (at the cost of a longer rebuild window)
echo 50000 | sudo tee /proc/sys/dev/raid/speed_limit_max
# Default is usually 200000 KB/s; 50000 reduces disk stress at the cost of time

After rebuild: verify and update mdadm config.

# Confirm the array is fully healthy
sudo mdadm --detail /dev/md0
# State : clean                    <-- no longer 'degraded'
# Active Devices : 4
# Failed Devices : 0
# Spare Devices : 0
#
# Number   Major   Minor   RaidDevice State
#    0     8:16      0      active sync   /dev/sdb
#    1     8:32      1      active sync   /dev/sdc
#    4     8:96      2      active sync   /dev/sdf  <-- new disk in slot 2
#    3     8:80      3      active sync   /dev/sde

# Update mdadm.conf to reflect the new disk
sudo mdadm --detail --scan | sudo tee /etc/mdadm/mdadm.conf
sudo update-initramfs -u
# Without this: a reboot may not reassemble the array correctly

# Run an array check to verify all parity is consistent
echo check | sudo tee /sys/block/md0/md/sync_action
# Monitor:
cat /sys/block/md0/md/sync_completed
# When done, mismatch_cnt should be 0:
cat /sys/block/md0/md/mismatch_cnt
# 0   <-- good; nonzero = parity inconsistencies found (investigate)

# Schedule regular checks (mdadm ships a cron script for this):
# /etc/cron.d/mdadm — runs a check monthly by default on Debian/Ubuntu