Recovering a NAS RAID with mdadm and LVM

A consumer NAS had died, but the disks were fine. These boxes almost always run Linux underneath, laying the data out as a Linux software RAID (md) with LVM on top, so you can pull a drive, attach it to any Linux machine and read it directly. I did this from a Kali live system, but any recent Debian or Ubuntu will do.

The one rule throughout: mount everything read only. If the array is degraded you do not want the recovery host writing to it.

Tools

apt-get install mdadm lvm2
modprobe dm-mod

Look at what is on the disk

Plug the disk in and see the partitions:

lsblk

A NAS disk has a lot of small partitions (the firmware, swap, config and so on) and one big one for your data. file -s on the data partition confirms it is a RAID member:

file -s /dev/sdb10

/dev/sdb10: Linux Software RAID version 1.2 (1) UUID=... level=1 disks=2

Now read the RAID superblock. The key field is Data Offset, which is how far into the partition the actual filesystem starts:

mdadm --examine /dev/sdb10

/dev/sdb10:
        Version : 1.2
     Raid Level : raid1
   Raid Devices : 2
  Avail Dev Size : 3898521911 (1858.96 GiB 1996.04 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
   Device Role : Active device 0
   Array State : A. ('A' == active, '.' == missing, 'R' == replacing)

That A. means device 0 is present and its mirror is missing. For a RAID 1 that is fine, as one half of a mirror is a complete copy. The data is all there.

Option 1: let mdadm assemble it

The clean way is to let mdadm assemble the array from the member, force it to run even though it is degraded, then bring up the LVM volume group that lives inside it:

mdadm -A -R /dev/md9 /dev/sdb10
vgchange -ay
mkdir -p /mnt/recovery
mount /dev/<vg-name>/<lv-name> /mnt/recovery -o ro,user

vgchange -ay activates whatever volume group it finds; lsblk or vgs afterwards will show you the volume group and logical volume names to use in the mount.

Option 2: skip mdadm and go straight to the data

Sometimes mdadm will not assemble the array (wrong metadata version, no matching mirror, a fussy live environment). Because this is RAID 1, the filesystem sits as a plain copy at the data offset, so you can expose it as a loop device and let LVM find it directly.

Take the Data Offset from mdadm --examine and multiply by the 512-byte sector size. For an offset of 2048 sectors that is 2048 * 512:

losetup --find --show --read-only --offset $((2048*512)) /dev/sdb10
vgscan
vgchange -ay
mount /dev/<vg-name>/<lv-name> /mnt/recovery -o ro,user

Older firmware used a much smaller offset; on one older box it was 272 sectors, so always read the real value from the superblock rather than assuming.

When the filesystem will not mount cleanly

If the kernel driver refuses the filesystem (a slightly nonstandard ext variant, or minor corruption), fuseext2 will often mount it read only where the kernel will not:

fuseext2 -o ro -o sync_read /dev/mapper/<vg-name>-<lv-name> /mnt/recovery

Get the data off

Once it is mounted, copy everything to a known-good disk and walk away. rsync with --progress lets you see it working and resume if it stops:

mkdir /backup
mount /dev/sdc1 /backup
rsync -a --progress /mnt/recovery/ /backup/

The whole job comes down to three layers in order: the partition, the md RAID on top of it, and the LVM volume inside that. Peel them back one at a time, keep it read only, and a dead NAS is just a disk full of files.