dd if=/dev/sd${OLD} of=/dev/sd${NEW} # where you can replace your drives with the ones fitting your case
And here is where I got lucky! I managed to copy the whole drive on the new drive. But I wasn't not that lucky. When I put the drive back in the array. BOOHOOO! My RAID5 had 2 good drives now (one of them was the new one) but the third one was a SPARE!!! So, I still couldn't assemble the array. But there is a catch, I could rebuild the array without touching the data... Here is how it was looking before:
cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md0 : inactive sdb3[4](S) sda3[0] sdc3[2]
XXXXXXXXXX blocks
As you can see, the sdb3 is seen as a SPARE! Now, that is wrong, and it shouldn't be. So, I had to risk, declare the whole array as wrong and hope that the SPARE could actually have the information to get me out of this stall state. So, here is how I activated the SPARE:
mdadm --stop /dev/md0 # Stop the md0 array mdadm -Cv /dev/md0 --assume-clean --level=5 --layout=left-symmetric --chunk=64 --raid-devices=4 /dev/sda3 /dev/sdb3 /dev/sdc3 missing # Remember, sdb was a SPARE and the sdd was GONE :) [...CUT...] Continue creating array? yes # I answered here with yes... mdadm: array /dev/md0 started.
After that, I stopped, took a break (or a deep breath) and then looked into the array status:
cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md0 : active raid5 sdc3[2] sdb3[1] sda3[0]
4391961792 blocks level 5, 64k chunk, algorithm 2 [4/3] [UUU_]
You can see that the last one is gone missing (is marked as "_"). Better way to look at it is via:
mdadm --detail /dev/md0
/dev/md0:
Version : [...CUT...]
Creation Time : [...CUT...]
Raid Level : raid5
Array Size : [...CUT...]
Used Dev Size : [...CUT...]
Raid Devices : 4
Total Devices : 3
Preferred Minor : 0
Persistence : Superblock is persistent
Update Time : [...CUT...]
State : clean, degraded
Active Devices : 3
Working Devices : 3
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 64K
UUID : [...CUT...]
Events : [...CUT...]
Number Major Minor RaidDevice State
0 8 3 0 active sync /dev/sda3
1 8 19 1 active sync /dev/sdb3
2 8 35 2 active sync /dev/sdc3
3 0 0 3 removed
Next, all you need to do is add another clean drive (no partition table, you can wipe it out with dd if=/dev/zero of=/dev/sd${CLEAN} bs=512 count=8) and then make sure to let the array md0 know about it like this:
mdadm -a /dev/md0 /dev/sdd3 mdadm: added /dev/sdd3
Now, if you have problems with the partition of the drive, you can replicate the partition from one of the active drives (in my case, /dev/sda as master, and /dev/sdd as the target) like this:
sfdisk -d /dev/sda | sfdisk /dev/sdd
At the end of the process, you should see something like this:
mdadm --detail /dev/md0
/dev/md0:
Version : [...CUT...]
Creation Time : [...CUT...]
Raid Level : raid5
Array Size : [...CUT...]
Used Dev Size : [...CUT...]
Raid Devices : 4
Total Devices : 4
Preferred Minor : 0
Persistence : Superblock is persistent
Update Time : [...CUT...]
State : clean, degraded, recovering
Active Devices : 3
Working Devices : 4
Failed Devices : 0
Spare Devices : 1
Layout : left-symmetric
Chunk Size : 64K
Rebuild Status : 0% complete
UUID : [...CUT...]
Events : 0.8
Number Major Minor RaidDevice State
0 8 3 0 active sync /dev/sda3
1 8 19 1 active sync /dev/sdb3
2 8 35 2 active sync /dev/sdc3
4 8 51 3 spare rebuilding /dev/sdd3
And if you want to go closer to the kernel, you can always enter the sysfs like this:
cd /sys/block/md0/md ls -al [...CUT...]
You can sync, (echo check > sync_action) etc...
Well, that is all folks. I hope it gives other lost souls (or damage raid5 disks unlucky ones) a good idea how to recover the data. Drop a comment if you have questions or additions...
