Recovering from a major RAID5 with 2 disks failing out of 4

Yes. The title is a bit unrealistic, because if you have 2 disks failing out of 4, you are out of business. But I was lucky. One disk was failing into WRITE (at least that is what S.M.A.R.T. was saying). The other one was failing into READ (well, I didn't need S.M.A.R.T. to tell me anything, it just failed!). So, how did I manage (and believe me, this was a long, sleepless night, full of events). First, I took the disk that was failing WRITE commands and put it in another computer where I had some extra, unused disks (well, I just have them hanging there...). There, I copied the disk using dd (ah, you need Linux for this to work! Is there anything else but Linux for these kind of jobs anyway?):

dd if=/dev/sd${OLD} of=/dev/sd${NEW} # where you can replace your drives with the ones fitting your case

And here is where I got lucky! I managed to copy the whole drive on the new drive. But I wasn't not that lucky. When I put the drive back in the array. BOOHOOO! My RAID5 had 2 good drives now (one of them was the new one) but the third one was a SPARE!!! So, I still couldn't assemble the array. But there is a catch, I could rebuild the array without touching the data... Here is how it was looking before:

cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]       
md0 : inactive sdb3[4](S) sda3[0] sdc3[2]
      XXXXXXXXXX blocks

As you can see, the sdb3 is seen as a SPARE! Now, that is wrong, and it shouldn't be. So, I had to risk, declare the whole array as wrong and hope that the SPARE could actually have the information to get me out of this stall state. So, here is how I activated the SPARE:

mdadm --stop /dev/md0 # Stop the md0 array
mdadm -Cv /dev/md0 --assume-clean --level=5 --layout=left-symmetric --chunk=64 --raid-devices=4 /dev/sda3 /dev/sdb3 /dev/sdc3 missing # Remember, sdb was a SPARE and the sdd was GONE :)
[...CUT...]
Continue creating array? yes # I answered here with yes...
mdadm: array /dev/md0 started.

After that, I stopped, took a break (or a deep breath) and then looked into the array status:

cat /proc/mdstat                                                                                                                     
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] 
md0 : active raid5 sdc3[2] sdb3[1] sda3[0]
      4391961792 blocks level 5, 64k chunk, algorithm 2 [4/3] [UUU_]

You can see that the last one is gone missing (is marked as "_"). Better way to look at it is via:

mdadm --detail /dev/md0                                                                                                              
/dev/md0:
        Version : [...CUT...]
  Creation Time : [...CUT...]
     Raid Level : raid5
     Array Size : [...CUT...]
  Used Dev Size : [...CUT...]
   Raid Devices : 4
  Total Devices : 3
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : [...CUT...]
          State : clean, degraded
 Active Devices : 3
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 64K

           UUID : [...CUT...]
         Events : [...CUT...]

    Number   Major   Minor   RaidDevice State
       0       8        3        0      active sync   /dev/sda3
       1       8       19        1      active sync   /dev/sdb3
       2       8       35        2      active sync   /dev/sdc3
       3       0        0        3      removed

Next, all you need to do is add another clean drive (no partition table, you can wipe it out with dd if=/dev/zero of=/dev/sd${CLEAN} bs=512 count=8) and then make sure to let the array md0 know about it like this:

mdadm -a /dev/md0 /dev/sdd3
mdadm: added /dev/sdd3

Now, if you have problems with the partition of the drive, you can replicate the partition from one of the active drives (in my case, /dev/sda as master, and /dev/sdd as the target) like this:

sfdisk -d /dev/sda | sfdisk /dev/sdd

At the end of the process, you should see something like this:

mdadm --detail /dev/md0 
/dev/md0:
        Version : [...CUT...]
  Creation Time : [...CUT...]
     Raid Level : raid5
     Array Size : [...CUT...]
  Used Dev Size : [...CUT...]
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : [...CUT...]
          State : clean, degraded, recovering
 Active Devices : 3
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 1

         Layout : left-symmetric
     Chunk Size : 64K

 Rebuild Status : 0% complete

           UUID : [...CUT...]
         Events : 0.8

    Number   Major   Minor   RaidDevice State
       0       8        3        0      active sync   /dev/sda3
       1       8       19        1      active sync   /dev/sdb3
       2       8       35        2      active sync   /dev/sdc3
       4       8       51        3      spare rebuilding   /dev/sdd3

And if you want to go closer to the kernel, you can always enter the sysfs like this:

cd /sys/block/md0/md
ls -al
[...CUT...]

You can sync, (echo check > sync_action) etc...

Well, that is all folks. I hope it gives other lost souls (or damage raid5 disks unlucky ones) a good idea how to recover the data. Drop a comment if you have questions or additions...

OpenSSH key, Amazon EC2 RSA, Android and dropbear (from Terminal IDE)

Here is the thing: I have quite complicated SSH key (uses AES-128-CBC) and while there are posts about converting the openssh key to dropbear format, they fail to mention that the openssh key needs to be in some PEM (or more streamline) format. That might seam easy, but when you try to convert using the common example, you might end up getting an error like this:

/usr/lib/dropbear/dropbearconvert openssh dropbear ~/.ssh/id_rsa /tmp/id_rsa.db
Error: Ciphers other than DES-EDE3-CBC not supported
Error reading key from '~/.ssh/id_rsa'

That happens especially if you have keys generate by the Amazon Web Services, or EC2. Those generally look like this:

head -3 ~/.ssh/id_rsa_amazon_ec2
-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: AES-128-CBC,1234567890ABCSFTDHIJKLMNOPQRSTUV

Now, that sucks! And google-ing didn't get me very far. So, if you have no idea what to make from that error, well, fear not, as there is a simple solution, coming from openssl. You will need to convert your key to PEM and then, use the dropbearconvert tool like this:

openssl rsa -in ~/.ssh/id_rsa -outform pem > ~/.ssh/id_rsa.pem

You will need to enter your pass phrase if using one:

Enter pass phrase for id_rsa:
writing RSA key

At this point, technically, that key is the same but you have removed the passphrase (you could do the same by using ssh-keygen -p -f ~/.ssh/id_rsa, but then, you will have to put it back).

And now, using the converter on the PEM:

/usr/lib/dropbear/dropbearconvert openssh dropbear ~/.ssh/id_rsa.pem ~/.ssh/id_rsa.db

And you will get your key:

Key is a RSA key
Wrote key to '~/.ssh/id_rsa.db'

Hope this helps someone like me that uses the Amazon generated SSH keys and needs them in the Android side (by the way, I needed it for use with Terminal IDE ssh, which I love)... NOTE: You should remove that file from your system as it is password-less. This could allow someone to use it without knowing your pass phrase!