Missing mdadm array after reboot

It is heart-attack when you reboot your linux storage server and realized mdadm array gone missing!

• RAID volume not showing in gnome-disks.*
• gparted reports empty partition on all the RAID disks

sudo mdadm --examine --scan -v shows:

mdadm: looking for devices for /dev/md0
mdadm: Cannot assemble mbr metadata on /dev/sda
mdadm: Cannot assemble mbr metadata on /dev/sdb
mdadm: Cannot assemble mbr metadata on /dev/sdc
mdadm: Cannot assemble mbr metadata on /dev/sdd
mdadm: Cannot assemble mbr metadata on /dev/sde

sudo mdadm --examine /dev/sd*  or mdadm --query /dev/sd* shows:

mdadm: No md superblock detected on /dev/sd*

sudo mdadm --assemble /dev/sd[b-f] shows:

mdadm: device /dev/sdb exists but is not an md array.
mdadm: No arrays found in config file or automatically

If you go into /etc/mdadm/mdadm.conf (Debian / Ubuntu) or /etc/mdadm.conf (Fedora), you would realize there is no ARRAY being defined. In some system the mdadm.conf simply does not exist.

No matter how hard you try, the mdadm array just won't show up. There are numerous suggestion on the web, being the most common is to add an auto examine during boot in the mdadm.conf:

 sudo mdadm --examine --scan --config=mdadm.conf >> /etc/mdadm/mdadm.conf

However the above simply does not work for me. I have also try adding the ARRAY manually by doing the following, in which the UUID is the first hdd used for the array (to show the UUID, use blkid. Strange enough I could not get the UUID of my GPT hdd in Ubuntu, and only can get by using Fedora):

ARRAY /dev/md0 metadata=1.2 UUID="0db0c336:f56bd888:2f9e92e4:c1d64c09" >> /etc/mdadm/mdadm.conf

It does not work as well. I have no other solution except try to re-create the array again. Be very careful of this step, I'm more daring to take this step because I already backup my data one day before I lost the array. When you re-create the array, BE SURE to use --assume-clean and make sure the parameter set is EXACTLY the same as what you used to create it the first time. In my case, it is very simple and straightforward:

mdadm --create --assume-clean --level=6 --raid-devices=5 /dev/md0 /dev/sd[b-f]

The RAID array is being created and immediately I got all my data back!! It is advisable to backup your data now and once the backup is done, do a data scrubbing to ensure it is running well.

I hope you will be as lucky as I do, to have all the data back without losing great memory and important works.


External link that I used as reference:

• A very useful read: https://raid.wiki.kernel.org/index.php/RAID_Recovery
• https://hiringsolved.com/blog/goodbye-big-data-mdadm-no-recognizeable-superblock/
• http://ubuntuforums.org/archive/index.php/t-2090527.html
• http://kevin.deldycke.com/2007/03/how-to-recover-a-raid-array-after-having-zero-ized-superblocks/ (be very careful with this, I would not suggest zero out the superblock unless you know what it does)
• http://ubuntuforums.org/showthread.php?t=1950154
• http://superuser.com/questions/603481/how-do-i-reactivate-my-mdadm-raid5-array
• Changing UUID of a mdadm partition: https://itooktheredpill.irgendwo.org/2010/change-uuid-of-a-mdadm-partition/
• This is not useful in my case, but maybe helpful for others: http://askubuntu.com/questions/247981/software-raid-mdadm-re-find-my-array

* If it is shown, then you are lucky because it simply changes its name from /dev/md0 to something like /dev/md127. You can still use it by changing the mounting command to the new array name.