Missing mdadm array after reboot

It is heart-attack when you reboot your linux storage server and realized mdadm array gone missing!

• RAID volume not showing in gnome-disks.*
• gparted reports empty partition on all the RAID disks

sudo mdadm --examine --scan -v shows:

mdadm: looking for devices for /dev/md0
mdadm: Cannot assemble mbr metadata on /dev/sda
mdadm: Cannot assemble mbr metadata on /dev/sdb
mdadm: Cannot assemble mbr metadata on /dev/sdc
mdadm: Cannot assemble mbr metadata on /dev/sdd
mdadm: Cannot assemble mbr metadata on /dev/sde

sudo mdadm --examine /dev/sd*  or mdadm --query /dev/sd* shows:

mdadm: No md superblock detected on /dev/sd*

sudo mdadm --assemble /dev/sd[b-f] shows:

mdadm: device /dev/sdb exists but is not an md array.
mdadm: No arrays found in config file or automatically

If you go into /etc/mdadm/mdadm.conf (Debian / Ubuntu) or /etc/mdadm.conf (Fedora), you would realize there is no ARRAY being defined. In some system the mdadm.conf simply does not exist.

No matter how hard you try, the mdadm array just won't show up. There are numerous suggestion on the web, being the most common is to add an auto examine during boot in the mdadm.conf:

 sudo mdadm --examine --scan --config=mdadm.conf >> /etc/mdadm/mdadm.conf

However the above simply does not work for me. I have also try adding the ARRAY manually by doing the following, in which the UUID is the first hdd used for the array (to show the UUID, use blkid. Strange enough I could not get the UUID of my GPT hdd in Ubuntu, and only can get by using Fedora):

ARRAY /dev/md0 metadata=1.2 UUID="0db0c336:f56bd888:2f9e92e4:c1d64c09" >> /etc/mdadm/mdadm.conf

It does not work as well. I have no other solution except try to re-create the array again. Be very careful of this step, I'm more daring to take this step because I already backup my data one day before I lost the array. When you re-create the array, BE SURE to use --assume-clean and make sure the parameter set is EXACTLY the same as what you used to create it the first time. In my case, it is very simple and straightforward:

mdadm --create --assume-clean --level=6 --raid-devices=5 /dev/md0 /dev/sd[b-f]

The RAID array is being created and immediately I got all my data back!! It is advisable to backup your data now and once the backup is done, do a data scrubbing to ensure it is running well.

I hope you will be as lucky as I do, to have all the data back without losing great memory and important works.


External link that I used as reference:

• A very useful read: https://raid.wiki.kernel.org/index.php/RAID_Recovery
• https://hiringsolved.com/blog/goodbye-big-data-mdadm-no-recognizeable-superblock/
• http://ubuntuforums.org/archive/index.php/t-2090527.html
• http://kevin.deldycke.com/2007/03/how-to-recover-a-raid-array-after-having-zero-ized-superblocks/ (be very careful with this, I would not suggest zero out the superblock unless you know what it does)
• http://ubuntuforums.org/showthread.php?t=1950154
• http://superuser.com/questions/603481/how-do-i-reactivate-my-mdadm-raid5-array
• Changing UUID of a mdadm partition: https://itooktheredpill.irgendwo.org/2010/change-uuid-of-a-mdadm-partition/
• This is not useful in my case, but maybe helpful for others: http://askubuntu.com/questions/247981/software-raid-mdadm-re-find-my-array

* If it is shown, then you are lucky because it simply changes its name from /dev/md0 to something like /dev/md127. You can still use it by changing the mounting command to the new array name.

[Ubuntu] The bad day!! How to recover from dmraid

Today is a real bad day, one of the Ubuntu server that I "managed" (it's not me who installed it) has gone down! There could be two reasons:

1. Incompatible version of Zentyal (Zentyal 3.0 was accidentally installed on this 12.10, causing the samba not functioning)
2. The electrical leak in the office due to DP box does not have proper earth line installed. Damn it, the whole rack is covered by electricity, and I got an electric shock! Luckily it was not at high current else all devices and I would have died =_=!

Well.. after hours of troubleshooting, I could not bring the server up and running again. It halted at different stage of loading, sometimes failed to load Samba, sometimes failed at Checking Quota...

I do suspect it is caused by the Zentyal... but time is running out. I rush back to home, taking my laptop SSD* installed with Ubuntu 12.10, together with as many as HDD I could found available for the backup**, and plugged into the server.

Voila! Now I'm inside my laptop Ubuntu, and by running dmraid -r, I could now see the RAID under name isw_xxxxx (isw is a prefix for Intel RST). By running dmraid -ay (activate with silent mode YES), I can now see the whole RAID in Nautilus.

I'm backing up those file now... hopefully the files and directories in the server are still in good condition. Losing a single files will be disaster. Cross finger!

=======================

* Why SSD? Because the Ubuntu server was installed using dmraid and Intel RST (Rapid Storage Technology), so SSD can work well as most of the time SSD is used under AHCI. SATA drive that previously installed without AHCI would fail to boot up.

** Please make sure you use a reliable HDD for the backup. You won't like to fail the second time where you could be only having once chance to perform the recovery.

*** Regarding the mdadm and dmraid, here are some good reads:

• https://help.ubuntu.com/community/FakeRaidHowto
• http://raid6.com.au/posts/mdadm_advantages/
• http://www.sourceware.org/lvm2/wiki/DMRAID

[Ubuntu] How to configure Smartmontools to send email report

There are a few steps-by-steps tutorial to teach you how to configure Smartmontools to email report to you when drives failed. But to get it works is may not be as simple as complete the steps written.

Especially for Ubuntu users, most of you that encountered this problem may have followed the instruction here:
https://help.ubuntu.com/community/Smartmontools

The documentation is very good indeed and it explains almost everything. However, it forgot to mention that if you are not sending email to root account, you are actually sending email externally and the mail simply may not work on that.

How to tackle this has been documented here:
https://wiki.archlinux.org/index.php/S.M.A.R.T.
*Despite it is for Arch Linux, but it works fine to configure Ubuntu.

Basically, you would have to configure ssmtp or any other MTA (Mail Transport Agent) so that Ubuntu can use it to send email to you.

The setup is rather simple, you can get guidance from:
https://wiki.archlinux.org/index.php/SSMTP

I hope this post will help any of you solve this issue quickly, need not to be like me wasting few hours to searching for solution just to get this simple function work.

[Fedora] HDD Benchmark with hdparm

To do a simple reading benchmark of Hdd, we can use hdparm. It may not install by default but we can easily get it by yum install hdparm.

After install, run hdparm -t -T /dev/sda (sdb, sdc, etc) and we can then see Cache Read and Buffered Disk Read speed.

My laptop has pretty poor performance with only 58.3MB/s for buffered disk reads, running on WD Scropio Blue 320GB.