Missing mdadm array after reboot

It is heart-attack when you reboot your linux storage server and realized mdadm array gone missing!

• RAID volume not showing in gnome-disks.*
• gparted reports empty partition on all the RAID disks

sudo mdadm --examine --scan -v shows:

mdadm: looking for devices for /dev/md0
mdadm: Cannot assemble mbr metadata on /dev/sda
mdadm: Cannot assemble mbr metadata on /dev/sdb
mdadm: Cannot assemble mbr metadata on /dev/sdc
mdadm: Cannot assemble mbr metadata on /dev/sdd
mdadm: Cannot assemble mbr metadata on /dev/sde

sudo mdadm --examine /dev/sd*  or mdadm --query /dev/sd* shows:

mdadm: No md superblock detected on /dev/sd*

sudo mdadm --assemble /dev/sd[b-f] shows:

mdadm: device /dev/sdb exists but is not an md array.
mdadm: No arrays found in config file or automatically

If you go into /etc/mdadm/mdadm.conf (Debian / Ubuntu) or /etc/mdadm.conf (Fedora), you would realize there is no ARRAY being defined. In some system the mdadm.conf simply does not exist.

No matter how hard you try, the mdadm array just won't show up. There are numerous suggestion on the web, being the most common is to add an auto examine during boot in the mdadm.conf:

 sudo mdadm --examine --scan --config=mdadm.conf >> /etc/mdadm/mdadm.conf

However the above simply does not work for me. I have also try adding the ARRAY manually by doing the following, in which the UUID is the first hdd used for the array (to show the UUID, use blkid. Strange enough I could not get the UUID of my GPT hdd in Ubuntu, and only can get by using Fedora):

ARRAY /dev/md0 metadata=1.2 UUID="0db0c336:f56bd888:2f9e92e4:c1d64c09" >> /etc/mdadm/mdadm.conf

It does not work as well. I have no other solution except try to re-create the array again. Be very careful of this step, I'm more daring to take this step because I already backup my data one day before I lost the array. When you re-create the array, BE SURE to use --assume-clean and make sure the parameter set is EXACTLY the same as what you used to create it the first time. In my case, it is very simple and straightforward:

mdadm --create --assume-clean --level=6 --raid-devices=5 /dev/md0 /dev/sd[b-f]

The RAID array is being created and immediately I got all my data back!! It is advisable to backup your data now and once the backup is done, do a data scrubbing to ensure it is running well.

I hope you will be as lucky as I do, to have all the data back without losing great memory and important works.


External link that I used as reference:

• A very useful read: https://raid.wiki.kernel.org/index.php/RAID_Recovery
• https://hiringsolved.com/blog/goodbye-big-data-mdadm-no-recognizeable-superblock/
• http://ubuntuforums.org/archive/index.php/t-2090527.html
• http://kevin.deldycke.com/2007/03/how-to-recover-a-raid-array-after-having-zero-ized-superblocks/ (be very careful with this, I would not suggest zero out the superblock unless you know what it does)
• http://ubuntuforums.org/showthread.php?t=1950154
• http://superuser.com/questions/603481/how-do-i-reactivate-my-mdadm-raid5-array
• Changing UUID of a mdadm partition: https://itooktheredpill.irgendwo.org/2010/change-uuid-of-a-mdadm-partition/
• This is not useful in my case, but maybe helpful for others: http://askubuntu.com/questions/247981/software-raid-mdadm-re-find-my-array

* If it is shown, then you are lucky because it simply changes its name from /dev/md0 to something like /dev/md127. You can still use it by changing the mounting command to the new array name.

CIFS failed to allocate memory?

If you are mounting your Windows network shares in your Ubuntu, do not hurry to move your files to this newly added mount point. Most probably you will be regret very soon, as the system may give you error Failed to allocate memory and soon after that your files will gone missing. Yes, all gone! That's what happened to me.

The issues seems to cause by Windows rather than Ubuntu itself. There is a good explanation in this link:
https://wiki.archlinux.org/index.php/Samba/Troubleshooting

But I will copy & paste some of the content here, in case the link broken in the future.

======== Copy & Paste from archlinux.org, please contact me if remove is required ========

Windows 7 connectivity problems - mount error(12): cannot allocate memory

A known Windows 7 bug that causes "mount error(12): cannot allocate memory" on an otherwise perfect cifs share on the Linux end can be fixed by setting a few registry keys on the Windows box as follows:

• HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management\LargeSystemCache (set to 1)
• HKLM\SYSTEM\CurrentControlSet\Services\LanmanServer\Parameters\Size (set to 3)

Alternatively, start Command Prompt in Admin Mode and execute the following:

reg add "HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management" /v "LargeSystemCache" /t REG_DWORD /d 1 /f
reg add "HKLM\SYSTEM\CurrentControlSet\Services\LanmanServer\Parameters" /v "Size" /t REG_DWORD /d 3 /f

Do one of the following for the settings to take effect:

• Restart Windows
• Restart the Server service via services.msc
• From the Command Prompt run: 'net stop lanmanserver' and 'net start lanmanserver' - The server may automatically restart after stopping it.

Note: Googling will reveal another tweak recommending users to add a key modifying the "IRPStackSize" size. This is incorrect for fixing this issue under Windows 7. Do not attempt it.

[Ubuntu] Temporarily / Permanently Mount Cifs Network Shares

It is strange that Ubuntu 12.10 doesn't come with cifs-utils installed. Whenever you try to mount a network folder via mount -t cifs -o username=user,password=pass //server-name/shared /mnt/shared, it will give you the following error message:

1:  mount: wrong fs type, bad option, bad superblock on //192.168.0.11/Dropbox,  
2:      missing codepage or helper program, or other error  
3:      (for several filesystems (e.g. nfs, cifs) you might  
4:      need a /sbin/mount.<type> helper program)  
5:      In some cases useful info is found in syslog - try  
6:      dmesg | tail or so  

Execute dmesg | tail will return the following result:

1:  [ 5537.959706] CIFS VFS: Connecting to DFS root not implemented yet  
2:  [ 5537.959815] CIFS VFS: cifs_mount failed w/return code = -22  

To solve this, simply apt-get install cifs-utils. After that run the same command again and you should be able to mount the network shared folder.

To permanently mount the shared folders, go to /etc/fstab, and add the following line:

1:  //server-name/shared     /mnt/shared     cifs     defaults,_netdev,credentials=/home/user/.smbcredentials     0     0  

_netdev parameter is being explained at the last section of this documentation, which in general mean connect only when network is available:
https://help.ubuntu.com/community/MountWindowsSharesPermanently

credentials and /home/user/.smbcredentials is a file created to store the username and password for the remote login, so that we don't put these sensitive info at risk because the /etc/fstab can be viewed by every users on the system. DO NOT forget to set the file permission for the .smbcredentials:

1:  chmod 600 /home/user/.smbcredentials  

Now enjoy working with your network shares!

If you would like to read on, the following articles is good to go:

1. https://help.ubuntu.com/community/Samba/SambaClientGuide
2. https://wiki.ubuntu.com/MountWindowsSharesPermanently
3. https://help.ubuntu.com/community/MountWindowsSharesPermanently

How to deal with space in file path?

In modern computing, we are not restricted to use file/folder name with space between each words. This tends to create trouble if we want to mount the path in /etc/fstab

The solution to this is to add "\040" where the space is. For example:

//192.168.0.2/Shared Folder

has to be written as

//192.168.0.2/Shared\040Folder

[Fedora] Mounting a network shared folder

The command to mount a network shared folder is:

mount -t cifs server:/folder-name /localpath -o username=userid,password=userpass

To unmount it, execute the following:

umount /localpath

It is so funny that the unmount command is 'umount', which is without "n". Wasted my half an hour to figure out this. Might be I really should read fundamental of Linux from scratch, not good to be "plug and play". :(

[Fedora] Mounting a network shared folder under LXDE X11 Desktop

Switching from Xfce to LXDE has made me unable to access my network shared folder. The reason is using Nautilus in Xfce will automatically mount the network folder. But in LXDE, the network folder is not mounted by default, and I need to mount it using command line.

The bad thing is as a novice user, command line doesn't work as smooth. Luckily there is a tools namely "Gigolo" to simplify the procedure. It is located under Application Menu > System Tools > Gigolo. In some case it does not come with LXDE installation, then one would need to manually install it, yum install gigolo.x86_64.

Open up Gigolo we can see a connect button, press it and set the following:

• Service type = Windows Share
• Server = your server hostname / IP address
• Share = the shared folder you would like to mount
• Domain = this can be left blank
• Username = username to access the server

Once enter / connect is pressed, there could be a pop up dialog (single line field) asking for password. For some reason entering password doesn't get rid of this dialog. Simply close it, then there will be another dialog box pop up, this time with two text field - Domain & Password. Simply ignore the Domain field and key in password. Voila! The network folder is now mounted.