Mdadm Must Be Configured on Partition

Refer to my previous post Missing mdadm array after reboot, I would like to add on the reason why auto-scan does not work once system rebooted - it requires the RAID to be configured on a partition instead of the disks!

This is going to have problem:

mdadm --create --level=6 --raid-devices=5 /dev/md0 /dev/sd[b-f]

 and this would be correct, please take note of the digit at the end of line:

mdadm --create --level=6 --raid-devices=5 /dev/md0 /dev/sd[b-f]1

It means you must format all the disks with at least one partition, and building the RAID using the partition.

I  hope this would be helpful for novice users who trying to get the RAID run properly.

Problem Updating Repo List in Ubuntu 12.10

The sources list has been changed to old-releases. To replace the sources list easily, use the following command:

sudo sed -i -e 's/archive.ubuntu.com\|security.ubuntu.com/old-releases.ubuntu.com/g' /etc/apt/sources.list

If you have choose another mirror then your sources list might contain us.archive.ubuntu.com, then the above changes would still be invalid. You would need to remove the us. from it. Using the command below will help remove it:

sudo sed -i -e 's/us.old-releases.ubuntu.com/old-releases.ubuntu.com/g' /etc/apt/sources.list

Houdini Start-Up Crashed on Render Node

AFAIK, Houdini could not start its main application (GUI) without OpenGL enabled graphics card. Hence if you are trying to start it in a render node with ASPEED on-board graphics, it is impossible. All you can do is to run the render using CLI.

As for user encounters the same issue, if you are using Nvidia graphics card, you would need to switch the XORG graphics driver to Nvidia proprietary driver. You can do it at:

Menu > Settings > Software and Update > Additional Drivers

Sometimes the latest driver may not work well. So you may have to tried different version to get it work.

Tips: To get a clearer picture of why Houdini failed to start, instead of running it from software menu, you can run it from terminal. The error message will be more readable than the crashed dump file.

Missing mdadm array after reboot

It is heart-attack when you reboot your linux storage server and realized mdadm array gone missing!

• RAID volume not showing in gnome-disks.*
• gparted reports empty partition on all the RAID disks

sudo mdadm --examine --scan -v shows:

mdadm: looking for devices for /dev/md0
mdadm: Cannot assemble mbr metadata on /dev/sda
mdadm: Cannot assemble mbr metadata on /dev/sdb
mdadm: Cannot assemble mbr metadata on /dev/sdc
mdadm: Cannot assemble mbr metadata on /dev/sdd
mdadm: Cannot assemble mbr metadata on /dev/sde

sudo mdadm --examine /dev/sd*  or mdadm --query /dev/sd* shows:

mdadm: No md superblock detected on /dev/sd*

sudo mdadm --assemble /dev/sd[b-f] shows:

mdadm: device /dev/sdb exists but is not an md array.
mdadm: No arrays found in config file or automatically

If you go into /etc/mdadm/mdadm.conf (Debian / Ubuntu) or /etc/mdadm.conf (Fedora), you would realize there is no ARRAY being defined. In some system the mdadm.conf simply does not exist.

No matter how hard you try, the mdadm array just won't show up. There are numerous suggestion on the web, being the most common is to add an auto examine during boot in the mdadm.conf:

 sudo mdadm --examine --scan --config=mdadm.conf >> /etc/mdadm/mdadm.conf

However the above simply does not work for me. I have also try adding the ARRAY manually by doing the following, in which the UUID is the first hdd used for the array (to show the UUID, use blkid. Strange enough I could not get the UUID of my GPT hdd in Ubuntu, and only can get by using Fedora):

ARRAY /dev/md0 metadata=1.2 UUID="0db0c336:f56bd888:2f9e92e4:c1d64c09" >> /etc/mdadm/mdadm.conf

It does not work as well. I have no other solution except try to re-create the array again. Be very careful of this step, I'm more daring to take this step because I already backup my data one day before I lost the array. When you re-create the array, BE SURE to use --assume-clean and make sure the parameter set is EXACTLY the same as what you used to create it the first time. In my case, it is very simple and straightforward:

mdadm --create --assume-clean --level=6 --raid-devices=5 /dev/md0 /dev/sd[b-f]

The RAID array is being created and immediately I got all my data back!! It is advisable to backup your data now and once the backup is done, do a data scrubbing to ensure it is running well.

I hope you will be as lucky as I do, to have all the data back without losing great memory and important works.


External link that I used as reference:

• A very useful read: https://raid.wiki.kernel.org/index.php/RAID_Recovery
• https://hiringsolved.com/blog/goodbye-big-data-mdadm-no-recognizeable-superblock/
• http://ubuntuforums.org/archive/index.php/t-2090527.html
• http://kevin.deldycke.com/2007/03/how-to-recover-a-raid-array-after-having-zero-ized-superblocks/ (be very careful with this, I would not suggest zero out the superblock unless you know what it does)
• http://ubuntuforums.org/showthread.php?t=1950154
• http://superuser.com/questions/603481/how-do-i-reactivate-my-mdadm-raid5-array
• Changing UUID of a mdadm partition: https://itooktheredpill.irgendwo.org/2010/change-uuid-of-a-mdadm-partition/
• This is not useful in my case, but maybe helpful for others: http://askubuntu.com/questions/247981/software-raid-mdadm-re-find-my-array

* If it is shown, then you are lucky because it simply changes its name from /dev/md0 to something like /dev/md127. You can still use it by changing the mounting command to the new array name.

Ubuntu Boot to Blank Screen with Blinking Underscore (Cursor)

Today I was installing the Ubuntu 14.04.02 LTS into a server using a USB thumbdrive. Everything went smooth, as usual the setup asked to remove the USB thumbdrive before continue to Reboot.

After reboot, it hangs on a black screen with blinking underscore / cursor. Waited for 15 mins, nothing happen. If you encounter the same error like I do, it is possibly caused by:

1. a corrupted GRUB
2. the setup simply installed the GRUB onto the USB thumbdrive accidentally. (I suspect it is because I have chosen to umount the thumbdrive during the setup)
3. graphics driver not supported (it sounds a bit weird to me, if the setup could run well in GUI, why should a reboot not?)

For the first two reasons, a simple solution is to repair the GRUB. Here's the complete guide for Ubuntu users. For my case, because I knew the GRUB is in the thumbdrive, so I simply plug it in and choose my HDD as first boot device, it then boot into Ubuntu successfully, then I run the following command in terminal:

sudo add-apt-repository ppa:yannubuntu/boot-repair
sudo apt-get update
sudo apt-get install -y boot-repair && boot-repair

and then follow what is being instructed by the boot-repair. The only steps that caused a confusion is which disk or partition to select as GRUB install location. A rule of thumbs is don't select the partition, select the disk.

For 3rd reason - graphics driver broken. I personally do not try it, but someone from online said it can be fix by inserting the -nomodeset in between ro quiet splash. And I do believe you realize one thing now... if you could see the GRUB and able to insert the -nomodeset, well, then it is most likely your problem is more challenging, good luck hunting it down!

CIFS failed to allocate memory?

If you are mounting your Windows network shares in your Ubuntu, do not hurry to move your files to this newly added mount point. Most probably you will be regret very soon, as the system may give you error Failed to allocate memory and soon after that your files will gone missing. Yes, all gone! That's what happened to me.

The issues seems to cause by Windows rather than Ubuntu itself. There is a good explanation in this link:
https://wiki.archlinux.org/index.php/Samba/Troubleshooting

But I will copy & paste some of the content here, in case the link broken in the future.

======== Copy & Paste from archlinux.org, please contact me if remove is required ========

Windows 7 connectivity problems - mount error(12): cannot allocate memory

A known Windows 7 bug that causes "mount error(12): cannot allocate memory" on an otherwise perfect cifs share on the Linux end can be fixed by setting a few registry keys on the Windows box as follows:

• HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management\LargeSystemCache (set to 1)
• HKLM\SYSTEM\CurrentControlSet\Services\LanmanServer\Parameters\Size (set to 3)

Alternatively, start Command Prompt in Admin Mode and execute the following:

reg add "HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management" /v "LargeSystemCache" /t REG_DWORD /d 1 /f
reg add "HKLM\SYSTEM\CurrentControlSet\Services\LanmanServer\Parameters" /v "Size" /t REG_DWORD /d 3 /f

Do one of the following for the settings to take effect:

• Restart Windows
• Restart the Server service via services.msc
• From the Command Prompt run: 'net stop lanmanserver' and 'net start lanmanserver' - The server may automatically restart after stopping it.

Note: Googling will reveal another tweak recommending users to add a key modifying the "IRPStackSize" size. This is incorrect for fixing this issue under Windows 7. Do not attempt it.

Remmina - Unable to connect RDP Server

Have you ever encountered sudden failure to RDP to your Windows desktop using Remmina? There are two known reasons to me:

1. your RDP plugin for Reminna is missing. In Fedora, type sudo yum install remmina-plugins-rdp, or Ubuntu sudo apt-get install remmina-plugin-rdp. You may require to reboot after install in order to load the RDP plugin. Sometimes Import simply won't work.
2. your remote desktop's fingerprint has changed. It is impossible to reset the fingerprint within Remmina's interface. What you can do is fire up terminal, go to ~/.freerdp/ and delete known_hosts.

Now try again and voila! Your remote desktop is back!

Terminal does not have history and not showing path?

What happen when a new user's terminal (xterm, lxterminal etc):

1. does not display any path (only show $>)
2. does not support Up key for recall previous command?

It is because the new user is using /bin/sh instead of /bin/bash. To solve this, simply do sudo chsh, then key in /bin/bash when prompted to choose an option.

Restart the terminal then you should be able to get all the features back.

The root cause for this is nothing is specified when the user is being created using command line. Be sure to specify to use /bin/bash to avoid this.

[Ubuntu] Can access samba network share by IP but not Netbios name from Windows

After whole long day of troubleshooting why we cannot resolve samba shared folders using the NetBIOS Name defined in smb.conf.. finally I have figured out the root of the problem:

Hostname CANNOT equal to NetBIOS name!!!

A bit of background...

1. our server hostname is server1
2. our NetBIOS name is server1 (defined in smb.conf)
3. IP-address is 10.0.0.2

From Windows 7 / OSX we can access the samba shares using \\10.0.0.2\share, as well as \\server1.local\share (the .local is a result of running avahi-daemon). We can also ping 10.0.0.2 or server1 or server1.local, but we simply cannot access with \\server1\share

By changing the Hostname in /etc/hosts & /etc/hostname to any name other than "server1", or change the NetBIOS name in smb.conf to any other name, we can now get it work.

This is rather weird for a Windows user like me, where only hostname need to be set and we can access directly from other PC in the network without bothering what is NetBIOS. Linux is indeed too tricky!

Anyway, kind of fun when able to figure out what's wrong. It makes me love and hate Linux at the same moment!

[Ubuntu] Can ping internet IP address but cannot resolve web address

If you got error when trying to ping any web address e.g. www.google.com, but can ping its IP address, there could be something wrong with the resolv.conf.

Despite you could have set the dns-nameservers in /etc/network/interfaces, the resolv.conf may not be updated yet. And if you have some running process and could not do ifdown / ifup, you can do nano /etc/resolv.conf, comment the original nameserver 127.0.0.1 to nameserver 8.8.8.8 (or the DNS provided by your ISP).

Ubuntu should now resolve the web address correctly!

Damn you, buggy Samba 3.6.3

I have been finding solution to address a samba sharing (Ubuntu 12.04.2 LTS) for few days, the errors that I faced:

1. windows 7 cannot access share
2. windows 7 keep asking password to access samba share
3. windows 7 computer on other VLAN cannot browse the Ubuntu server with Windows Explorer
4. troubleshoot using smbclient -L localhost -U% give errors such as:
   - tree connect failed, NT_STATUS_CONNECTION_INVALID

And very very luckily... I found a discussion & an article which discussed about the bug found in Samba 3.6.3, and suggest to upgrade to 3.6.7:

http://unix.stackexchange.com/questions/55534/samba-server-3-6-3-crashes-on-simple-setup-on-linux-mint-13

https://www.monlore.nl/blog/?p=226 (Marcel's blog)

Thanks god their suggestion work.. and damn Ubuntu & Samba, how could they release such a buggy version for LTS and do not issue immediate upgrade in the official repository?!!

Alright, here's the solution copy from Marcel's blog:

• sudo add-apt-repository ppa:nathan-renniewaldock/ppa
• apt-get update
• apt-get install samba

Voila! Now Samba works like a charm!

[Ubuntu] The bad day!! How to recover from dmraid

Today is a real bad day, one of the Ubuntu server that I "managed" (it's not me who installed it) has gone down! There could be two reasons:

1. Incompatible version of Zentyal (Zentyal 3.0 was accidentally installed on this 12.10, causing the samba not functioning)
2. The electrical leak in the office due to DP box does not have proper earth line installed. Damn it, the whole rack is covered by electricity, and I got an electric shock! Luckily it was not at high current else all devices and I would have died =_=!

Well.. after hours of troubleshooting, I could not bring the server up and running again. It halted at different stage of loading, sometimes failed to load Samba, sometimes failed at Checking Quota...

I do suspect it is caused by the Zentyal... but time is running out. I rush back to home, taking my laptop SSD* installed with Ubuntu 12.10, together with as many as HDD I could found available for the backup**, and plugged into the server.

Voila! Now I'm inside my laptop Ubuntu, and by running dmraid -r, I could now see the RAID under name isw_xxxxx (isw is a prefix for Intel RST). By running dmraid -ay (activate with silent mode YES), I can now see the whole RAID in Nautilus.

I'm backing up those file now... hopefully the files and directories in the server are still in good condition. Losing a single files will be disaster. Cross finger!

=======================

* Why SSD? Because the Ubuntu server was installed using dmraid and Intel RST (Rapid Storage Technology), so SSD can work well as most of the time SSD is used under AHCI. SATA drive that previously installed without AHCI would fail to boot up.

** Please make sure you use a reliable HDD for the backup. You won't like to fail the second time where you could be only having once chance to perform the recovery.

*** Regarding the mdadm and dmraid, here are some good reads:

• https://help.ubuntu.com/community/FakeRaidHowto
• http://raid6.com.au/posts/mdadm_advantages/
• http://www.sourceware.org/lvm2/wiki/DMRAID

[Ubuntu] Wireless stop working after upgrade

If wireless fail to work after upgrade, be sure to use lspci | grep network to look for your wireless device and include this info when you post online for help.

If you're using Broadcom BCM4312 (full name: "Broadcom Corporation BCM4312 802.11b/g LP-PHY (rev 01)") same as mine, then you probably could easily fix by installing the following:

sudo apt-get install firmware-b43-installer b43-fwcutter

or try another one if the above doesn't work:

sudo apt-get install broadcom-sta-common

You're adviced to reboot your system, but before that you may try:

1. off & on your wireless device, it may work straight away without system reboot.
2. run command modprobe b43

If you choose to reboot and the wireless device doesn't start, try modprobe b43. To autoload the device at startup, edit /etc/modules, add "b43" to the list. That should load it during startup.

Good luck!

[Ubuntu] iNotify-Tools: The Realtime File Monitoring Tool

I have been searching for a good real time file monitoring tools to save time on checking what directories or files has been updated by my colleagues. It seems the iNotify-Tools that built into Linux kernel has gained a lot of recommendation, and even IBM has an article on their website address for this little tool.

To setup inotify is fairly simple:

1. apt-get install inotify-tools
2. after install, it comes with two module:
• inotifywait - it runs and keep monitor for changes happen in the target folder/file, and the results can be output to an external file.
• inotifywatch - it outputs a summary count of the events received on each file or directory.
3. We will use inotifywait to monitor our folder and output a text file in the same folder.

inotifywait /targetFolder -mr -e create,delete,move --timefmt "%D_%a_%H:%M" --format "%T | %w%f | %e" -o /targetFolder/FolderReport.txt --exclude "FolderReport.txt"

• "-m" is used to continuously monitor the target. Without this, by default it will exit after the first event occurs.
• "-e" is to specify the event that we want to monitor. Full list can be viewed by execute inotifywait -h
• "--timefmt" is used to add date and time to the log. By default the result doesn't has such info.
• "--format" has to be specified if --timefmt is used. %T is the time, %w is the path of the folder being monitored, %f is the file name and %e is the event (action done to the folder/file). We can add any words in between the parameter, and it will be output as well. For example, %w the folder is %f %e will output = "/home/tecsun/folder the folder is TestFolder CREATE,ISDIR. (blue is the output specified by parameter) 
• "-o" is to output the result to file (else it would display on the terminal)
• "--exclude" is to exclude the result file being monitored, especially when modify event is triggered, so it won't get into infinity loops.

There is one more tools namely iwatch, which is based on inotify, would allow sending of email when event being triggered. Another one which I have not tested is gamin, if anyone of you happen to test this, please let me know how it works.

External links:

• http://www.ibm.com/developerworks/linux/library/l-ubuntu-inotify/index.html
• A great article with more advanced setup and few other options: http://andries.filmer.nl/kb/Monitoring-file-system-events-with-inotify,-incron-and-authctl/129
• Advanced setup and installation under multiple distros: https://github.com/rvoicilas/inotify-tools/wiki
• http://linuxaria.com/article/introduction-inotify?lang=en

[Ubuntu] Xrdp password failed after user password changed

If the password of users has been changed, Xrdp service must be restarted to reflect the changes made. Else, it would give an error of either "password failed" or "login error".

To restart xrdp service, simply type this command in terminal:

/etc/init.d/xrdp restart

After that you should be able to login from remote connection.

[Ubuntu] Temporarily / Permanently Mount Cifs Network Shares

It is strange that Ubuntu 12.10 doesn't come with cifs-utils installed. Whenever you try to mount a network folder via mount -t cifs -o username=user,password=pass //server-name/shared /mnt/shared, it will give you the following error message:

1:  mount: wrong fs type, bad option, bad superblock on //192.168.0.11/Dropbox,  
2:      missing codepage or helper program, or other error  
3:      (for several filesystems (e.g. nfs, cifs) you might  
4:      need a /sbin/mount.<type> helper program)  
5:      In some cases useful info is found in syslog - try  
6:      dmesg | tail or so  

Execute dmesg | tail will return the following result:

1:  [ 5537.959706] CIFS VFS: Connecting to DFS root not implemented yet  
2:  [ 5537.959815] CIFS VFS: cifs_mount failed w/return code = -22  

To solve this, simply apt-get install cifs-utils. After that run the same command again and you should be able to mount the network shared folder.

To permanently mount the shared folders, go to /etc/fstab, and add the following line:

1:  //server-name/shared     /mnt/shared     cifs     defaults,_netdev,credentials=/home/user/.smbcredentials     0     0  

_netdev parameter is being explained at the last section of this documentation, which in general mean connect only when network is available:
https://help.ubuntu.com/community/MountWindowsSharesPermanently

credentials and /home/user/.smbcredentials is a file created to store the username and password for the remote login, so that we don't put these sensitive info at risk because the /etc/fstab can be viewed by every users on the system. DO NOT forget to set the file permission for the .smbcredentials:

1:  chmod 600 /home/user/.smbcredentials  

Now enjoy working with your network shares!

If you would like to read on, the following articles is good to go:

1. https://help.ubuntu.com/community/Samba/SambaClientGuide
2. https://wiki.ubuntu.com/MountWindowsSharesPermanently
3. https://help.ubuntu.com/community/MountWindowsSharesPermanently

[Ubuntu] Oracle Java 7 Runtime Installation Error

First, I didn't not use manual method for the installation. Instead, I used the PPA method which discussed under here:
http://www.ubuntugeek.com/how-to-install-oracle-java-7-in-ubuntu-12-04.html

However, the installation halted in half way due to my HDD running out of space.

After cleaning up the space and re-run with:

apt-get install -f

It keeps giving me this error:

 Setting up oracle-java7-installer (7u15-0~webupd8~1) ...  
 Installing from local file /var/cache/oracle-jdk7-installer/jdk-7u15-linux-x64.tar.gz  
 Removing outdated cached downloads...  t
 mv: cannot move `jdk1.7.0_15' to `java-7-oracle/jdk1.7.0_15': Directory not empty  
 dpkg: error processing oracle-java7-installer (--configure):  
  subprocess installed post-installation script returned error exit status 1  
 Errors were encountered while processing:  
  oracle-java7-installer  

And this error has disabled Synaptic Package Manager from function properly.

Tried a few solutions but to no avail:

• apt-get purge

  - cannot be run

• apt-get autoremove

  - cannot be run

• dpkg --remove --force-remove-reinstreq oracle-java7-installer

• http://www.khattam.info/solved-subprocess-pre-removal-script-returned-error-exit-status-2-error-2009-08-04.html - This is a great article suggesting a few solutions, big thanks to the author. It may help you to solve Java7 installation problem, give it a try. Especially the step 6, it has successfully remove the oracle-java7-installer and get apt-get work again, but that does not install Java 7 which is our initial intention. Read on...
• Remove the installer file located in /var/cache/oracle-jdk7-installer, then re-download & run the installer... cannot resolve the problem as well

And what did the magic is... delete the directory 'jdk1.7.0 15' located inside the 'java-7-oracle', then re-run the installer apt-get install -f. Voila! The installation should complete without any more hassle.

What happened?

The 'jdk1.7.0 15' & 'java-7-oracle/jdk1.7.0 15' are actually located in /var/cache, but for some reason the 'jdk1.7.0 15' is already exist in 'java-7-oracle' folder. When the installer try to copy it into the parent folder, it cannot overwrite it, suggested by the error message 'Directory not empty'.

Somehow, this cannot be solved by changing the directory write permission, so delete the directory has become the immediate working method.

[Ubuntu] How to configure Smartmontools to send email report

There are a few steps-by-steps tutorial to teach you how to configure Smartmontools to email report to you when drives failed. But to get it works is may not be as simple as complete the steps written.

Especially for Ubuntu users, most of you that encountered this problem may have followed the instruction here:
https://help.ubuntu.com/community/Smartmontools

The documentation is very good indeed and it explains almost everything. However, it forgot to mention that if you are not sending email to root account, you are actually sending email externally and the mail simply may not work on that.

How to tackle this has been documented here:
https://wiki.archlinux.org/index.php/S.M.A.R.T.
*Despite it is for Arch Linux, but it works fine to configure Ubuntu.

Basically, you would have to configure ssmtp or any other MTA (Mail Transport Agent) so that Ubuntu can use it to send email to you.

The setup is rather simple, you can get guidance from:
https://wiki.archlinux.org/index.php/SSMTP

I hope this post will help any of you solve this issue quickly, need not to be like me wasting few hours to searching for solution just to get this simple function work.

[Ubuntu] It's Alacarte time! Menu Editing in Ubuntu

Editing application menu in Lubuntu is not as straightforward as in Windows. Right-clicking only allow adding shortcut to desktop.

Editing .xml to change how the application menu look is not fun to work with and very time consuming.

Very luckily, the is a small utility namely "alacarte" that let you modify it in GUI. To install, simply:

sudo apt-get install alacarte

This is how alacarte look like. Simple & intuitive

[Ubuntu] An Easy Way to Remove Unwanted Locale

There are many unwanted Locale settings file in Ubuntu which can be deleted to save some space.

sudo apt-get install localepurge

Run the command above, a multiple choice DOS-liked user interface will be prompted. Selected those that wish to be kept and all deselected will purge.