Device Not Found when Removing HDD from mdadm RAID

I have encountered a case when I'm trying to fail and remove a disk from a mdadm RAID array using the following two commands:

• mdadm --manage /dev/md1 --fail /dev/sdf --remove /dev/sdf
• mdadm /dev/md1 --fail /dev/sdf --remove /dev/sdf

I encountered error "device not found".

Looking at the Disks (gnome-disks), the device /dev/sdf is still shows as a member of RAID /dev/md1.

What is the solution?

After examine using both:

• mdadm --detail /dev/md1
• cat /proc/mdstat

Confirming that /dev/sdf is not currently active in the RAID array, I execute mdadm --zero-superblock /dev/sdf. This reset the HDD and now it appears as an empty disk in Disks (gnome-disks).

I can then re-add the HDD to the array again as a spare unit, mdadm --add /dev/md1 /dev/sdf.

Setting Up Red Hat Virtualization with Single Server

Red Hat Virtualization is an interesting and powerful platform software, but it is rather challenging to setup with only a single host when compared to Hyper-V, VMware and VirtualBox. One may argue that RHEV is not design for single host (I guess that is why there is no single article about how to do the setup), but who really care if just want to use for a tiny scale deployment? Single host rules!

Well, I could go for RHEL + KVM, but I simply like the interface of RHEV.

Getting it up and running requires user to have experience from numerous area of expertise. This simplified guide is not for newbie.

Setting up the RHVH is not hard. I'm going to skip it. Now, in order to get the hosted-engine up and running, please make sure the following are done:

1. Change the hostname localhost.localdomain to hostname. In my case, I use robustpoc.
2. Be sure to set a fix IP in /etc/sysconfig/network-scripts/ifcfg-eth0 and reconnect using the new IP using ifdown eth0 and ifup eth0.
3. Edit /etc/hosts to resolve the host IP address to the hostname.
4. In order to make this IP change persistent, be sure to change the settings in /var/lib/vdsm/persistence/netconf/nets/ovirtmgmt

Note: If the server is in a VLAN Network, be sure to set a correct gateway otherwise it would not be able to connect to Internet or reach by others in the network.

5. Now, before proceed with executing hosted-engine --deploy, you require to add your FQDN to /etc/hosts. In my case, I use 192.168.0.26 robustpoc.com. This is critical.
6. Next, setup NFS share. Follow the guide below:

Setup NFS server: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/storage_administration_guide/nfs-serverconfig

Configuring NFS server: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/storage_administration_guide/nfs-serverconfig

7. At this point, we are ready to execute hosted-engine --deploy.

Three critical settings to take note during the process:
8. Be sure NOT to set the memory of the VM to maximum available memory of the system as per suggested by the setup. Setting to maximum will lead to no available memory to create any single Virtual Machine.
9. Be sure to use a fixed IP for the engine. Make sure gateway is correct too.
10. Be sure to answer YES when asked whether to update the /etc/hosts for both the host and engine VM. The default answer is NO. Missing out this would be a killer later.

Until this point, everything critical should have already been taken care of. Cross finger and I wish you the best luck!

Common errors that I have encountered but do not guarantee to happen:
1. Error: [Get Local VM IP] failed
2. Info: [Waiting for local VM to be up]
3. [ ERROR ] Failed to execute stage 'Closing up': Failed executing ansible-playbook; then I can access to RHEV Manager but an failed external hosted engine present in the Compute > Hosts.

Welcome to share with me your installation experience, I'm happy to listen and learn from you.

Mdadm Must Be Configured on Partition

Refer to my previous post Missing mdadm array after reboot, I would like to add on the reason why auto-scan does not work once system rebooted - it requires the RAID to be configured on a partition instead of the disks!

This is going to have problem:

mdadm --create --level=6 --raid-devices=5 /dev/md0 /dev/sd[b-f]

 and this would be correct, please take note of the digit at the end of line:

mdadm --create --level=6 --raid-devices=5 /dev/md0 /dev/sd[b-f]1

It means you must format all the disks with at least one partition, and building the RAID using the partition.

I  hope this would be helpful for novice users who trying to get the RAID run properly.

Problem Updating Repo List in Ubuntu 12.10

The sources list has been changed to old-releases. To replace the sources list easily, use the following command:

sudo sed -i -e 's/archive.ubuntu.com\|security.ubuntu.com/old-releases.ubuntu.com/g' /etc/apt/sources.list

If you have choose another mirror then your sources list might contain us.archive.ubuntu.com, then the above changes would still be invalid. You would need to remove the us. from it. Using the command below will help remove it:

sudo sed -i -e 's/us.old-releases.ubuntu.com/old-releases.ubuntu.com/g' /etc/apt/sources.list

Houdini Start-Up Crashed on Render Node

AFAIK, Houdini could not start its main application (GUI) without OpenGL enabled graphics card. Hence if you are trying to start it in a render node with ASPEED on-board graphics, it is impossible. All you can do is to run the render using CLI.

As for user encounters the same issue, if you are using Nvidia graphics card, you would need to switch the XORG graphics driver to Nvidia proprietary driver. You can do it at:

Menu > Settings > Software and Update > Additional Drivers

Sometimes the latest driver may not work well. So you may have to tried different version to get it work.

Tips: To get a clearer picture of why Houdini failed to start, instead of running it from software menu, you can run it from terminal. The error message will be more readable than the crashed dump file.

Missing mdadm array after reboot

It is heart-attack when you reboot your linux storage server and realized mdadm array gone missing!

• RAID volume not showing in gnome-disks.*
• gparted reports empty partition on all the RAID disks

sudo mdadm --examine --scan -v shows:

mdadm: looking for devices for /dev/md0
mdadm: Cannot assemble mbr metadata on /dev/sda
mdadm: Cannot assemble mbr metadata on /dev/sdb
mdadm: Cannot assemble mbr metadata on /dev/sdc
mdadm: Cannot assemble mbr metadata on /dev/sdd
mdadm: Cannot assemble mbr metadata on /dev/sde

sudo mdadm --examine /dev/sd*  or mdadm --query /dev/sd* shows:

mdadm: No md superblock detected on /dev/sd*

sudo mdadm --assemble /dev/sd[b-f] shows:

mdadm: device /dev/sdb exists but is not an md array.
mdadm: No arrays found in config file or automatically

If you go into /etc/mdadm/mdadm.conf (Debian / Ubuntu) or /etc/mdadm.conf (Fedora), you would realize there is no ARRAY being defined. In some system the mdadm.conf simply does not exist.

No matter how hard you try, the mdadm array just won't show up. There are numerous suggestion on the web, being the most common is to add an auto examine during boot in the mdadm.conf:

 sudo mdadm --examine --scan --config=mdadm.conf >> /etc/mdadm/mdadm.conf

However the above simply does not work for me. I have also try adding the ARRAY manually by doing the following, in which the UUID is the first hdd used for the array (to show the UUID, use blkid. Strange enough I could not get the UUID of my GPT hdd in Ubuntu, and only can get by using Fedora):

ARRAY /dev/md0 metadata=1.2 UUID="0db0c336:f56bd888:2f9e92e4:c1d64c09" >> /etc/mdadm/mdadm.conf

It does not work as well. I have no other solution except try to re-create the array again. Be very careful of this step, I'm more daring to take this step because I already backup my data one day before I lost the array. When you re-create the array, BE SURE to use --assume-clean and make sure the parameter set is EXACTLY the same as what you used to create it the first time. In my case, it is very simple and straightforward:

mdadm --create --assume-clean --level=6 --raid-devices=5 /dev/md0 /dev/sd[b-f]

The RAID array is being created and immediately I got all my data back!! It is advisable to backup your data now and once the backup is done, do a data scrubbing to ensure it is running well.

I hope you will be as lucky as I do, to have all the data back without losing great memory and important works.


External link that I used as reference:

• A very useful read: https://raid.wiki.kernel.org/index.php/RAID_Recovery
• https://hiringsolved.com/blog/goodbye-big-data-mdadm-no-recognizeable-superblock/
• http://ubuntuforums.org/archive/index.php/t-2090527.html
• http://kevin.deldycke.com/2007/03/how-to-recover-a-raid-array-after-having-zero-ized-superblocks/ (be very careful with this, I would not suggest zero out the superblock unless you know what it does)
• http://ubuntuforums.org/showthread.php?t=1950154
• http://superuser.com/questions/603481/how-do-i-reactivate-my-mdadm-raid5-array
• Changing UUID of a mdadm partition: https://itooktheredpill.irgendwo.org/2010/change-uuid-of-a-mdadm-partition/
• This is not useful in my case, but maybe helpful for others: http://askubuntu.com/questions/247981/software-raid-mdadm-re-find-my-array

* If it is shown, then you are lucky because it simply changes its name from /dev/md0 to something like /dev/md127. You can still use it by changing the mounting command to the new array name.

Never sign up VPS from this company

Recently I have sign up a VPS from a company call WideVPS, and it is really the worst hosting company that I have ever encounter. For those of you who like an affordable hosting, please be careful with the company:

1. The VPS is activated after a few hours of waiting. (they claimed it is instant activation)
2. Support ticket tooks 4 hours to response. (they claimed 1-hour response)
3. Ticket was closed even the issue has not been resolved. I created a Windows VPS for testing before moving to linux, during the sign up I have key-in a password for the VPS. When it is finally up after 4-hours, the password I entered is not working!! I then open a ticket and email, it tooks 2 days and never resolved! All I get is.... I will send you the info in next email. Then? I receive no email at all.
4. Submitted a PayPal dispute, but they never response to the dispute. (so you can see they are not serious in the business at all)

And this morning I receive their invoice asking me to renew the hosting, but until today I'm not able to use the VPS at all!

Guys and Gals, be careful. I'm glad if you have good experience with them. But if you are still considering, just make sure you contact them before sign up and then judge yourself. I hope you will then not wasting money like I did.

Ubuntu Boot to Blank Screen with Blinking Underscore (Cursor)

Today I was installing the Ubuntu 14.04.02 LTS into a server using a USB thumbdrive. Everything went smooth, as usual the setup asked to remove the USB thumbdrive before continue to Reboot.

After reboot, it hangs on a black screen with blinking underscore / cursor. Waited for 15 mins, nothing happen. If you encounter the same error like I do, it is possibly caused by:

1. a corrupted GRUB
2. the setup simply installed the GRUB onto the USB thumbdrive accidentally. (I suspect it is because I have chosen to umount the thumbdrive during the setup)
3. graphics driver not supported (it sounds a bit weird to me, if the setup could run well in GUI, why should a reboot not?)

For the first two reasons, a simple solution is to repair the GRUB. Here's the complete guide for Ubuntu users. For my case, because I knew the GRUB is in the thumbdrive, so I simply plug it in and choose my HDD as first boot device, it then boot into Ubuntu successfully, then I run the following command in terminal:

sudo add-apt-repository ppa:yannubuntu/boot-repair
sudo apt-get update
sudo apt-get install -y boot-repair && boot-repair

and then follow what is being instructed by the boot-repair. The only steps that caused a confusion is which disk or partition to select as GRUB install location. A rule of thumbs is don't select the partition, select the disk.

For 3rd reason - graphics driver broken. I personally do not try it, but someone from online said it can be fix by inserting the -nomodeset in between ro quiet splash. And I do believe you realize one thing now... if you could see the GRUB and able to insert the -nomodeset, well, then it is most likely your problem is more challenging, good luck hunting it down!

XFCE4 Desktop Not Showing

Sometimes when you login the desktop wallpaper, icons are all missing. It is due to the xfdesktop not loaded properly. To fix it, simple type xfdesktop in terminal, there will have some message appeared but can be safely ignore.

[Fedora] Fix Yum Update crashed at the middle of updat

If the problem you having is duplicates packages, simply refer to the following info to attempt the fix:

sudo yum check all                # tells you of any problems
sudo package-cleanup --problems   # lists all known package problems
sudo package-cleanup --dupes      # lists duplicate packages
sudo package-cleanup --cleandupes # actually cleans up duplicates
sudo yum check all                # run again to check for remaining problems
sudo yum-complete-transaction --cleanup-only

Thanks to https://destefano.wordpress.com/2013/08/13/yum-update-fail/#comment-2714

CIFS failed to allocate memory?

If you are mounting your Windows network shares in your Ubuntu, do not hurry to move your files to this newly added mount point. Most probably you will be regret very soon, as the system may give you error Failed to allocate memory and soon after that your files will gone missing. Yes, all gone! That's what happened to me.

The issues seems to cause by Windows rather than Ubuntu itself. There is a good explanation in this link:
https://wiki.archlinux.org/index.php/Samba/Troubleshooting

But I will copy & paste some of the content here, in case the link broken in the future.

======== Copy & Paste from archlinux.org, please contact me if remove is required ========

Windows 7 connectivity problems - mount error(12): cannot allocate memory

A known Windows 7 bug that causes "mount error(12): cannot allocate memory" on an otherwise perfect cifs share on the Linux end can be fixed by setting a few registry keys on the Windows box as follows:

• HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management\LargeSystemCache (set to 1)
• HKLM\SYSTEM\CurrentControlSet\Services\LanmanServer\Parameters\Size (set to 3)

Alternatively, start Command Prompt in Admin Mode and execute the following:

reg add "HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management" /v "LargeSystemCache" /t REG_DWORD /d 1 /f
reg add "HKLM\SYSTEM\CurrentControlSet\Services\LanmanServer\Parameters" /v "Size" /t REG_DWORD /d 3 /f

Do one of the following for the settings to take effect:

• Restart Windows
• Restart the Server service via services.msc
• From the Command Prompt run: 'net stop lanmanserver' and 'net start lanmanserver' - The server may automatically restart after stopping it.

Note: Googling will reveal another tweak recommending users to add a key modifying the "IRPStackSize" size. This is incorrect for fixing this issue under Windows 7. Do not attempt it.

How to extend screen to second monitor in Linux / Fedora?

Here's the command that could extend the screen to second monitor, in case Linux does not do the job automatically:

xrandr --output VGA1 --mode 1920x1080 --output eDP1 --mode 1366x768 --left-of VGA1

Where you need to replace the screen size manually. By running xrandr in terminal, you could get the screen size range for each display output.

Beside --left-of, there are other options such as --above, --below, --right-of

[Alt] key problem when remote desktop Windows via Remmina

If you are using Xfce 4 and encounter the following problems using [Alt] key inside Microsoft Windows remote desktop session:

• [Alt] key does not work as per expected
• clicking [Alt] + mouse middle button move you away from your remote desktop session

then you should try this simple and straight forward trick before going too technical.

• Go Settings > Settings Manager > Window Manager Tweaks > Accessibility > "Key used to grab and move windows" > set it to "None"

Voila! Now it should works like charm. Otherwise... you might have to Google around and read about re-map key, etc.. Good luck!

Remmina - Unable to connect RDP Server

Have you ever encountered sudden failure to RDP to your Windows desktop using Remmina? There are two known reasons to me:

1. your RDP plugin for Reminna is missing. In Fedora, type sudo yum install remmina-plugins-rdp, or Ubuntu sudo apt-get install remmina-plugin-rdp. You may require to reboot after install in order to load the RDP plugin. Sometimes Import simply won't work.
2. your remote desktop's fingerprint has changed. It is impossible to reset the fingerprint within Remmina's interface. What you can do is fire up terminal, go to ~/.freerdp/ and delete known_hosts.

Now try again and voila! Your remote desktop is back!

[Fedora] Omit GUI and boot with or into command line

Before you start, you need to know what is runlevel.

runlevel is a number that indicates which mode the computer to boot into. For instance, runlevel 5 is graphical mode, where runlevel 3 is text-only mode. Runlevel 1 is single-user mode, commonly used for troubleshooting.

And GRUB (GRand Unified Bootloader), a bootloader which allow you to choose which OS to boot into during computer start up. Here's a screenshot of GRUB.

Now, there are two options:

1. boot with command line, means omit the filling up Fedora logo and show the boot progress by command line
2. boot into command line, means completely omit booting into GUI and end up in command line

What you need to do is at the GRUB screen, select your Fedora OS name and press "e".
Then find rhgb in the line start with linuxefi /vmlinuz. On the right side of rhgb should be word "quiet".

For option 1: Remove the rhgb from the line and press CTRL+X or F10. This should show the command lines as boot progress instead of the Fedora logo. Useful for identifying what has failed during boot progress. The system will still end up in GUI afterwards.

Sample = ... rhgb quiet ... 

For option 2: Here's why you have to understand the runlevel. You achieve this by simply add a runlevel number "3" after the word "quiet" (space in between two words) and press CTRL+X or F10. This should boot the system into CLI instead of GUI.

 Sample = ... rhgb quiet 3 ...

Note:

• rhgb = RedHat Graphical Boot

Enable Samba browsing in Thunar

If you enter location path start with smb:// and get a red stop sign, it means you don't have gvfs-smb install. Simply fire up your terminal and type:

sudo yum install gvfs-smb -y

You might want to install Gigolo too to manage your network folder.

sudo yum install gigolo -y

 Voila! Your address bar should work with smb:// now.

Problem connecting Xrdp

If the error appeared as "problem connecting" when connecting to remote desktop using xrdp, it is most likely the remote Linux does not have VNC server installed.

(1) To verify this, you can type:
sudo tail /var/log/xrdp-sesman.log

(2) If you see the error message below, then it means you don't have VNC server installed.
[ERROR] another Xserver is already active on display xx 

(3) To fix this, simply run this command:
sudo apt-get install tightvncserver

(4) After that, restart xrdp service by running:
sudo service xrdp restart

Here's why:
xrdp is the daemon that handles RDP remote desktop access from Windows machines to your Linux Mint PC. But it's not enough to install only xrdp - you need a VNC server. Unfortunately it's not automatically installed with the xrdp package.

The error message "error - problem connecting" after a successful login message isn't very helpful either. Searching the Internet can drive you in circles - I wasted at least 2 hours on finding out that I forgot to install the VNC server - ouch.

Thanks powerhouse from Linux Mint forum who shared this useful information.

Samba Share Permission in a Nutshell

In this post I'm writing all the troubles that I faced when setting up samba shares for Windows and MAC, to help novice users get the samba up and running as quickly as possible. Do note that all the tips & tricks shared here is meant to get things up and running quickly, it might create security hole. Use at your own risk and be sure to research more if you are deploying for high security concern mission.

New created folder or copy & paste into samba shared folder do not follow permission set by smb.conf
The key settings are the following (e.g. permissions 775)

• create mask = 0775
• force create mode = 0775
• security mask = 0775
• force security mode = 0775

• directory mask = 2775
• force directory mode = 2775 (If you change this to 0755, it won't work. So this parameter is very important)
• directory security mask = 2775
• force directory security mode = 2775

Now, why 2775?

• 2 = the setgid bit is set (2) which makes newly created directories have the same group ownership as the parent.
• 77 = owners and group members of the files have full access to them
• 5 = everyone else can enter the directories and view the file names (use with caution)

To be continue...

Terminal does not have history and not showing path?

What happen when a new user's terminal (xterm, lxterminal etc):

1. does not display any path (only show $>)
2. does not support Up key for recall previous command?

It is because the new user is using /bin/sh instead of /bin/bash. To solve this, simply do sudo chsh, then key in /bin/bash when prompted to choose an option.

Restart the terminal then you should be able to get all the features back.

The root cause for this is nothing is specified when the user is being created using command line. Be sure to specify to use /bin/bash to avoid this.

Fast apt-get with apt-fast

Ever experience slow download even with 10Mbps++ internet? The author has written a very useful article to share how to speed everything up with apt-fast:

http://www.webupd8.org/2012/10/speed-up-apt-get-downloads-with-apt.html