XenServer 5.6 SP2 hangs after Hotfix 19 applied

I have recently updated several XenServer installs from base 5.6 SP2 to the latest hotfix levels and encountered an issue that doesn’t seem to be specifically documented on the Citrix support site.  All of these sites had a similar hardware configuration, IBM servers with fibre channel adapters and switching connecting to IBM DS3400 storage arrays.  With the first of these servers, when applying the hotfix updates I installed them all at once, without rebooting in between.  The server then hung on restart, at the screen below:

Fortunately the process of reinstalling a XenServer host is quite straight-forward, so I performed a fresh install, configured the host and added back into the pool, and installed the hotfixes one by one rebooting in between.  The server then hung again after installing Hotfix 19.  After another fresh install and performing tasks in a slightly different order I isolated the problem, on the original server build and in my first reinstall I had enabled RDAC multipathing in order for XenServer to use the IBM DS3400 storage array before installing any hotfixes.  On my last reinstall, I didn’t enable RDAC multipathing until after the hotfixes were all installed.  If RDAC multipathing is enabled prior to HotFix 19 being installed then it will hang on restart.

So if this catches you out, a full reinstall is not necessary.  Restart your XenServer host and during the startup phase enter Safe Mode.  To do this, when XenServer is at the boot loader stage type “menu.c32” (be quick!)

You will then be presented with a blue screen with several options, scroll down to “safe” and hit Enter.

You may see a lot of “Buffer I/O Error” and “end_request: I/O Error” messages, this is normal on a system that has multiple paths to the storage array but doesn’t have a multipath driver enabled.  If you want, you can unplug your server from the fibre channel switch during startup to prevent these errors showing.

Once Xenserver has booted in Safe Mode, either in XenCenter or on the console of the host itself, from command prompt and re-enable RDAC multipathing as per the admin guide.  Ie type the following command and reboot (don’t forget to reconnect your storage array if you unplugged it).

/opt/xensource/libexec/mpp-rdac --enable

You should now be back in business!  Based on this experience, I would imagine it’s possible this issue would manifest itself with other XenServer versions and hotfixes / service pack installs – every version up to XenServer 6.1 still uses RDAC drivers for LSI storage arrays such as the IBM DS series.  Based on some reading since I encountered this issue, it appears installing certain hotfixes generate a new initrd which results in some sort of incompatible configuration with the RDAC drivers.  Running the “mpp-rdac –enable” command regenerates the initrd that is then correctly configured for RDAC.

XenApp / XenDesktop, Android 4.0 and Certificates

I’ve encountered various certificate related issues with Citrix Receiver on mobile devices over the years, and most of the time the issues related to incorrectly installed or configured certificates.  Some of the common issues are the intermediate or root certificates aren’t installed, or the certificates aren’t linked correctly.

However Android 4.0 however seems to be fussier about certificates, and I’ve had working setups with Android 2.3 stop working under Android 4.0 (while Windows/Mac/iOS etc work all along).  To help assist those thinking about a new certificate, here are a couple of certificates that I have found to work 100% reliably with Android as well as some I know that seem to have issues:

Working Certificates:
GeoTrust TrueBusinessID http://www.geotrust.com/ssl/ssl-certificates/
GeoTrust TrueBusinessID Wildcard http://www.geotrust.com/ssl/wildcard-ssl-certificates/
GoDaddy Standard or Premium SSL http://www.godaddy.com/ssl

Certificates I have had trouble with:
RapidSSL http://www.rapidssl.com/
Digicert http://www.digicert.com/

There will be more certificates out there that work reliably, but I have not tried them – once I discovered the GeoTrust ones work reliably I’ve stuck with them.  If you have reliable experiences with other certs, feel free to post a comment so we can all benefit from each other’s experiences.