I have recently updated several XenServer installs from base 5.6 SP2 to the latest hotfix levels and encountered an issue that doesn’t seem to be specifically documented on the Citrix support site. All of these sites had a similar hardware configuration, IBM servers with fibre channel adapters and switching connecting to IBM DS3400 storage arrays. With the first of these servers, when applying the hotfix updates I installed them all at once, without rebooting in between. The server then hung on restart, at the screen below:
Fortunately the process of reinstalling a XenServer host is quite straight-forward, so I performed a fresh install, configured the host and added back into the pool, and installed the hotfixes one by one rebooting in between. The server then hung again after installing Hotfix 19. After another fresh install and performing tasks in a slightly different order I isolated the problem, on the original server build and in my first reinstall I had enabled RDAC multipathing in order for XenServer to use the IBM DS3400 storage array before installing any hotfixes. On my last reinstall, I didn’t enable RDAC multipathing until after the hotfixes were all installed. If RDAC multipathing is enabled prior to HotFix 19 being installed then it will hang on restart.
So if this catches you out, a full reinstall is not necessary. Restart your XenServer host and during the startup phase enter Safe Mode. To do this, when XenServer is at the boot loader stage type “menu.c32” (be quick!)
You will then be presented with a blue screen with several options, scroll down to “safe” and hit Enter.
You may see a lot of “Buffer I/O Error” and “end_request: I/O Error” messages, this is normal on a system that has multiple paths to the storage array but doesn’t have a multipath driver enabled. If you want, you can unplug your server from the fibre channel switch during startup to prevent these errors showing.
Once Xenserver has booted in Safe Mode, either in XenCenter or on the console of the host itself, from command prompt and re-enable RDAC multipathing as per the admin guide. Ie type the following command and reboot (don’t forget to reconnect your storage array if you unplugged it).
You should now be back in business! Based on this experience, I would imagine it’s possible this issue would manifest itself with other XenServer versions and hotfixes / service pack installs – every version up to XenServer 6.1 still uses RDAC drivers for LSI storage arrays such as the IBM DS series. Based on some reading since I encountered this issue, it appears installing certain hotfixes generate a new initrd which results in some sort of incompatible configuration with the RDAC drivers. Running the “mpp-rdac –enable” command regenerates the initrd that is then correctly configured for RDAC.