About The XenDesktop Guy

XenDesktop and other Citrix experiences from the trenches...

PVS and the dreaded 0x0000007B error

A client was trying to setup a lab of HP T610 thin clients to stream Windows 7 from Citrix Provisioning Services 6.1.  The T610 they used to create the image worked flawlessly, they captured the image, and could then stream it back fine in private or standard mode.  The issue appeared when they went to stream it to more than one T610 – while the original machine worked fine, on every other T610 they tried it would blue-screen on startup with the dreaded 0x0000007B error.

“It’s probably something really simple” I said – famous last words – and set about checking for differences between the T610s in the BIOS settings, firmware versions and so on, deleting and recreating machines in the PVS console, recreating the AD accounts and more.  I was already familiar with the “ghosted network card” issue (CTX133188) but this was not the issue.  A couple of hours later and no “simple” fixes in sight – I eventually stumbled across this thread and on page 4 one of the posters references MS hotfix KB2550978.  I booted the VHD in Hyper-V, removed the PVS Target Device software, installed this hotfix and reinstalled the PVS Target device – problem solved!

The issue is the T610 clients support a feature called “PCI Express Serial Number Capability” so from PVS’s point of view, the network card in every individual T610 is seen as a new NIC and needs the driver installed before it can boot successfully.  But as it’s trying to boot across the network, this causes the 0x0000007B error.  Not many clients I have come across stream to physical desktops, so this probably won’t impact too many people, but I hope this info helps someone out!

XenServer 5.6 SP2 hangs after Hotfix 19 applied

I have recently updated several XenServer installs from base 5.6 SP2 to the latest hotfix levels and encountered an issue that doesn’t seem to be specifically documented on the Citrix support site.  All of these sites had a similar hardware configuration, IBM servers with fibre channel adapters and switching connecting to IBM DS3400 storage arrays.  With the first of these servers, when applying the hotfix updates I installed them all at once, without rebooting in between.  The server then hung on restart, at the screen below:

Fortunately the process of reinstalling a XenServer host is quite straight-forward, so I performed a fresh install, configured the host and added back into the pool, and installed the hotfixes one by one rebooting in between.  The server then hung again after installing Hotfix 19.  After another fresh install and performing tasks in a slightly different order I isolated the problem, on the original server build and in my first reinstall I had enabled RDAC multipathing in order for XenServer to use the IBM DS3400 storage array before installing any hotfixes.  On my last reinstall, I didn’t enable RDAC multipathing until after the hotfixes were all installed.  If RDAC multipathing is enabled prior to HotFix 19 being installed then it will hang on restart.

So if this catches you out, a full reinstall is not necessary.  Restart your XenServer host and during the startup phase enter Safe Mode.  To do this, when XenServer is at the boot loader stage type “menu.c32” (be quick!)

You will then be presented with a blue screen with several options, scroll down to “safe” and hit Enter.

You may see a lot of “Buffer I/O Error” and “end_request: I/O Error” messages, this is normal on a system that has multiple paths to the storage array but doesn’t have a multipath driver enabled.  If you want, you can unplug your server from the fibre channel switch during startup to prevent these errors showing.

Once Xenserver has booted in Safe Mode, either in XenCenter or on the console of the host itself, from command prompt and re-enable RDAC multipathing as per the admin guide.  Ie type the following command and reboot (don’t forget to reconnect your storage array if you unplugged it).

/opt/xensource/libexec/mpp-rdac --enable

You should now be back in business!  Based on this experience, I would imagine it’s possible this issue would manifest itself with other XenServer versions and hotfixes / service pack installs – every version up to XenServer 6.1 still uses RDAC drivers for LSI storage arrays such as the IBM DS series.  Based on some reading since I encountered this issue, it appears installing certain hotfixes generate a new initrd which results in some sort of incompatible configuration with the RDAC drivers.  Running the “mpp-rdac –enable” command regenerates the initrd that is then correctly configured for RDAC.

XenApp / XenDesktop, Android 4.0 and Certificates

I’ve encountered various certificate related issues with Citrix Receiver on mobile devices over the years, and most of the time the issues related to incorrectly installed or configured certificates.  Some of the common issues are the intermediate or root certificates aren’t installed, or the certificates aren’t linked correctly.

However Android 4.0 however seems to be fussier about certificates, and I’ve had working setups with Android 2.3 stop working under Android 4.0 (while Windows/Mac/iOS etc work all along).  To help assist those thinking about a new certificate, here are a couple of certificates that I have found to work 100% reliably with Android as well as some I know that seem to have issues:

Working Certificates:
GeoTrust TrueBusinessID http://www.geotrust.com/ssl/ssl-certificates/
GeoTrust TrueBusinessID Wildcard http://www.geotrust.com/ssl/wildcard-ssl-certificates/
GoDaddy Standard or Premium SSL http://www.godaddy.com/ssl

Certificates I have had trouble with:
RapidSSL http://www.rapidssl.com/
Digicert http://www.digicert.com/

There will be more certificates out there that work reliably, but I have not tried them – once I discovered the GeoTrust ones work reliably I’ve stuck with them.  If you have reliable experiences with other certs, feel free to post a comment so we can all benefit from each other’s experiences.

XenDesktop VM PXE boot issue on XenServer

I had a previously stable XenDesktop farm start having issues today, with the client saying they could not connect to any of the Windows 7 desktops in their farm.  A quick investigation revealed most of the desktop VMs had been placed in maintenance mode by the Desktop Delivery Controller.  It will do this automatically if the VMs do not register after several reboots, see CTX126704 for the registry entries that control this behaviour.

After much investigation looking at the PXE and TFTP services on the Provisioning Servers, checking DHCP scopes weren’t full, confirming switch ACLs and VLANs were correct and more, I found the setting on the DHCP server for Conflict Detection had been recently enabled.  It was set to 2 attempts, but it seems this setting caused enough of a delay that the VMs would time out trying to PXE boot.  Once the setting was reverted to zero everything was back to normal.

Note that this client’s XenDesktop setup was XenServer 6.0.2, and the XenDesktop VMs were in the same VLAN as the Provisioning Servers so the client VMs could just broadcast to receive the PXE settings.  This is one of the preferred options for enabling TFTP high availablility as per Citrix Blog post by Nick Rintalan last year here (option 4 in his post).

I haven’t yet checked to see if this issue manifests itself in all environments, but now that I’ve seen it I recall having similar issues at another site where we couldn’t get PXE boot behaving reliably and we ended up handing out the TFTP server via DHCP options 66 & 67.  Hopefully this helps if you are troubleshooting a similar issue, and I’d be keen to hear from anyone that has the same issue and whether my fix above worked for you.

Citrix Profile Manager, Sophos and corrupt profiles

I’ve had various random issues with some sites that use Citrix Profile Manager and Sophos AntiVirus, usually manifesting as corrupt user profiles.  I logged an issue with Sophos but they weren’t too interested in investigating or resolving the issue so I’ve found the following two workarounds.  Some customers only needed one of these workarounds, some needed both, so your mileage may vary.

Workaround 1:

In the Sophos Enterprise Console, under “Anti-virus and HIPS” edit the policy that applies to your XenDesktop and/or XenApp servers.  Under the “Web Protection” box, set “Download Scanning” to off.

Workaround 2:

Disable the “Profile Streaming” and “Active Write Back” features of the Citrix Profile Manager, either via the INI file or Group Policy.  Edit your policy containing the Profile Manager administrative template and settings, and navigate to Computer Configuration -> Policies -> Administrative Templates -> Citrix (if using the Profile Manager 4.1 ADMX template, otherwise Classic Administrative Templates -> Citrix) -> Profile Management.  Change the “Active Write Back” setting to Disabled, then under “Streamed User Profiles” change the “Profile Streaming” setting to Disabled also.

I hope this information helps someone out, as I was unable to find any other references to this issue in Citrix or Sophos KBs, Citrix forums etc.  It’s also possible that other AV products may cause the same behaviour, but I’m yet to see it.  I’m not a fan of Sophos in general, and the performance of the product is that bad that one of my customers recommends turning off the on-acces scanning on file servers and running a nightly scan instead.  Enough said I think!

Step by Step – Installing NetScaler Green Bubble Theme with no internet access

For those new to NetScaler and unix (like me), this might save you some time.  In attempting to follow this Citrix blog post on uploading the Green Bubble theme (http://blogs.citrix.com/2012/04/19/green-bubble-theme-for-citrix-netscaler/), I found the instructions are high level and assume you already know your way around a NetScaler :)

I spent nearly an hour messing around trying to determine the exact commands to install this theme without Internet access, so to save others the time and frustration here are the step by step commands.

1. Download the script file and theme to your PC, eg http://cdn.ws.citrix.com/wp-content/uploads/2012/04/GreenBubble.txt and http://cdn.ws.citrix.com/wp-content/uploads/2012/04/GreenBubble1.gz
2. SSH to your NetScaler (ie use Putty) and enter the shell command
3. Run the following command:

mkdir /var/vpn/customizations

4. Use WinSCP to connect to your NetScaler. Copy the downloaded script file (GreenBubble.txt) to the /root folder and the GreenBubble1.gz file to /var/vpn/customizations
5. Run the following commands:

cd /var/vpn/customizations
gunzip GreenBubble1.gz
tar -xvf GreenBubble1
cd /root
mv GreenBubble.txt GreenBubble1.sh
chmod +x GreenBubble1.sh
./GreenBubble1.sh

6. Hopefully now you have the theme installed, watch for errors in the script. A successfull output should look like this:

+ basename ./GreenBubble1.sh .sh
+ SKINNAME=GreenBubble1
+ SKINARC=GreenBubble1.gz
+ SKINDIR=/var/vpn/customizations
+ DL=/tmp
+ EPA=ns_gui/epa/epa.html
+ SKINURL=http://citrixdownloads.techstur.com/GreenBubble1.gz
+ [ -d /var/vpn/customizations/GreenBubble1 ]
+ fgrep var nsversion= /var/vpn/customizations/GreenBubble1/ns_gui/epa/epa.html
+ cut -d -f2
+ OLDCOMMAVER=10.0.54.7
+ echo 10.0.54.7
+ tr , .
+ OLDDOTVER=10.0.54.7
+ nsapimgr -d hwinfo
+ grep Version:
+ sed -e s/Version: NetScaler NS// -e s/: Build /\./ -e s/, Date.*//
+ cut -d. -f1,2,3,4
+ DOTVER=10.0.54.7
+ echo 10.0.54.7
+ tr . ,
+ COMMAVER=10,0,54,7
+ [ 10.0.54.7 != 10.0.54.7 ]
+ cp -rf /var/vpn/customizations/GreenBubble1/ /netscaler/
+ cp ./GreenBubble1.sh /var/vpn/customizations
+ chmod 755 /var/vpn/customizations/GreenBubble1.sh
+ touch /nsconfig/nsafter.sh
+ chmod 755 /nsconfig/nsafter.sh
+ fgrep -q /var/vpn/customizations/GreenBubble1.sh /nsconfig/nsafter.sh

7. Reboot and enjoy green bubbly goodness :)

Note that if you have two NetScalers in a HA pair, you will need to perform these steps on both units.