About The XenDesktop Guy

XenDesktop and other Citrix experiences from the trenches...

NetScaler – StoreFront monitor quick tip

Citrix NetScaler has a built-in monitor that monitors the state of your StoreFront store – rather than just checking port 443 is up, it checks the state of several services.  For a description of what it does and how it works check here:

http://docs.citrix.com/en-us/netscaler/11/traffic-management/load-balancing/load-balancing-builtin-monitors/monitor-citrix-sf-services.html

When you configure a monitor and bind it to your StoreFront services, you may find it does not work as expected and marks the services as “down”.  As you can see in the screenshot below, the details in the monitor are correct for the store in question yet the services are still marked as down.

NS-Monitors

This is because if you have a Store name with a space in it, eg “Company Apps” and enter this into the monitor it doesn’t work.  What the monitor really needs rather than the store name, is the sub-path on the IIS site – usually this will be “CompanyApps” from the full path of https://storefront.company.com/Citrix/CompanyApps.  Check your StoreFront or IIS console to be sure.

One last tip – in NetScaler 10 the detail of why a monitor failed was quite easy to find, in NetScaler 11 less so – look under Traffic Management -> Load Balancing -> Service Groups -> Edit the service group -> Service Group Members -> Monitor Details.

 

Citrix PVS prep script for Sophos

Some of you will know that Sophos is my, ahem, favourite Sophosanti-virus product.  However it was the product of choice at the time (they are now moving to McAfee) for a large school district I deal with hence I had no choice but to remain acquainted with it irrespective of it’s poor performance (Sophos have advised to leave real-time scanning *off* for performance reasons?!?), it’s poor management console (customer had to write their own deployment scripts for reliable deployment) and poor customer service from Sophos.

I wrote a small script to prepare Sophos for PVS deployment on XenApp and XenDesktop machines, run this script just prior to shutting down your Private mode image and switching back to Standard.

@echo off
REM Script by Daniel Marsh
REM run at shutdown to prep Sophos on Citrix XenApp/VDI machines.
REM refer to https://www.sophos.com/en-us/support/knowledgebase/12561.aspx

net stop "Sophos Agent"
net stop "Sophos Anti-Virus"
net stop "Sophos Anti-Virus status reporter"
net stop "Sophos AutoUpdate Service"
net stop "Sophos Message Router"
net stop "Sophos Web Control Service"
net stop "Sophos Web Intelligence Service"

del C:\ProgramData\Sophos\AutoUpdate\data\machine_ID.txt

reg delete "HKLM\SOFTWARE\Wow6432Node\Sophos\Remote Management System\ManagementAgent\Private" /v pkp /f
reg delete "HKLM\SOFTWARE\Wow6432Node\Sophos\Remote Management System\ManagementAgent\Private" /v pkc /f
reg delete "HKLM\SOFTWARE\Wow6432Node\Sophos\Messaging System\Router\Private" /v pkp /f
reg delete "HKLM\SOFTWARE\Wow6432Node\Sophos\Messaging System\Router\Private" /v pkc /f

When I have a spare few minutes I may even see if I can another feature – it could be run via GPO as a machine shutdown script that only executes if the image is in private mode, which would save you having to run it manually.  I haven’t figured out how to do this yet, if you do know please drop me a comment below.

Refer to https://www.sophos.com/en-us/support/knowledgebase/12561.aspx for more detail on what each key/service etc does.

Machine connection failure – “Refused”

Firstly apologies for the slowdown of blog posts in recent times – between work and family life there never seems to be a free moment, I’ve been meaning to write this one up for a couple of months.

A client with a XenDesktop 7.6 farm of Windows 7 pooled desktops was experiencing, seemingly out of the blue, a large number of machine connection failures.  Most failure reasons listed were the extremely informative “Refused” along with the equally descriptive “Timeout” and “None”.

2015-11-02 10_37_19-Management Desktop - Desktop Viewer

Checking the Studio console showed nothing out of the ordinary, all machines were booted and registered.  Logging onto the console of a machine that had refused a connection also showed nothing unusual, in fact the event log didn’t even have a record of an attempted connection.  Very strange.

My first clue came when trying to open the event viewer remotely to one of the affected desktops.

2015-11-02 10_38_01-Management Desktop - Desktop Viewer

The above error appeared which I wasn’t immediately familiar with, but a quick Google pointed me in the right direction.  Most of the results mentioned kerberos errors relating to mismatched SPNs and account names which got me thinking.  Checking DNS for the hostnames and IPs of the affected virtual desktops showed multiple instances of an IP address being assigned to more than one DNS record.

2015-11-02 10_43_33-Management Desktop - Desktop Viewer

When the connection attempts to a virtual desktop were being made, the broker connection process involves a DNS lookup that in this situation resulted in the wrong IP being returned.

What had happened was the DHCP scope and DNS scavenging settings were misaligned meaning expired DHCP leases did not have their DNS records cleaned up.  For a read-through of recommended settings and how the process works have a look at the following two links:

http://blogs.technet.com/b/askpfe/archive/2011/06/03/how-dns-scavenging-and-the-dhcp-lease-duration-relate.aspx

http://blogs.technet.com/b/networking/archive/2008/03/19/don-t-be-afraid-of-dns-scavenging-just-be-patient.aspx

Long story short, once the stale DNS records were manually purged, then the DHCP lease time was extended from 1 day to 14, and the DNS refresh/no-refresh intervals brought down, everything started to behave again.

PVS Image Updates and Windows Update cleanup

A customer called me in to look at an issue occurring on their Windows 8.1 image that was deployed in their XenDesktop site.  Images were deployed using PVS, using the Write Cache in RAM with Overflow to Disk feature.  Write Cache RAM size was set to 256MB, a fairly standard size for a desktop OS, and the write cache disk was 10GB.  The issue the customer noticed was the write cache was filling very rapidly, and in some cases in as little as 30 minutes after startup.  As PVS admins will know once the write cache fills it’s game over for that VM – it will blue screen or in our case just lock up and had to be forcibly reset using the hypervisor console.

Upon investigation, the write cache was certainly filling rapidly.Write CacheAs you will notice in the above screenshot, write cache usage is over 65% and this was approximately 15 minutes after the VM booted.  Obviously some process was performing a large number of disk writes to cause the cache to fill.  A quick look at the task manager leads us to our first suspect, TiWorker.exe which was consuming a large amount of CPU and also performing a lot of disk IO.  In fact, this was occurring across all VMs on the farm and saturating the host as per the below screenshot.  Rather curiously, it seemed to become worse around midday. (It should be noted there were very few active users on the farm while all this resource usage was occurring.)

XenCenter

 

Looking at an individual VM showed the following:

Win8 Performance

You can see a spike in CPU activity, with a corresponding increase in disk activity.  So, to find out the cause.  A bit of research showed TiWorker.exe was part of the Windows Update engine, but there were no updates being installed at the time, in fact none available as the customer had recently updated this image with the latest Windows Updates.  I came across a Microsoft forum post here that started to shed some light here.  Turns out that periodically, Windows Updates will do some additional compression and cleanup work on the downloaded Windows Updates folder (C:\Windows\WinSxS) and compress the entire lot using the LZX compression routine that was introduced in March 2014 (see here).

What had happened is the client booted the PVS image in private mode, installed the latest batch of Windows Updates then shut down and reverted to Standard mode (with Write Cache in RAM w/Overflow to Disk).  Windows hadn’t completed all these post-Windows Update cleanup routines however, so it kicked them off on the next startup, which happened to be when all VMs were in standard mode.  This filled the write cache on the VMs, they would lock up/freeze/crash, reboot, and start the same process all over again.

The resolution was pretty simple – open the image in private mode again, and let it sit there for Windows Update to do it’s thing.  In this instance, going by the XenCenter performance graph for the VM it took around 4 hours for disk activity to subside.  Once that was complete we reverted to Standard mode, and no more issues!

 

This one slipped under the radar!

This change that was quietly slipped into the recent NetScaler 10.5 56.12 build should please a lot of people.  You may be aware that version 10.5 did away with a lot, but not all, of the requirements for Java when administering the NetScaler GUI.  However a few features still required it – the upgrade wizard, most options under the Diagnostics tab, visualisers, AppFW and a handful of other features.

However as of build 56.12 it seems all requirements for Java have completely disappeared, the upgrade wizard, diagnostic settings and more no longer require Java – as a NetScaler admin this removes a major pain point.  I no longer need to worry about maintaining that old install of Java 1.6 on my machine and having to manipulate the various and ever changing Java security settings just so I can administer my NetScalers! Woo hoo!

I can’t find anything official in the release notes regarding this change, and at time of writing this blog entry the Citrix eDocs still refers to the Java requirements here: http://support.citrix.com/proddocs/topic/ns-rn-main-release-10-5-map/ns-rn-changes-gui-10-5-con.html But regardless, this change made me happy enough I wanted to share it with anyone that wasn’t aware, so enjoy :)

Java Upgrade Wizard
The old Java upgrade wizard present in builds 55.8 and earlier
HTML Upgrade Wizard
The new HTML upgrade wizard present in builds 56.12 onwards

XenServer 6.2 and updating drivers

A while back I posted an article on updating XenServer 6.0.2, with a quick tip on how to check what drivers are loaded on your system.  For reference that article is here – “XenServer 6.0.2 hotfix and driver disk install summary“.  I’ve been caught out though – my quick tip relies on the module name matching the driver disk filename which works in all but one instance.  Eg as you can see in the screenshot below, the bnx2x driver filename also starts with bnx2x.

XS62ESP1009 Driver Disks There is an anomaly though – the filename for the Emulex be2net driver does not match!

XS62ESP1009 Emulex

As you can see the driver name is “be2net” but the filename is “emulex”.  So when running your lsmod command, double-check the filenames match the driver names for all future driver updates, as I’m sure there will be other inconsistencies in the future.

For reference, the command to check your system for any required XenServer 6.2 SP1 Hotfix 9 driver updates (as per CTX141192) would be:

lsmod | egrep 'bnx2x|tg3|bnx2|cxgb4|enic|fnic|be2net|aacraid|hpsa|e1000e|megaraid_sas|qlcnic|qla4xxx|qla2xxx|qlge'

Happy updating :-)

Reset PVD for a Desktop Group via PowerShell and WMI

A client wanted the ability to reset the Personal vDisk for every desktop in a given group, without having to visit each desktop individually via the Director console.  So this is a little quick and dirty, with limited error checking, but I knocked up a PowerShell script to do the job.  Feel free to take it and use it as you see fit, and let me know of any improvements you make :)

The scripts works regardless of whether desktops are in maintenance mode or not, for my purposes it was not necessary to have the script do it but if you are automating this as part of a larger process you may want to add a line that places all desktops into maintenance mode first.  Other potential improvements are adding commands to start all desktops if they aren’t started, and a loop to check when they all become registered.

Copy the text below and save it into a PowerShell script, and execute from a PowerShell prompt on your Desktop Delivery Controller.  You will need to have your execution policy set to unsigned or unrestricted.  Note that desktops will reboot, then shut down as part of the PVD reset process.  You can either boot them manually afterwards, or let the pool power settings start the desktops as users log in.

$NameSpace = "ROOT\citrix" 
$ClassName = "Citrix_PvDPool"

# Change this value to the name of the desktop group you want to reset
$DesktopGroupName = "My PVD Desktop Group"

# Add Citrix PowerShell snap-ins
asnp Citrix*

Write-Host 
Write-Host "Preparing to reset the Personal vDisk on all VMs in Desktop Group " -nonewline; Write-Host -foregroundcolor green """$DesktopGroupName"""
Write-Host
Write-Host -foregroundcolor cyan "IMPORTANT" -nonewline; Write-Host " - please ensure all VMs are in a started and registered state!"
Write-Host

#Insert a pause
Write-Host "Press any key to continue..."
$x = $host.UI.RawUI.ReadKey("NoEcho,IncludeKeyDown")

# Work out if all desktops are registered
$PVDDesktopGroup = Get-BrokerDesktopGroup -name $DesktopGroupName
if ($PVDDesktopGroup.DesktopsAvailable -lt $PVDDesktopGroup.TotalDesktops)
{
Write-Host
Write-Host -foregroundcolor cyan "WARNING" -nonewline; Write-Host " - Not all desktops appear to be registered, are you sure you want to continue? If so, press any key..."
$x = $host.UI.RawUI.ReadKey("NoEcho,IncludeKeyDown")
}

$PVDDesktops = Get-Brokermachine -DesktopGroupName $DesktopGroupName

Write-Host
Write-Host "Resetting Personal vDisks for the following machines:"

# Perform the PVD Reset via WMI
foreach ($PVDDesktopsMachine in $PVDDesktops)
{
Write-Host $PVDDesktopsMachine.DNSName
Invoke-WMIMethod -Class $classname -ComputerName $PVDDesktopsMachine.DNSName -Namespace $namespace -Name Reset
}

Write-Host
Write-Host "All done!"
Write-Host