Citrix PVS prep script for Sophos

Some of you will know that Sophos is my, ahem, favourite Sophosanti-virus product.  However it was the product of choice at the time (they are now moving to McAfee) for a large school district I deal with hence I had no choice but to remain acquainted with it irrespective of it’s poor performance (Sophos have advised to leave real-time scanning *off* for performance reasons?!?), it’s poor management console (customer had to write their own deployment scripts for reliable deployment) and poor customer service from Sophos.

I wrote a small script to prepare Sophos for PVS deployment on XenApp and XenDesktop machines, run this script just prior to shutting down your Private mode image and switching back to Standard.

@echo off
REM Script by Daniel Marsh
REM run at shutdown to prep Sophos on Citrix XenApp/VDI machines.
REM refer to https://www.sophos.com/en-us/support/knowledgebase/12561.aspx

net stop "Sophos Agent"
net stop "Sophos Anti-Virus"
net stop "Sophos Anti-Virus status reporter"
net stop "Sophos AutoUpdate Service"
net stop "Sophos Message Router"
net stop "Sophos Web Control Service"
net stop "Sophos Web Intelligence Service"

del C:\ProgramData\Sophos\AutoUpdate\data\machine_ID.txt

reg delete "HKLM\SOFTWARE\Wow6432Node\Sophos\Remote Management System\ManagementAgent\Private" /v pkp /f
reg delete "HKLM\SOFTWARE\Wow6432Node\Sophos\Remote Management System\ManagementAgent\Private" /v pkc /f
reg delete "HKLM\SOFTWARE\Wow6432Node\Sophos\Messaging System\Router\Private" /v pkp /f
reg delete "HKLM\SOFTWARE\Wow6432Node\Sophos\Messaging System\Router\Private" /v pkc /f

When I have a spare few minutes I may even see if I can another feature – it could be run via GPO as a machine shutdown script that only executes if the image is in private mode, which would save you having to run it manually.  I haven’t figured out how to do this yet, if you do know please drop me a comment below.

Refer to https://www.sophos.com/en-us/support/knowledgebase/12561.aspx for more detail on what each key/service etc does.

Machine connection failure – “Refused”

Firstly apologies for the slowdown of blog posts in recent times – between work and family life there never seems to be a free moment, I’ve been meaning to write this one up for a couple of months.

A client with a XenDesktop 7.6 farm of Windows 7 pooled desktops was experiencing, seemingly out of the blue, a large number of machine connection failures.  Most failure reasons listed were the extremely informative “Refused” along with the equally descriptive “Timeout” and “None”.

2015-11-02 10_37_19-Management Desktop - Desktop Viewer

Checking the Studio console showed nothing out of the ordinary, all machines were booted and registered.  Logging onto the console of a machine that had refused a connection also showed nothing unusual, in fact the event log didn’t even have a record of an attempted connection.  Very strange.

My first clue came when trying to open the event viewer remotely to one of the affected desktops.

2015-11-02 10_38_01-Management Desktop - Desktop Viewer

The above error appeared which I wasn’t immediately familiar with, but a quick Google pointed me in the right direction.  Most of the results mentioned kerberos errors relating to mismatched SPNs and account names which got me thinking.  Checking DNS for the hostnames and IPs of the affected virtual desktops showed multiple instances of an IP address being assigned to more than one DNS record.

2015-11-02 10_43_33-Management Desktop - Desktop Viewer

When the connection attempts to a virtual desktop were being made, the broker connection process involves a DNS lookup that in this situation resulted in the wrong IP being returned.

What had happened was the DHCP scope and DNS scavenging settings were misaligned meaning expired DHCP leases did not have their DNS records cleaned up.  For a read-through of recommended settings and how the process works have a look at the following two links:

http://blogs.technet.com/b/askpfe/archive/2011/06/03/how-dns-scavenging-and-the-dhcp-lease-duration-relate.aspx

http://blogs.technet.com/b/networking/archive/2008/03/19/don-t-be-afraid-of-dns-scavenging-just-be-patient.aspx

Long story short, once the stale DNS records were manually purged, then the DHCP lease time was extended from 1 day to 14, and the DNS refresh/no-refresh intervals brought down, everything started to behave again.

PVS Image Updates and Windows Update cleanup

A customer called me in to look at an issue occurring on their Windows 8.1 image that was deployed in their XenDesktop site.  Images were deployed using PVS, using the Write Cache in RAM with Overflow to Disk feature.  Write Cache RAM size was set to 256MB, a fairly standard size for a desktop OS, and the write cache disk was 10GB.  The issue the customer noticed was the write cache was filling very rapidly, and in some cases in as little as 30 minutes after startup.  As PVS admins will know once the write cache fills it’s game over for that VM – it will blue screen or in our case just lock up and had to be forcibly reset using the hypervisor console.

Upon investigation, the write cache was certainly filling rapidly.Write CacheAs you will notice in the above screenshot, write cache usage is over 65% and this was approximately 15 minutes after the VM booted.  Obviously some process was performing a large number of disk writes to cause the cache to fill.  A quick look at the task manager leads us to our first suspect, TiWorker.exe which was consuming a large amount of CPU and also performing a lot of disk IO.  In fact, this was occurring across all VMs on the farm and saturating the host as per the below screenshot.  Rather curiously, it seemed to become worse around midday. (It should be noted there were very few active users on the farm while all this resource usage was occurring.)

XenCenter

 

Looking at an individual VM showed the following:

Win8 Performance

You can see a spike in CPU activity, with a corresponding increase in disk activity.  So, to find out the cause.  A bit of research showed TiWorker.exe was part of the Windows Update engine, but there were no updates being installed at the time, in fact none available as the customer had recently updated this image with the latest Windows Updates.  I came across a Microsoft forum post here that started to shed some light here.  Turns out that periodically, Windows Updates will do some additional compression and cleanup work on the downloaded Windows Updates folder (C:\Windows\WinSxS) and compress the entire lot using the LZX compression routine that was introduced in March 2014 (see here).

What had happened is the client booted the PVS image in private mode, installed the latest batch of Windows Updates then shut down and reverted to Standard mode (with Write Cache in RAM w/Overflow to Disk).  Windows hadn’t completed all these post-Windows Update cleanup routines however, so it kicked them off on the next startup, which happened to be when all VMs were in standard mode.  This filled the write cache on the VMs, they would lock up/freeze/crash, reboot, and start the same process all over again.

The resolution was pretty simple – open the image in private mode again, and let it sit there for Windows Update to do it’s thing.  In this instance, going by the XenCenter performance graph for the VM it took around 4 hours for disk activity to subside.  Once that was complete we reverted to Standard mode, and no more issues!

 

Reset PVD for a Desktop Group via PowerShell and WMI

A client wanted the ability to reset the Personal vDisk for every desktop in a given group, without having to visit each desktop individually via the Director console.  So this is a little quick and dirty, with limited error checking, but I knocked up a PowerShell script to do the job.  Feel free to take it and use it as you see fit, and let me know of any improvements you make :)

The scripts works regardless of whether desktops are in maintenance mode or not, for my purposes it was not necessary to have the script do it but if you are automating this as part of a larger process you may want to add a line that places all desktops into maintenance mode first.  Other potential improvements are adding commands to start all desktops if they aren’t started, and a loop to check when they all become registered.

Copy the text below and save it into a PowerShell script, and execute from a PowerShell prompt on your Desktop Delivery Controller.  You will need to have your execution policy set to unsigned or unrestricted.  Note that desktops will reboot, then shut down as part of the PVD reset process.  You can either boot them manually afterwards, or let the pool power settings start the desktops as users log in.

$NameSpace = "ROOT\citrix" 
$ClassName = "Citrix_PvDPool"

# Change this value to the name of the desktop group you want to reset
$DesktopGroupName = "My PVD Desktop Group"

# Add Citrix PowerShell snap-ins
asnp Citrix*

Write-Host 
Write-Host "Preparing to reset the Personal vDisk on all VMs in Desktop Group " -nonewline; Write-Host -foregroundcolor green """$DesktopGroupName"""
Write-Host
Write-Host -foregroundcolor cyan "IMPORTANT" -nonewline; Write-Host " - please ensure all VMs are in a started and registered state!"
Write-Host

#Insert a pause
Write-Host "Press any key to continue..."
$x = $host.UI.RawUI.ReadKey("NoEcho,IncludeKeyDown")

# Work out if all desktops are registered
$PVDDesktopGroup = Get-BrokerDesktopGroup -name $DesktopGroupName
if ($PVDDesktopGroup.DesktopsAvailable -lt $PVDDesktopGroup.TotalDesktops)
{
Write-Host
Write-Host -foregroundcolor cyan "WARNING" -nonewline; Write-Host " - Not all desktops appear to be registered, are you sure you want to continue? If so, press any key..."
$x = $host.UI.RawUI.ReadKey("NoEcho,IncludeKeyDown")
}

$PVDDesktops = Get-Brokermachine -DesktopGroupName $DesktopGroupName

Write-Host
Write-Host "Resetting Personal vDisks for the following machines:"

# Perform the PVD Reset via WMI
foreach ($PVDDesktopsMachine in $PVDDesktops)
{
Write-Host $PVDDesktopsMachine.DNSName
Invoke-WMIMethod -Class $classname -ComputerName $PVDDesktopsMachine.DNSName -Namespace $namespace -Name Reset
}

Write-Host
Write-Host "All done!"
Write-Host

XenDesktop 7.1 Studio XML Error

On several customer sites now I have seen the error displayed in the screenshot below, when clicking into the Seach page whether directly or via something like right-click a Delivery Group and shoosing “Show Machines”.  Note that clicking Close lets you continue to use the Studio console as normal.

There is an error in XML Document (12,11)

“There is an error in XML document (12,11).”  Clicking the “View Error Details” button doesn’t reveal anything immediately useful, but if you look closely there are some nuggets of information hiding there that hint at the cause:

Error Id: XDDS:14743EA8

Exception:
 System.InvalidOperationException There is an error in XML document (12, 11).
 at System.Xml.Serialization.XmlSerializer.Deserialize(XmlReader xmlReader, String encodingStyle, XmlDeserializationEvents events)
 at System.Xml.Serialization.XmlSerializer.Deserialize(TextReader textReader)
 at Citrix.Console.CommonControls.Mmc.SearchTabbedResultPaneViewModelBase.GetSavedSearchesFromDisk()
  
Inner Exception:
 System.ArgumentNullException Value cannot be null.
 Parameter name: enumType
 at System.Enum.GetValues(Type enumType)
 at Citrix.Console.Models.Search.EnumSearchFilterTerm.UpdateValue()
 at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderObservableCollection1.Read10_EnumSearchFilterTerm(Boolean isNullable, Boolean checkType)
 at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderObservableCollection1.Read14_SearchFilterTermModel(Boolean isNullable, Boolean checkType)
 at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderObservableCollection1.Read15_SearchFilterModel(Boolean isNullable, Boolean checkType)
 at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderObservableCollection1.Read16_AdvancedSearchModel(Boolean isNullable, Boolean checkType)
 at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderObservableCollection1.Read17_ArrayOfAdvancedSearchModel()

The word “GetSavedSearchesFromDisk” drew my attention.  I searched the user profile folder and the ProgramData\Citrix folder for Citrix related XML files, and found just one:

C:\Users\<username>\AppData\Local\Citrix\DesktopStudio\SearchResultsPaneViewModel.xml

Viewing this file in Notepad it seemed to contain a saved search of some description.  I experimented with removing this file and re-opening the Studio console to the searches page, and the error did not appear.  Creating a new search in the Studio console and saving it created a corresponding new XML file.  Curiously enough after this new search was saved, closing and re-opening the Studio console brought back the error, so I can only assume there is a bug in the XenDesktop Studio console that does not allow this file to be properly parsed once it has been created.

So – if you get this error, it seems you can’t take advantage of the Saved Searches feature, you will have to remove the XML file and make do without it for now.  I’m going to log a ticket with Citrix Support about this issue and will update the page when I have a response or fix.

XenDesktop 7 upgrade and Citrix Policy errors

You may encounter an issue after upgrading XenDesktop 7 where upgraded policies in the Citrix Studio cannot be edited or deleted, and you receive the error “The given key was not present in the dictionary” as per CTX138498.  What the article doesn’t tell you is some of the additional steps and requirements in order to remediate your policies.  When you setup the temporary server, you will need to:

a) need to choose the option to install SQL Express, or have a SQL server available elsewhere on your network.

b) You need to configure a temporary XenDesktop site, don’t try and add the XenDesktop 5 controller back into your XenDesktop 7 site.  Also when running the wizard to configure the temporary site you don’t need to choose a hypervisor, just pick “None”.

c) If your old pre-XD7 controllers had the Citrix Group Policy 1.7 update applied (which is part of XenDesktop 5.6 FP1) you will need to re-install this as well.  Otherwise you will receive errors such as “Found invalid data while decoding” trying to view your imported policies on the XD5 server.  If you need to re-download this component, go to http://www.citrix.com/downloads/xendesktop/product-software.html (login required).  Drill into XenDesktop 5.6 Feature Pack 1, choose your edition, and download and install the “HDX and Group Policy Update”.

If this information was present in CTX138498 it would have saved me a good couple of hours while I figured all this out – hopefully I can save someone else some time instead!

PVS and the dreaded 0x0000007B error

A client was trying to setup a lab of HP T610 thin clients to stream Windows 7 from Citrix Provisioning Services 6.1.  The T610 they used to create the image worked flawlessly, they captured the image, and could then stream it back fine in private or standard mode.  The issue appeared when they went to stream it to more than one T610 – while the original machine worked fine, on every other T610 they tried it would blue-screen on startup with the dreaded 0x0000007B error.

“It’s probably something really simple” I said – famous last words – and set about checking for differences between the T610s in the BIOS settings, firmware versions and so on, deleting and recreating machines in the PVS console, recreating the AD accounts and more.  I was already familiar with the “ghosted network card” issue (CTX133188) but this was not the issue.  A couple of hours later and no “simple” fixes in sight – I eventually stumbled across this thread and on page 4 one of the posters references MS hotfix KB2550978.  I booted the VHD in Hyper-V, removed the PVS Target Device software, installed this hotfix and reinstalled the PVS Target device – problem solved!

The issue is the T610 clients support a feature called “PCI Express Serial Number Capability” so from PVS’s point of view, the network card in every individual T610 is seen as a new NIC and needs the driver installed before it can boot successfully.  But as it’s trying to boot across the network, this causes the 0x0000007B error.  Not many clients I have come across stream to physical desktops, so this probably won’t impact too many people, but I hope this info helps someone out!