XenDesktop 7.1 Studio XML Error

On several customer sites now I have seen the error displayed in the screenshot below, when clicking into the Seach page whether directly or via something like right-click a Delivery Group and shoosing “Show Machines”.  Note that clicking Close lets you continue to use the Studio console as normal.

There is an error in XML Document (12,11)

“There is an error in XML document (12,11).”  Clicking the “View Error Details” button doesn’t reveal anything immediately useful, but if you look closely there are some nuggets of information hiding there that hint at the cause:

Error Id: XDDS:14743EA8

Exception:
 System.InvalidOperationException There is an error in XML document (12, 11).
 at System.Xml.Serialization.XmlSerializer.Deserialize(XmlReader xmlReader, String encodingStyle, XmlDeserializationEvents events)
 at System.Xml.Serialization.XmlSerializer.Deserialize(TextReader textReader)
 at Citrix.Console.CommonControls.Mmc.SearchTabbedResultPaneViewModelBase.GetSavedSearchesFromDisk()
  
Inner Exception:
 System.ArgumentNullException Value cannot be null.
 Parameter name: enumType
 at System.Enum.GetValues(Type enumType)
 at Citrix.Console.Models.Search.EnumSearchFilterTerm.UpdateValue()
 at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderObservableCollection1.Read10_EnumSearchFilterTerm(Boolean isNullable, Boolean checkType)
 at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderObservableCollection1.Read14_SearchFilterTermModel(Boolean isNullable, Boolean checkType)
 at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderObservableCollection1.Read15_SearchFilterModel(Boolean isNullable, Boolean checkType)
 at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderObservableCollection1.Read16_AdvancedSearchModel(Boolean isNullable, Boolean checkType)
 at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderObservableCollection1.Read17_ArrayOfAdvancedSearchModel()

The word “GetSavedSearchesFromDisk” drew my attention.  I searched the user profile folder and the ProgramData\Citrix folder for Citrix related XML files, and found just one:

C:\Users\<username>\AppData\Local\Citrix\DesktopStudio\SearchResultsPaneViewModel.xml

Viewing this file in Notepad it seemed to contain a saved search of some description.  I experimented with removing this file and re-opening the Studio console to the searches page, and the error did not appear.  Creating a new search in the Studio console and saving it created a corresponding new XML file.  Curiously enough after this new search was saved, closing and re-opening the Studio console brought back the error, so I can only assume there is a bug in the XenDesktop Studio console that does not allow this file to be properly parsed once it has been created.

So – if you get this error, it seems you can’t take advantage of the Saved Searches feature, you will have to remove the XML file and make do without it for now.  I’m going to log a ticket with Citrix Support about this issue and will update the page when I have a response or fix.

XenMobile Device Manager and SAN Certificates

I knew that most Citrix components work with SAN certificates, as per the eDocs at http://support.citrix.com/proddocs/topic/xmob-deployment/xmob-deploy-certificates-con.html.  However when installing my first site that wanted to use a SAN certificate for their XenMobile Device Manager server, it would not accept the SAN certificate during the setup process.

xm-sancert

Note in the above screenshot, the certificate and password are accepted but you cannot click Next, and the FQDN and other details are not picked up from the certificate.

There is an easy way round this however, instead complete your XenMobile install using the self signed certificate that gets generated during the install.  Then swap the self-signed certificate for your SAN certificate by following this excellent post from Port25Guy:

http://port25guy.com/2013/11/18/import-a-3rd-party-certificate-into-xenmobile/

Once you have followed the above instructions and restarted the XenMobile service, the console should be accessible, and devices able to communicate with the XDM server and enroll etc using the SAN certificate without complaints.

XenDesktop 7 upgrade and Citrix Policy errors

You may encounter an issue after upgrading XenDesktop 7 where upgraded policies in the Citrix Studio cannot be edited or deleted, and you receive the error “The given key was not present in the dictionary” as per CTX138498.  What the article doesn’t tell you is some of the additional steps and requirements in order to remediate your policies.  When you setup the temporary server, you will need to:

a) need to choose the option to install SQL Express, or have a SQL server available elsewhere on your network.

b) You need to configure a temporary XenDesktop site, don’t try and add the XenDesktop 5 controller back into your XenDesktop 7 site.  Also when running the wizard to configure the temporary site you don’t need to choose a hypervisor, just pick “None”.

c) If your old pre-XD7 controllers had the Citrix Group Policy 1.7 update applied (which is part of XenDesktop 5.6 FP1) you will need to re-install this as well.  Otherwise you will receive errors such as “Found invalid data while decoding” trying to view your imported policies on the XD5 server.  If you need to re-download this component, go to http://www.citrix.com/downloads/xendesktop/product-software.html (login required).  Drill into XenDesktop 5.6 Feature Pack 1, choose your edition, and download and install the “HDX and Group Policy Update”.

If this information was present in CTX138498 it would have saved me a good couple of hours while I figured all this out – hopefully I can save someone else some time instead!

XenServer iSCSI LUNs not mapping

I’m not a big fan of iSCSI – probably a hangover from years of fibre channel experience before ethernet based storage networks became commonplace – and the issue I ran into today didn’t do anything to increase my comfort level.  My experience to date is that fibre channel storage systems seem to “just work” whereas iSCSI can be finnicky and temperamental.

A customer had purchased a new HP P2000 G3 iSCSI storage array to provide a shared storage repository for XenServer HA for the blades hosting their XenDesktop farm, in addition to providing a storage location for templates and a handful of test VMs.  I configured two ports on each P2000 controller and assigned two NICs from each XenServer blade for iSCSI traffic as per the XenServer multipathing best practice guide which you can find at CTX136354.  I enabled multipathing on all hosts in the farm and proceeded to create the storage repository.

After creating the SR it was visible on one host only, and on the rest it showed with a status of “Unplugged” in XenCenter.  Watching the SMLog file while trying to “repair” the SR from the GUI, or running “xe pbd-plug uuid=…” from the command line, generated errors similar to the below:

Aug 21, 2013 4:27:14 AM Error: Repairing SR P2000_HA - Internal error: Failure("Storage_access failed with: SR_BACKEND_FAILURE: [ non-zero exit; ; Traceback (most recent call last):\n  File \"/opt/xensource/sm/LVMoISCSISR\", line 549, in ?\n    SRCommand.run(LVHDoISCSISR, DRIVER_INFO)\n  File \"/opt/xensource/sm/SRCommand.py\", line 250, in run\n    sr = driver(cmd, cmd.sr_uuid)\n  File \"/opt/xensource/sm/SR.py\", line 136, in __init__\n    self.load(sr_uuid)\n  File \"/opt/xensource/sm/LVMoISCSISR\", line 150, in load\n    self.iscsi = self.iscsiSRs[0]\nIndexError: list index out of range\n ]")

Needless to say this didn’t make a lot of sense.  Eventually after running out of ideas, some semi-random googling uncovered that the /etc/iscsi/initiatorname.iscsi file was missing (on 10 of 11 servers in the pool!!) and was not recreated by changing the IQN in the XenCenter console.  To fix this, I ran the following commands (note the initiator name must be the same as what is set in the XenCenter console)

[root@XenHome ~]# echo InitiatorName=iqn.2011-07.com.xenserver01:10f967a6 > /etc/iscsi/initiatorname.iscsi
[root@XenHome ~]# echo InitiatorAlias=XenServer01 >> /etc/iscsi/initiatorname.iscsi
[root@XenHome ~]# /etc/init.d/open-iscsi stop
[root@XenHome ~]# /etc/init.d/open-iscsi start

To test iSCSI was now operational, I ran the following command (replace the IP address with the address of your iSCSI SAN):

[root@XenHome ~]# iscsiadm -m discovery -t sendtargets -p 192.168.1.10

A list of target LUNs was returned, I was able to successfully “repair” the SR and get on with my day.

XenApp 6.5 HRP02 install hangs

Citrix recently released the eagerly anticipated HRP02 for XenApp 6.5, which contains around 87 hotfixes bundled into this one rollup.  You can download it here.

While installing this patch onto several servers in a farm, most of them worked fine however on one server the patch installation hung part way through (at around 90%) with no apparent errors or issues recorded in the event log.

HRP02_install_hang

 

Restarting the server and retrying the install with AV disabled, Citrix services stopped, third party services stopped had no effect.  Next step was to enable logging by installing the patch using the following command line:

C:\Temp> msiexec /p XA650W2K8R2X64R02.msp /L*V c:\temp\hrp02.log

Examining the resulting log file revealed the issue – this particular server had the XenApp install run from a UNC network share that no longer existed, and could not locate the original install media:

HRP02_logA quick registry search revealed this original UNC path of the install media was set in multiple places in the registry, for each of the individual components that make up a XenApp install (eg base XenApp install, Flash extensions, Delivery Services console etc all have individual installers).  After a bit of find and replace work, and a restart of the server, the patch install completed successfully!

Winning with StoreFront!

OK, so winning might be stretching it a little bit, but I’ve been spending a lot of time on StoreFront lately and I’ve gathered a few tips and figured out a few things to help your StoreFront deployment go a little smoother.  On top of this these tips will (hopefully) make your StoreFront deployment work a little faster, but I still can’t get it to operate as quick as Web Interface.

What I won’t cover is how to set it up, load balance it, or front it with Access Gateway/NetScaler as there are plenty of guides already out there for that.  Some examples are here, here, here, and more – just google it.

Access Gateway builds

You need to use a .e, or “Enhancement” build to work properly with StoreFront.  As at time of writing, the recommended version is 10.0.73.5002.e which you can download from Citrix here (login required).  In the future you will be able to use mainstream builds, however even with the initial 10.1 release of NetScaler this is not yet the case (AGEE feature is still in tech preview mode).

Certificates

Ensure you have the right certificate on your StoreFront server(s) before you start installing as StoreFront will hard code it’s URL based on the certificate installed and bound to IIS.  Note that even if you have an Access Gateway with a “real” trusted certificate your devices will still need to trust the internal certificate on the StoreFront servers when they are inside your network.  iPads have an option to ignore trust issues with the certificate, but Android and Windows devices do not.

.Net Config Files

Get familiar with the Microsoft .Net config file structure as you will end up editing many of them.  A lot of options, even some that were previously in the GUI with Web Interface, are now controlled by options in the .config files and must be edited by hand which unfortunately makes them more prone to human error.

Aside from configuration options though, an important setting particularly if your StoreFront servers do not have direct internet access is the the following line:

<generatePublisherEvidence enabled="false"/>

As per CTX117273 make the change to the .Net config files in both C:\Windows\Microsoft.NET\Framework and C:\Windows\Microsoft.NET\Framework64.  Then search for every .config file under C:\Program Files\Citrix\Receiver StoreFront and C:\inetpub\wwwroot and check each one.  If it has the following lines

<runtime>
 <!-- Set enabled=false, if this machine does not have external access to the internet -->
 <generatePublisherEvidence enabled="true" />
 </runtime>

then go ahead and change the “true” to “false”. This will disable the checking of the certificate used to sign the code, and part of this process does CRL lookups on the fly. If you server does not have internet access the whole process slows down waiting for the lookups to time out. Even if the server does have direct internet access, disabling all these checks seems to have a measurable improvement in the responsiveness of your StoreFront site.

Note that after you enable certain features, eg you enable the pass-through authentication method, a new folder will be created under StoreFront program files folder with an additional .config file so best to check for these regularly during the setup process and a final check when you are done configuring and installing all features.
In addition to these, you can disable the CRL check in the Internet control panel as per below.CRL Check

Miscellaneous settings

In the WINS tab under Advanced Settings on your network adapter, choose the “Disable NetBIOS over TCP/IP” option.

Change the Startup mode of the “Citrix Credential Wallet” service to “Automatic (Delayed Start)”.

Change PowerShell execution policy to “bypass” using the below command.  Similar to the CRL check, it disables checking for digitally signed scripts and associated CRL lookups.

Set-ExecutionPolicy -scope LocalMachine bypass

Set-ExecutionPolicy

Use DebugView.  It’s a handy tool that will tell you why something is failing when you get no feedback from the console or event viewer.  The logfiles it generates can be pretty large, but it’s worth combing through to see what’s going on.  You can download it from the Microsoft SysInternals page here.

Conclusion

I think that’s about it!  Not all these tips will have an effect in every environment, and please understand there is a security risk that comes with disabling CRL checks.  However I’ve had much greater success with StoreFront with making all the changes here so now they are part of my standard build procedure for new installs.

If you’ve found something that works for you and I haven’t mentioned it here, please leave a comment and I’ll update the post.

UPDATE for StoreFront 2.0

Fortunately for those installing StoreFront 2.0, Citrix have made life a little easier and in every .config file I have looked at so far (generated by StoreFront), the generatePublisherEvidence setting is already set to “false”.

Everything else I’ve written here still applies, including the generatePublisherEvidence setting in Aspnet.config files under the various C:\Windows\Microsoft.NET folders.  Note also a new KB article from Citrix on speeding up the first time login for StoreFront, CTX137400.

StoreFront WTF?

Citrix’s Web Interface replacement product, StoreFront, has certainly given me many clenched teeth moments recently (blog post coming on this soon).  However this WTF momemnt today made me chuckle…

Storefront_WTF

Technically it’s probably a Google Chrome issue as none of the other browsers I tried offered to translate the page, I guess I just found it amusing given some of the other trials I’ve had with it recently :)

XenServer 6.0.2 hotfix and driver disk install summary

Confused about the multitude of XenServer 6.0.2 hotfixes and associated drivers currently available? The public hotfixes and drivers are listed here and as of April 2013 totals 88!  It’s not as daunting as it looks though – there are currently 21 hotfixes, the remainder are the drivers associated with various versions of hotfixes.  And of the 21 hotfixes a number of the older ones are superceded and included with newer hotfixes, but there are some dependencies so they need to be installed in the correct order.

Firstly, download the required hotfixes.  As mentioned we don’t need all 21 – so download the following 5 hotfixes (that’s 16 less reboots to do!)

Hotfix 6 CTX134130
Hotfix 10 CTX135225
Hotfix 19 (includes hotfixes 1 and 2) CTX137134
Hotfix 20 (includes hotfixes 4, 8, 14, 16, 18) CTX136478
Hotfix 21 (includes hotfixes 1, 3, 5, 7, 11, 13) CTX136479

Hotfix 21 links to an article (CTX136621) listing the various upgraded drivers to go with it.  To work out the drivers you need, fire up a XenServer command line via the physical console, XenCenter or SSH.  Enter the “lsmod” command to return all running modules and drivers, or combine it with the egrep command to narrow the search down.  Eg the driver page shows the modules that require updating as per below graphic (this is not the complete list, cropped for brevity):

XS602E012-Drivers

So to search for these drivers, issue the command as follows:

[root@XenHost ~]# lsmod | egrep 'bnx2x|bnx2|tg3|cxgb3|cxgb4'

Any lines that get returned indicate the drivers that are installed and need updating.  If no lines get returned, you have no drivers that need updating :)

Now we are prepared, install the hotfixes in the following order:

1 – Hotfix 6
2 – Hotfix 10
3 – Hotfix 20
4 – Hotfix 19
5 – Hotfix 21
6 – Drivers (if required)

Time and enthusiasm permitting, I’ll attempt to keep this post updated as future hotfixes get released – happy patching!

Citrix Profile Manager and cookies

If you’ve implemented Citrix Profile Manager you will no doubt be aware of the policy options to manage cookies and the index.dat file (see here)

http://support.citrix.com/proddocs/topic/user-profile-manager-sou/upm-manage-cookies.html

What isn’t mentioned anywhere (that I’ve been able to find, anyway) is that if you are using Windows 7 / 2008 R2 and redirect the APPDATA folder using Windows group policy settings, then the Citrix policy settings for cookie processing are not required and do not take effect.  This is because the cookies folder resides under the APPDATA folder (%userprofile%\AppData\Roaming\Microsoft\Windows\Cookies) therefore if you redirect it, cookies will never be stored on the local machine for the Citrix profile manager to process.

For Windows XP / Server 2003, the cookie processing settings are still required as the Cookies folder is %userprofile%\Cookies.

IBM BladeCenter CEE Switch connecting to Cisco switch

While installing an IBM BladeCenter H that had a Brocade Converged Enhanced Ethernet Switch module (IBM part# 61Y1909, Brocade model 8470) I found pretty much zero documentation on how to connect it to an external Cisco switch.  So if you are trying to figure this out yourself, hopefully I can save you the trouble.

On the Cisco switch, I had 2 x 10Gb ports connected to the Brocade.  The config on those ports is as follows:

interface TenGigabitEthernet1/5
 description -- Connected to Brocade CEE SW01 --
 switchport trunk native vlan 111
 switchport mode trunk
 channel-protocol lacp
 channel-group 2 mode active
end
interface TenGigabitEthernet1/6
 description -- Connected to Brocade CEE SW01 --
 switchport trunk native vlan 111
 switchport mode trunk
 channel-protocol lacp
 channel-group 2 mode active
end

I wanted to use LACP to bond these two ports together, partly for throughput (which shouldn’t really be an issue 10Gb networking) and partly for redundancy – as you can see the ports are members of a channel group.  The port channel interface config is:

interface Port-channel2
 description -- Connected to Brocade CEE SW01 --
 switchport
 switchport trunk native vlan 111
 switchport mode trunk
end

Note the native vlan statement – I specifically wanted VLAN1 traffic passed to the Brocade, but the Brocade expects all VLANs to be tagged.  The work-around for this was to create a dummy VLAN (VLAN111) and make this the native vlan.  This then ensures VLAN1 will be tagged :)

Now for the Brocade side of the configuration.  First, the internal 10Gb ports will need to be configured as follows:

interface InTengigabitEthernet 0/1
 fcoeport
 switchport
 switchport mode trunk
 switchport trunk allowed vlan all
 no shutdown

The two external ports are configured as follows:

interface ExTengigabitEthernet 0/15
 channel-group 2 mode active type standard
 no shutdown
 lacp timeout long

interface ExTengigabitEthernet 0/16
 channel-group 2 mode active type standard
 no shutdown
 lacp timeout long

Note you will need to issue a “no switchport” command before you can configure the channel group.  Now for the port channel interface itself:

interface Port-channel 2
 switchport
 switchport mode trunk
 switchport trunk allowed vlan all
 no shutdown

While there may be a better way to do it, this is the configuration worked for me and with the lack of documentation on Cisco interoperability I wasn’t going to burn any more time on it.  I know you can set the Cisco switch to not have a default VLAN, like the Brocade, but this would have meant reworking a lot of the Cisco side of the network which I didn’t want to do.