Citrix : Slow Login, Profile Cleanup Script

Ran into a strange issue today with two Citrix servers, both suddenly started to exhibit extremely long logins of ~3 minutes – this initially meant that users could not access any published applications on the servers due to the default LogoffCheckerStartupDelayInSeconds timeout of 60 seconds. Sessions would start, hang at the “Welcome” stage of the Citrix application startup and then the session would terminate after 60 seconds.

We modified this registry key to 180 seconds and applications would then launch, workaround in-place, but what was really causing the issue?

After a LOT of troubleshooting it was discovered that there were 100+ profile folders on the servers. We scripted cleanup of these profiles using the script below, login times dropped to ~25 seconds.

Continue reading “Citrix : Slow Login, Profile Cleanup Script”

Citrix : Enable Session Time Zone Redirection

Citrix : Enable Session Time Zone Redirection

It may be that like me, you have Citrix servers in one time zone and users in another. It is possible to configure per-session time zones based upon the connecting client device time zone. I have tested this with XenApp 5.0 running on Windows Server 2003 R2 x64 SP2, with clients connecting via Citrix Online client v10 & 11, RDP and Itium thin client devices.

First you need to add the following registry entries on each server:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\CtxHook\AppInit_Dlls\PerSessionTimeZone]
“FilePathName”=”tzhook.dll”
“Flag Data Type”=dword:00000004
“Settings Data Type”=dword:00000001

Next, in Group Policy enable Time Zone redirection:

Computer Configuration >  Windows Components > Terminal Services > Cliet / Server Data Redirection > Allow Time Zone Redirection > Set to Enabled

Finally, in a Citrix policy:

HDX Plug and Play >  Time Zones > DoNot Use Clients’ Local Time >Disabled

This requires no reboots of any kind, a new session logon should show that the redirectionis working.

Citrix : LMC Authentication Error

Citrix : LMC Authentication Error

On opening the LMC you may receive the following error:
   “You did not authenticate correctly. Please try again or contact your System Administrator.”

To resolve this review the users listed in the following configuration file:
   %ProgramFiles%\Citrix\Licensing\LMC\Tomcat\conf\tomcat-users.xml

I experienced this error after migrating an administartive user account to a new domain, the account definition in the xml file was listed under the old domain. Modifying the old domain NETBIOS name to match the new domain NETBIOS name resolve the authentication issue.

It may be necessary to restart the Citrix Licensing Service after modification of the file.

Citrix : The RPC server cannot be contacted on server .

Citrix : The RPC server cannot be contacted on server .

This issue has plague me infrequently over the last 3 – 6 months – a Citrix server in a PS4.5 farm would suddenly be unable to use the Citrix Access Management Console as when the discovery process was running it would report:

 The RPC server cannot be contacted on server .

The solution for this is simple. If the IMA Service has been restarted or terminated unexpectantly the Citrix COM+ components will sometimes fail to refresh. To resolve this perform te follwoing steps:

  • Terminate ConfigMgrSvc.exe’ using Task Manager
  • Re-open the Citrix Access Management Console and Run Discovery.

 Further information is availabke here: http://support.citrix.com/article/CTX116752

Citrix / Terminal Server Performance Registry Settings

Terminal Server / Citrix Performance Registry Settings

I have gathered a list of registry and operating system tweaks that improve Citrix performance. I use these tweaks on all Citrix servers deployed in order to ensure reliable performance when under heavy user load.

 

Registry Modifications

Firstly, we disable paging of the NT Executive – this keeps core system components in memory and out of the page file. If there is only one tweak you take away with you today, this should be it:

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\LanmanWorkStation\Parameters]
“UtilizeNTCaching”=dword:00000000

 

Next, I configure addition worker threads to increae available CPU threads to users:

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Executive]
“AdditionalCriticalWorkerThreads”=dword:00000016
“AdditionalDelayedWorkerThreads”=dword:00000016

 

Now we increase the functionality of the lanmanserver service which controls file and print resource / access on the server:

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Lanmanserver\Parameters]
“MaxWorkItems”=dword:00002004
“MaxMpxCt”=dword:00000800
“MaxRawWorkItems”=dword:00000200
“MaxFreeConnections”=dword:00000064
“MinFreeConnections”=dword:00000020

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\LanmanServer\Parameters]
“IRPStackSize”=dword:0000000f

 

Now we configure the lanmanworkstation service which is the file and print client:

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\LanmanWorkstation\Parameters]
“MaxThreads”=dword:000000ff
“MaxCollectionCount”=dword:0000ffff
”MaxCmds”=dword:00000800

 

Operating System Configuration

 

Firstly, change the server processing scheduling and memory usage bias towards programs:

 

Next change the performance bias on the network file and print sharing fr Microsoft networks to maximaise data throughput for network applications:

 

 

Troubleshooting Citrix Session Poor Response / High Latency

Troubleshooting Citrix Session Poor Response / High Latency

I was recently tasked with troubleshooting very poor performance on a Citrix Presentation Server 4.5 Advanced Edition Farm.

Hardware Requirements

Your first port of call should be server specification: is the server ‘man’ enough for the task being asked of it? Use the built-in Windows Performance Counters to troubleshoot here.

Check you CPU usage and troubleshoot specific processes if your CPU utilisation is very high. For my environment CPU usage was at < 5%; this was not the governing performance issue.

Memory utilisation can also hinder performance. As a Citrix server runs out of RAM the number of pages /second dramatically increase. Memory usage was circa 50% and the number of pages /second was low.

Network utilisation; whilst ICA is a low-bandwidth application other applications on your environment may be increasing network latency due to saturation of the network link. Use the built-in Windows Performance counters and your switch management tools to ascertain if this is your governing issue. For my environment network usage was < 5% on a 100MB Full-Duplex link.

Network Problems

Are there specific problems with your network that are causing peaks in latency and dropped packets?

Using the Metaframe Servers SDK (MFCOMSDK) v2.3 tool; smcconsole.exe I was able to monitor individual user sessions.

 

Using this tool you can view individual sessions bandwidth utilisation and latency. This tool is incredibly useful when troubleshooting issues regarding session performance. Session latency can also be viewed using the WMI performance counters for ICA Session that are installed when Citrix is installed on a Windows Server.

 

The Metaframe Server SDK version 2.3 is available from here

 

The image above shows a latency figure of 32ms. This equates to 0.03 seconds – a more than acceptable latency figure for an ICA session. When troubleshooting my issues I was receiving figures of 27000ms (yes, 27 seconds!).

Common causes of high latency are:

Ø Network topology issues including port mismatches

Ø MTU issues

Ø Link saturation / QoS

 

A quick and easy check, which should identify any serious network issues, is to conduct what I call a ‘loaded ICMP echo request’ from a network that is experiencing the latency issues to a server in the Citrix farm. A normal ICMP echo request is 32 bytes; we are able to load the packet with up to 1500 bytes. This is achieved using the following command:

> ping ctxserver1 –t –l 1472 -f

 

Let me explain the command. The ‘-t’ option forces the ping to repeat until instructed otherwise (i.e. cancelled with Ctrl-C). The ‘-l 1472’ option sets a packet length of 1472 bytes; there is a 28byte packet overhead therefore, the total packet size is 1500.Finally, the ‘-f’ option forces the packet to not fragment over multiple packets.

 

First, verify that the MTU for your network is in fact 1500 bytes. You can verify this by using the same test to other servers and devices across your network. If you see many dropped packets you know there is a network fault, which may well be the cause of your performance woes.

 

Using this test I identified that there was a fault. The next step is to identify where this fault is occurring.

 

Use ‘tracert’ to identify the path that your packets travel in order to reach your citrix server. Then perform this loaded ping test to each of these addresses one at a time.

 

Configuring & Replicating SpeedScreen in Citrix PS 4.5

Configuring & Replicating SpeedScreen in Citrix PS 4.5

‘Speedscreen’ is a very useful feature built in to all versions of Citrix Presentation Server 4.. Configuring this feature is remarkably easy, but it is also remarkably easy to overlook.

For more information regarding speedscreen functionality and benifits see the ‘Presentation Server 4.5 Bandwidth & Usability Study in Graphics-Rich Scenarios’ whitepaper at the following URL: http://www.citrix.com/English/ps2/products/documents_onecat.asp?contentid=186&cid=White+Papers

To configure SpeedScreen log into a Citrix Server which has the Citrix Toolbar / Administration Tools installed. From the Administrative Toolbar select the following Icon:

 

You will then be presented with the following window:

You can see that this server has been configured for SpeedScreen on all of the listed executables. To add SpeedScreen functionality to another application simply click ‘New… ‘ you will then be presented with a Wizard which will ask you to browse for the desired executable to utilise SpeedScreen with.

Replicating Configuration Between Servers

If you have more than a couple of Citrix Servers in your environment the last thing you want to do is set this up manually on all servers. There is a very simple and quick way of replicating your SpeedScreen configuration between all servers.

Browse the filesystem of a server which has been configured to utilise SpeedScreen, copy the follwoing folder: %Citrix-Install-Dir%\ss3config to all servers under the Citrix Installation Directory. Note this folder may also be under %windir%\system32\ss3config

Provided you have enabled Speed Screen at the Farm level via the Access Management Console you’re good to go!

 

 

 

Understanding and configuring the Citrix XML Service

Understanding and configuring the Citrix XML Service’, ‘Recently caught out by modifying the Citrix XML Service port I thought I would share my experiences!

 
Citrix XML Service Port / ctxxmlss

The Citrix XML Service Port is used by the ICA Client for connection to the Citrix server / published application:When TCP/IP + HTTP is selected and you specify servers in the Address List box, the client communicates with the Citrix XML Service on a specified server for Enumeration.If you modify the XML service port from port 80 and rely on your clients to connect via HTTP & TCP/IP using the dns host entry for ‘ica’ for round-robin DNS resiliency you will find that this round-robin DNS for this entry will fail. This is because you cannot specify the port number, which the XML service is running on in DNS.Therefore, if the first Citrix server in your farm becomes unresponsive or is taken offline connections to the farm will failAs a result you need to configure your clients to use the default server address if ica:pn where pn is the port number you are using for the XML Service. For example’; ica:8080:

 This can be manually specified in an unattended install of the ica client. Run msiexec /a ica32pkg.msi and create an extracted network install source. Then once created edit the \\yourserver\yourshare\ Program Files\Citrix\Application\ICA Client\appsrv.ini file and add the following line at the end of the file:

HttpBrowserAddress=ICA:8080

This will also affect Thin Client devices that utilise HTTP & TCP/IP. For example WYSE 1200LE and S10 Thin Client devices. The solution for these devices is to edit the wnos.ini file on you FTP server so that the port number is specified:

browserip=10.0.0.1:8080,10.0.0.2:8080,10.0.0.3:8080,10.0.0.4:8080

You’ll find that without this if the first server in the list goes offline the TC devices will NOT connect to the next server in the list.

 

Changing the XML Service Port

You have two options when configuring the XML Service port; one, run the XML Service alongside IIS; two, run it on a dedicated port.To configure the XML service to run alongside IIS on port 80 see the following guide:

http://support.citrix.com/article/CTX107683

To configure the XML service to use a dedicated port:

First un-register the XML Service on the server you wish to modify the port:ctxxmlss /u

Now re-register the service on your desired port number:ctxxmlss /r8080

Troubleshooting Citrix Slow Performance Issues

Troubleshoot Citrix / Thin Client Performance’

After a long project that was aimed at improving Thin Client performance I though I would post my experiences and solutions in order to aid those in a similar situation.

 

Citrix Server Performance Improvement

I was recently tasked with improving a Citrix XP and PS 4.5 Farm’s performance; by no means was this simple project which I could simply throw more servers at the farm hoping to resolve the issue.

By far, the most useful tool in diagnosing slow logons is the userenv.dll debugging available in your Windows out of the box. This will really spell out where your problem is coming from.

For further information read this link: http://support.microsoft.com/kb/221833

 

External File Server Performance

External file servers, especially servers holding roaming user profiles can cause significant delays; if these are running out of free connections or worker threads then logon delays are inevitable.

Symptoms: Long pause / very slow / hangs at logon ‘Loading Your Personal Settings”

Long logon delays often indicate issues with remote file access; namely GPO’s and Profile data if roaming profiles are used. Not only are these logon delays a nuisance for end-users, they have a knock-on effect; the duration of the delay often effects all users on a particular server. I have seen logon delays of 50+ second’s effect all users on a single server until the logon process has finished for the user

To Diagnose: Use userenv.dll debugging – http://support.microsoft.com/kb/221833– log file is located under %Systemroot%DebugUserModeUserenv.log.

Solution: Watch out for ‘Srv’ events in the System Event Log with Error code ‘2022’; see the following KB article for more details: http://support.microsoft.com/kb/317249I would definitely suggest rolling out the MaxFree Connections /MinFree Connections registry tweak described in more detail here: http://support.microsoft.com/kb/830901 Note that Windows Server / Advanced Server 2000 require a hotfix, which is free to obtain form MS Technical support.The following web site is also a great resource: http://support.microsoft.com/kb/324446 – if you’re running RAID cards with battery backup units get the Delayed Write Cache setting enabled!

NOTES: Please note that Microsoft does not support the use of PST files across a network. This can cause significant performance issues to file servers hosting them. For further details please see: http://blogs.technet.com/askperf/archive/2007/01/21/network-stored-pst-files-don-t-do-it.aspx If you’re hosting PST files on the same server as your profiles you’ve more than likely found your problem. I would suggest separating the profiles and PST files on separate servers. Profile access needs to be quick to ensure smooth logons.

 

Active Directory Access

Slow access to domain controllers, namely Global Catalogue (GC) servers can cause significant delays in logon as group memberships are referenced and permissions are established from the Active Directory.If you have only a single domain in your forest each Domain Controller can be setup as a GC server. In a multi-domain forest you should ensure that the Infrastructure Master FSMO role is not placed on a GC. The first DC in a domain is always automatically configured as a GC, subsequent DC’s are not.

Symptoms: Long pause / delay / hang / slow at logon “Applying computer settings” and loading Logon Scripts

To Diagnose: Use userenv.dll debugging – http://support.microsoft.com/kb/221833 – log file is located under %Systemroot%DebugUserModeUserenv.log.

Solution: Setup dedicated DC’s; DC’s are central to yourActive Directory Domain. Quick access for LDAP queries is essential for performance. Running print/file server roles on these servers is simply not smart and not reccommended.

 

Citrix Server Hardware / Number of Users Per Citrix Server

There are many myths about the number of users you can effectively have on a single Citrix server. I have seen single servers handle 60 users without any issues what so ever. I have seen servers struggle to handle 20 users when applications or external problems, such as file server access, can cause slowdowns. There isn’t a Citrix reccomended number of users per server. This limit is dictated by the applications your user operates during their session. The only way to find out what your Citrix servers can handle is to test them.

Symptoms: High CPU/ Memory / Page File usage on all Citrix servers within a farm.

To Diagnose: Create a performance benchmark using the built in Window Performance counters. You’ll know if this is an issue when you examine the results.

Solution: Setup and introduce further servers into a farm. Unless you’re seeing high CPU/RAM usage there is little point in adding more servers to the farm; your problem is elsewhere my friend.

 

Logon Scripts

It’s worth noting at this point a poor logon script can cause more problems than the few issues it may automatically fix. Avoid, where possible, calling network applications held on File servers – these shares will be in high demand at peak hours and could cause delays.Script type; I’m not going to get into which is better and which is worse programming language wise. I’ve had great success implementing vbscript over KIX scripts and DOS scripts; this may not be the same in your environment.Scripts to look at in particular; • Scripts being called by UsrLogn2.cmd (found under %SystemRoot%System32)• Group Policy Active Directory Account Logon Scripts

Symptoms: Long pause after the ‘Applying your personal settings’ box disappears.

To Diagnose: Test a user account with the same profile settings other than logon script; ensure it has no logon script.

Solution: Scale back / Streamline your scripts where possible. Alternatively you’re looking at a long night rebuilding them. There is no one-fix-fits all here; your scripts are bespoke to your network… good luck!

 

Network Adapter Configuration

UPDATE 31/01/2008: Simple, yet easy to overlook is the Network Adapter configuration.

Symptoms: Running Citrix Presentation Server 4.5 on Windows Server 2003 I experienced delays of up to 5 minutes for some user accounts whilst logging on. Specifically the logon would get stuck at ‘Loading your personal settings.’

Solution: The cause was simple; a network configuration mismatch. The switch to which the serevr was connected was configured for auto, as was the server. The link infact had auto-negotiated to 10Mb Half Duplex. Forcing the server to 100Mb Full-Duplex reduced logon to around 15 seconds.This can be explained by the use of roaming profiles. The delay was caused by the slow NIC configuration. This means that copying users roaming profiles took up to 5 minutes prior to logon.

 

Antivirus Configuration

UPDATE: 27/09/2009: Antivirus software should be installed and configured correctly for Citrix XenApp/Presentation Server in order to ensure that there is no performance overhead.

Symptoms: Generally slow performance across all applicationsand file access.

To Diagnose: TEMPORARILY disable all anti-virus components (especially the on-access scanner and any application filters/buffer overflow protection)

Solution: You should configure the anti-virus on-access scanner as follows:

• Scan on write events only
• Scan local drives only
• Exclude the pagefile from being scanned
• Exclude the Print Spooler directory to improve print performance
• Exclude the Program FilesCitrix folder from being scanned (the heavily accessed local host cache and Resource Manager local database are contained inside this folder)
• If ICA pass-through connections are used, exclude the user‘s XenApp Plugin bitmap cache and the XenApp Plugin folders

More information is available here

 

Antivirus Configuration

UPDATE: 11/11/2009: If using McAfee Virus Scan 8.7i ensure that at least patch version 2 is installed.

Symptoms: Slow Windows startup and logon performance. Windows boot takes several minutes and gets stuck on ‘Applying Computer Settings…’

To Diagnose: Set the ‘Network Location Awareness’ service startup type to ‘Automatic’

Solution: Install patch 2 for McAfee 8.7i – there is a known issue with version before this causing network communication requests to be sent prior to the ‘Network Location Awareness’ service starting

 

Session Latency

UPADTE 26/02/2010: I thought I would streamline this article, incorporating an additional troubleshooting step from another article in the cb-net archives.

Symptoms: Slow responses when entering text into applications. Refresh of application GUI appears slow, menus etc appear ‘sluggish.’

To Diagnose: Use the Metaframe Servers SDK (MFCOMSDK) v2.3 tool; smcconsole.exe. Using this tool you can view individual sessions bandwidth utilisation and latency.This tool is incredibly useful when troubleshooting issues regarding session performance. Session latency can also be viewed using the WMI performance counters for ICA Session that are installed when Citrix is installed on a Windows Server.

SolutionWhen troubleshooting my issues I was receiving figures of 27000ms (yes, 27 seconds!).

Common causes of high latency are:
  Ø Network topology issues including port mismatches
  Ø MTU issues
  Ø Link saturation / QoS misconfiguration

I have seen latency figures as high as 27,000ms (yes, 27 seconds!) due to NIC / switch port mismatches.

 

Speed Screen Configuration

Symptoms:  Slow responses when entering text into applications

Solution: An often overlooked setting is Speedscreen. Speedscreen will significantly improve the speed at which applications appear to respond to text input from a thin user. You should configure speed screen and replicate settings across the server farm. For instruction see this link:

http://www.cb-net.co.uk/citrix-articles/16-presentation-server/24-configuring-a-replicating-speedscreen-in-citrix-ps-45

 

Virtualised Servers

UPDATE: 28/01/2012

Symptoms: Generally slow performance of virtualised Citrix servers, especially on AMD ESX/ESXi virtualisation platforms. I had similar issues with physical servers which had been converted to virtual servers.

Solution: For AMD RVI deployments beware that on Windows 2003 Hardware-assisted MMU virtualisation (AMD RVI) will not automatically be enabled. This is because of performance related issues in versions of Windows 2003 prior to Service Pack 2. I would suggest that any VM running Windows 2003 SP2 or newer should have hardware MMU manually enabled if your virtualisation platform supports it. You can confirm that Hardware-assisted MMU virtualisation is in use by viewing you vmware.log file that is stored alongised the vmx file, look for virtual exec = ‘hardware’; virtual mmu = ‘hardware’

Less is more; just because your old platform had 4 physical CPU’s, or even more, doesn’t mean that the virtualised platform will perform better. I’ve run 50 users on a single VM with 4GB RAM and 2vCPU’s – performance was good!  Also check the %RDY and MLMTD values for you Citridx VM’s in esxtop; these counters can help identify CPU contention or limits that are affecting VM performance. %RDY should always be below 10-15% higher than this and it’s likely you have an over subscribed host – try reducing physical to virtual CPU ratio’s first. With regard to MLMTD; this should be carefully considered – if this has a value it means that ESX is limiting resources to your VM due to limits you have set (i.e. CPU MHz limits). Further ESX/ESXi performance troubleshooting steps can be found here: http://www.cb-net.co.uk/vmwareesxi-articles/32-performance/61-vmware-troubleshooting-vm-performance