CRM 2011 Add-in causes Outlook to crash when used on Citrix XenDesktop environment

Today I am taking about a strange issue in a unique environment. Outlook crashes with CRM Add-in on Citrix XenDesktop. Crashes were random, affecting all users, experienced 8-10 times a day, while performing any core Outlook or CRM operation. Crashes were unpredictable as there were no pattern of repro steps which would lead to a crash of Outook.exe.

Had to learn some basics of XenDesktop; its Citirx product that provides client machine virtualization on the fly. VM is created from a source image (source image has OS and all required applications installed). When user initiates/login to a XenDesktop session VM is created from source image and VM is destroyed when user logs out. User profile is stored on a Network location, pulled into session on logon and changes are written back on logout.

As part of normal troubleshooting, isolation all other Add-ins were disabled but crashes continued to happen even when there was only CRM Add-in was enabled.

After checking few other known issue, next step was to enable CRM client platform trace logs using built-in diagnostics utility installed along with CRM for Outlook and wait for next crash to occur. With little bit of user education it was less than half-day and we got the CRM client platform traces witnessing a crash of Outlook. Being a crash, focus was on unhandled exception when I was looking at the client platform trace and below is the relevant extract that was complaining about a sharing violation for one of the CRM client SQL CE database files.

> HandleException: Unhandled Exception: System.Data.SqlServerCe.SqlCeException: There is a file sharing violation.
A different process might be using the file. [ C:\Documents and Settings\berniem\Local
Settings\Application Data\Microsoft\MSCRM\Client\d32489ad-413d-e111-8e2e-005056971f28False.sdf ]
at System.Data.SqlServerCe.SqlCeConnection.ProcessResults(Int32 hr)
at System.Data.SqlServerCe.SqlCeConnection.Open(Boolean silent)
at System.Data.SqlServerCe.SqlCeConnection.Open()
at Microsoft.Crm.MapiStore.ServerStore.<>c__DisplayClass22.b__1f()
at Microsoft.ExceptionHelper.ExceptionFilter.TryFilter[TEx](Action body, Predicate`1 filter, Action`1 catchClause)
at Microsoft.Crm.MapiStore.ServerStore.ExecuteInNewConnection(String entityTypeName, ConnectionActionsDelegate actions)
at Microsoft.Crm.MapiStore.ServerStore.ExecuteInNewConnectionAndTransaction(String entityTypeName, TransactionActionsDelegate actions)
at Microsoft.Crm.MapiStore.DataCache.InclusionUpdateWithCacheDb()
at Microsoft.Crm.MapiStore.DataCache.BackgroundUpdateFromCacheDb(Boolean forceUpdate)
at Microsoft.Crm.MapiStore.ServerStore.<>c__DisplayClass40.b__3e(Object unused)
at Microsoft.Crm.Application.SMWrappers.OutlookInstanceCache.<>c__DisplayClass17.<>c__DisplayClass1a.b__15()
at Microsoft.ExceptionHelper.ExceptionFilter.TryFilterAllCatch(Action body, Action`1 filter)

>Unhandled Exception in MAPIInterop: Microsoft.Crm.MapiStore.MapiServiceEntryPoints+KnownStackTraceException:
Error in the application. at Microsoft.Crm.MapiStore.MapiServiceEntryPoints.MapiApplication_MapiUnhandledException(IntPtr ep)
at ExceptionFilter(_EXCEPTION_POINTERS* ep)
at System.Data.SqlServerCe.SqlCeConnection.ProcessResults(Int32 hr)
at System.Data.SqlServerCe.SqlCeConnection.Open(Boolean silent)
at System.Data.SqlServerCe.SqlCeConnection.Open()
at Microsoft.Crm.MapiStore.ServerStore.<>c__DisplayClass22.b__1f()
at Microsoft.ExceptionHelper.ExceptionFilter.TryFilter[TEx](Action body, Predicate`1 filter, Action`1 catchClause)
at Microsoft.Crm.MapiStore.ServerStore.ExecuteInNewConnection(String entityTypeName, ConnectionActionsDelegate actions)
at Microsoft.Crm.MapiStore.ServerStore.ExecuteInNewConnectionAndTransaction(String entityTypeName, CommandActionsDelegate actions)
at Microsoft.Crm.MapiStore.DataCache.LoadFromDatabase(ICrmServerStore serverStore, CacheSources sources)
at Microsoft.Crm.MapiStore.ServerStore.GetEntityTypeDataInner(CacheSources sources, Boolean create, Boolean allowServerConnection, Boolean notifyGrid)
at Microsoft.Crm.MapiStore.ServerStore.GetEntityTypeData(CacheSources sources, Boolean create, Boolean allowServerConnection,
Boolean synchronousUpdateCheck, Boolean forceUpdate, Boolean notifyGrid)
at Microsoft.Crm.MapiStore.ContentsTable.GetDataCache(CacheSources sources, Boolean synchronousUpdateCheck)
at Microsoft.Crm.MapiStore.ContentsTable.EnsureData(Boolean synchronousUpdateCheck, Boolean notify)
at Microsoft.Crm.MapiStore.ContentsTable.QueryRows(Int32 RowCount, IMAPITable_QueryRows_Flags Flags)
at Microsoft.Interop.Mapi.MAPITableImpl.
QueryRowsManaged(MAPITableImpl* , Int32 lRowCount, UInt32 ulFlags, _SRowSet** ppRows)
Will report to Watson as Native Exception.

Situation was perplexing, we know CRM client uses SQL CE database for storing multiple information but not sure how it’s running into a sharing violation. The tool that comes in mind as first thought on file access issues is obviously ProcMon. So we set up ProcMon logging, ProcDump (to capture crash dump) and CRM client tracing on Client machine. Wait for few hours and we had these logs captured. Start of analysis was confirming Sharing Violation in CRM client trace and next eye was on ProcMon logs. After spending some time looking through ProcMon I could find solid evidences.

The sharing violation was for d32489ad-413d-e111-8e2e-005056971f28False.sdf file and this is visibly due to another process UserProfileManager.exe having this file open with Read ShareMode whereas Outlook.exe is trying to access the same file with desired Read/Write access. Even before the file is closed by UserProfileManager.exe! Below are the extracts and screenshots form ProcMon log. UserProfileManager.exe was visibly operating on user profile files throughout the ProcMon log.

             10:08:12.9654623 UserProfileManager.exe          1316    CreateFile        
            C:\Documents and Settings\\Local Settings\Application Data\Microsoft\MSCRM\
Client\d32489ad-413d-e111-8e2e-005056971f28False.sdf           
                SUCCESS        
Desired Access: Generic Read, Disposition: Open, Options: Sequential Access, Synchronous IO Non-Alert, 
                Non-Directory File, 
Open Reparse Point, Attributes: n/a, 
ShareMode: Read, AllocationSize: n/a, 
                Impersonating: S-1-5-21-257275866-466943691-1091232502-1211, 
OpenResult: Opened
            

10:08:13.2316028       OUTLOOK.EXE                      5444    CreateFile        
            C:\Documents and Settings\\Local Settings\Application Data
\Microsoft\MSCRM\Client\d32489ad-413d-e111-8e2e-005056971f28False.sdf            
                SHARING VIOLATION 
 Desired Access: Generic Read/Write, Disposition: Open, 
                Options: Synchronous IO Non-Alert, Non-Directory File, 
Random Access, Attributes: N, ShareMode: Read, Write, AllocationSize: n/a

            

With that being said, question was why this C:\Program Files\Citrix\User Profile Manager\UserProfileManager.exe is near constantly operation on files in User profile? User Profile Manager is a Citrix component so this question was out for Citirx Admins. After a bit of research on their side and what came out was a configuration possibility within User Profile Management called Active Write back. This feature is enabled by default and below is the link which describes the feature, how to disable it.

Citrix User Profile Management, to configure active write back - https://support.citrix.com/proddocs/topic/user-profile-manager-sou/upm-active-profile-write-back.html

Disabling Active Write back feature of Citrix User Profile Management completely eliminated Outlook crashes from the affected environment.

Using CRM 2011 client with Citrix XenApp is officially tested and supported by CRM product team. Here is the whitepaper describing the environment in which both were tested and test results. However, Citrix XenDesktop hasn’t gone through such compatibility test cycle.

Hope this was helpful.

Cheers,
Bhavesh Shastri
Support Escalation Engineer