Microsoft.ApplicationServer.Caching.DataCacheException: ErrorCode<ERRCA0017>:SubStatus<ES0006>:There is a temporary failure. Please retry later. Microsoft.ApplicationServer.Caching.DataCacheException: ErrorCode<ERRCA0018>:SubStatus<ES0001>:The request timed out.
Microsoft.ApplicationServer.Caching.DataCacheException: ErrorCode<ERRCA0016>:SubStatus<ES0001>:The connection was terminated.

To resolve or alleviate these issues, you can try the following solutions/workarounds:

1.
ErrorCode<ERRCA0017>:SubStatus<ES0006>:There is a temporary failure. Please retry later.

• Install the latest cumulative update for AppFabric 1.1 (CU7 as of this writing)
• Upgrade .NET Framework to 4.5 on all cache servers
• Enable backgroundGC per the instructions in the CU3 article (see the CU3 article for the bug fixed and how to enable backgorundGC)

All CU links can be found in How to Update Windows Server AppFabric 1.1 with Cumulative Update Packages. CU7 and backgroundGC on the cache server can be very helpful to reduce the number of ErrorCode<ERRCA0017>:SubStatus<ES0006>. CU7 is not required to install on the cache clients.

2.
If you have local cache enabled, you need update client side caching assemblies to CU6 to fix the local cache bug (see the CU4 article for the details on the local cache bug.) It is not required to install CU7 on the cache servers to resolve the local cache bug.

3.
ErrorCode<ERRCA0018>:SubStatus<ES0001>:The request timed out.

3.1 
Increase maxConnectionsToServer. The default value is 1. The article Tune MaxConnectionsToServer suggests to set it to the number of cores (of the application server.) In our experience it may be too big. The value should not be greater than 4 in most of cases. Setting a big number can result in many open connections to the cache cluster nodes and high CPU on Lsass and the event logging service.

3.2
There is a known WCF 4.0 issue, which can affect the performance of your application if the cache client is a WCF application. You need either upgrade the .NET framework to 4.5 on the client or implement the workaround described in the article WCF service may scale up slowly under load.

3.3
For more realistic test results, do not use hot keys in load tests, which can lead to high CPU on one particular cache host. Instead use randomly generated keys (for example, Guid key = Guid.NewGuid() for example) for Get/Put operations in load tests. This is because AppFabric is a partition cache rather than a replication cache.

4.

One possibility of this exception is due to a timing issue as the cache client and cache server have the same 10 minute setting. We have seen cases where a client is getting a request on the connection just as the server is closing its side due to the timeout. If you have seen this error, we recommend you to set the client receiveTimeout less than the cache server.

The client side receiveTimeout setting (in milliseconds) can be found in Application Configuration Settings. This needs set in each client web.config or app.config.

The server side receiveTimeout (in milliseconds) can be set in the cluster config file:

<dataCache size="Small">
<!—Other Settings Here-->
   <advancedProperties>
       <transportProperties receiveTimeout=”600000” />
    </advancedProperties>
</dataCache>

The default value of DataCacheServiceAccountType is DataCacheServiceAccountType.SystemAccount, in which case AppFabric client assumes the caching service account is network service. It uses Kerberos and the host SPN (the service principal name) in the Kerberos TGS Request. This works when the caching service account is indeed Network Service because the host SPN is registered for the machine account (machinename$) by default in DC. Note that the Network Service account is machinename$. You will see “Authentication Package: Kerberos” in the security event log when the security auditing is enabled:

Log Name:      Security
Source:        Microsoft-Windows-Security-Auditing
Date:          2/9/2015 2:10:28 PM
Event ID:      4624
Task Category: Logon
Level:         Information
Keywords:      Audit Success
User:          N/A
Computer:     Server
Description:
An account was successfully logged on.

Subject:
Security ID:  NULL SID
Account Name:  -
Account Domain:  -
Logon ID:  0x0

Logon Type:   3

New Logon:
Security ID:  Domain\User
Account Name:  User
Account Domain:  Domain
Logon ID:  0x6a8fba
Logon GUID:  {116d7aa7-7569-a5b7-ea37-7074c066e4b9}

Process Information:
Process ID:  0x0
Process Name:  -

Network Information:
Workstation Name:
Source Network Address: -
Source Port:  -

Detailed Authentication Information:
Logon Process:  Kerberos
Authentication Package: Kerberos
Transited Services: -
Package Name (NTLM only): -
Key Length:  0

This event is generated when a logon session is created. It is generated on the computer that was accessed.

When the value of DataCacheServiceAccountType is set to DataCacheServiceAccountType.SystemAccount, AppFabric client assumes the caching service account is network service. It attempts to use Kerberos and the host SPN in the Kerberos TGS request. However this eventually fails if the caching service account is in fact a domain account rather than network service, because the host SPN is registered for the machine account but not for the domain account. You will see the error “The target principal name is incorrect” described in the blog. AppFabric does not fall back to use NTLM after encountering this error.

When the value of DataCacheServiceAccountType is set to DataCacheServiceAccountType.DomainAccount, AppFabric client assumes the caching service account is a domain account. It attempts to use Kerberos and creates a SPN identity “AppFabricCachingService/targetHostName:22233” in the Kerberos TGS request. This SPN is most likely never registered for the caching service account in DC. AppFabric encounters the error KDC_ERR_S_PRINCIPAL_UNKNOWN on the client and falls back to use NTLM.

Log Name:      Security
Source:        Microsoft-Windows-Security-Auditing
Date:          2/9/2015 2:17:43 PM
Event ID:      4624
Task Category: Logon
Level:         Information
Keywords:      Audit Success
User:          N/A
Computer:      Server
Description:
An account was successfully logged on.

Subject:
Security ID:  NULL SID
Account Name:  -
Account Domain:  -
Logon ID:  0x0

Logon Type:   3

New Logon:
Security ID:  Domain\User
Account Name:  Domain
Account Domain:  User
Logon ID:  0x719641
Logon GUID:  {00000000-0000-0000-0000-000000000000}

Process Information:
Process ID:  0x0
Process Name:  -

Network Information:
Workstation Name: ClientComputer
Source Network Address: -
Source Port:  -

Detailed Authentication Information:
Logon Process:  NtLmSsp
Authentication Package: NTLM
Transited Services: -
Package Name (NTLM only): NTLM V2
Key Length:  128

This event is generated when a logon session is created. It is generated on the computer that was accessed.

Enabling Kerberos event logging on the client, you will see the SPN is AppFabricCachingService/ServerFQDN:22233.

Log Name:      System
Source:        Microsoft-Windows-Security-Kerberos
Date:          2/9/2015 2:45:07 PM
Event ID:      3
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      ClientComputer
Description:
A Kerberos error message was received:
on logon session
Client Time:
Server Time: 19:45:7.0000 2/9/2015 Z
Error Code: 0x7  KDC_ERR_S_PRINCIPAL_UNKNOWN
Extended Error:
Client Realm:
Client Name:
Server Realm: REDMOND.CORP.MICROSOFT.COM
Server Name: AppFabricCachingService/ServerFQDN:22233
Target Name: AppFabricCachingService/ServerFQDN:22233@Domain
Error Text:
File: 9
Line: 12c4
Error Data is in record data.

Network trace:
0.0816158 4916 2:27:15 PM 2/9/2015 31.9676333 Unavailable 65.53.10.43 10.219.0.218 KerberosV5 KerberosV5:TGS Request  {TCP:763, IPv4:751}
0.0806043 4928 2:27:15 PM 2/9/2015 32.0482376 Unavailable 10.219.0.218 65.53.10.43 KerberosV5 KerberosV5:KRB_ERROR  - KDC_ERR_S_PRINCIPAL_UNKNOWN (7) {TCP:763, IPv4:751}

If you really wish to use Kerberos rather than NTLM when the caching service account is a domain account, you can set DataCacheServiceAccountType to DomainAccount, and register the SPN “AppFabricCachingService/targetHostName:22233” for the service account using setspn:

setspn -a AppFabricCachingService/targetHostName:22233 Domain\CachingServiceAccount

In summary, AppFabric always tries Kerberos first. It uses the host SPN when DataCacheServiceAccountType = SystemAccount or a custom SPN as “AppFabricCachingService/targetHostName:22233” when DataCacheServiceAccountType = DomainAccount. The result of success or failure and the protocol used depend on whether the SPN is registered or not and whether the caching service account is network service or a domain account.

Finally I want to ask a question: will it work if the value of DataCacheServiceAccountType is set to DomainAccount but the caching service account is network service? The answer to the question is it works and uses NTLM if the SPN “AppFabricCachingService/targetHostName:22233” is not registered. It is also because AppFabric falls back to NTLM in case of KDC_ERR_S_PRINCIPAL_UNKNOWN. This combination of DataCacheServiceAccountType and caching service account is confusing and is certainly not recommended. The only benefit I can think of is the same client setting works irrespective of the caching service account at the cost of using NTLM rather than Kerberos.

The Distributed Transaction Coordinator service on Local Computer started and then stopped. Some services stop automatically if they are not in use by other services.

If the value is 1, change it to 0 and try starting DTC again, which may resolve your problem. It is by design that DTC cannot run during sysprep. The default value of SysprepInProgress is 0, indicating the system is not in the middle of sysprep. How can this value become 1? One possibility is the machine was created with sysprep and a previous run failed.

The SysprepInProgress registry value is set during the MSDTC setup. Open the C:\windows\ DtcInstall.log and you could see the differences at the bottom of the log file between a good installation and a bad installation.

Good_DtcInstall.log

…
08-12-2013 15:07 : DTC Install error = 0, SysPrepDtcSpecialize : Enter, d:\w7rtm\com\complus\dtc\dtc\adme\deployment.cpp (1897)
08-12-2013 15:07 : DTC Install error = 0, SysprepDtcSpecialize: SysprepInProgress == 0. DTC has been already specialized., d:\w7rtm\com\complus\dtc\dtc\adme\deployment.cpp (1915)
08-12-2013 15:07 : DTC Install error = 0, SysPrepDtcSpecialize : Exit, d:\w7rtm\com\complus\dtc\dtc\adme\deployment.cpp (1959)

Bad_DtcInstall.log

…
12-20-2013 13:18 : DTC Install error = 0, SysPrepDtcSpecialize : Enter, d:\w7rtm\com\complus\dtc\dtc\adme\deployment.cpp (1897)
12-20-2013 13:18 : DTC Install error = 0, SysprepDtcSpecialize: SysprepInProgress == 0. DTC has been already specialized., d:\w7rtm\com\complus\dtc\dtc\adme\deployment.cpp (1915)
12-20-2013 13:18 : DTC Install error = 0, SysPrepDtcSpecialize : Exit, d:\w7rtm\com\complus\dtc\dtc\adme\deployment.cpp (1959)
01-14-2014 13:01 : DTC Install error = 0, SysPrepDtcCleanup : Enter, d:\w7rtm\com\complus\dtc\dtc\adme\deployment.cpp (1763)
01-14-2014 13:01 : DTC Install error = 0, SysPrepDtcCleanup : Exit, d:\w7rtm\com\complus\dtc\dtc\adme\deployment.cpp (1814)
01-14-2014 13:01 : DTC Install error = 0, SysPrepDtcGeneralize : Enter, d:\w7rtm\com\complus\dtc\dtc\adme\deployment.cpp (1820)
01-14-2014 13:01 : DTC Install error = 0, SysPrepDtcGeneralize : Exit, d:\w7rtm\com\complus\dtc\dtc\adme\deployment.cpp (1891)

SysPrepDtcSpecialize sets the value to 0 and SysPrepDtcGeneralize sets SysprepInProgress to 1

If SysprepInProgress is not your cause, you can consider the process documented in Disaster Recovery for MSDTC on Windows Server 2003 and 2008.

"A value of the wrong type was retrieved from the instance store. A value of type {/INSTANCE_1/Workflow/}myWFCloudService.xamlx was expected, but a value of type {/INSTANCE_2/Workflow/} myWFCloudService xamlx was encountered instead."

As it turns out, when you scale the cloud service to two or more instances, the site name in IIS is different for each site instance. They will look something like "<servicename>_IN_<instance number>. So with a servicename of "myWFCloudService" and 2 instances, the site names will be:

• MyWFCloudService_IN_0
• MyWFCloudService_IN_1

For workflow services, the Site name is used to calculate both the WorkflowHostType value and for calculating correlation keys used with Workflow instance persistence. For the above two instance nodes the values will be calculated differently so if we attempt to load our workflow instance on a different node than last time we will get the above error.

My first thought for a workaround was to configure Azure so that the site names would match. You can accomplish this manually using remote desktop with Azure and renaming the site in the IIS manager, however, the moment one of these sites is moved or reloaded you’ll find that the site name reverts back to the original format. To make this site name change “permanent” in Azure you’ll have to use a Web Role and the RoleEntryPoint.OnStart() function. I was able to test that this approach did indeed work, but then I discovered (by way of our Workflow Product team!) an undocumented parameter workflowHostingOptions whose sole purpose is to allow you to override the Site name dependency. You’ll need to add the following to your Workflow service config file:

<configuration>
    <configSections>
        <sectionGroup name="system.serviceModel.activities" type="System.ServiceModel.Activities.Configuration.ServiceModelActivitiesSectionGroup, System.ServiceModel.Activities, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35">
            <section name="workflowHostingOptions" type="System.ServiceModel.Activities.Configuration.WorkflowHostingOptionsSection, System.ServiceModel.Activities, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"/>
        </sectionGroup>
    </configSections>

    <system.serviceModel.activities>
        <workflowHostingOptions overrideSiteName="true"/>
    </system.serviceModel.activities>

…. 
    <system.serviceModel>
….
    </system.serviceModel>
….

</configuration>

I found that the order these are listed in the config file does matter. The section group has to come before the “WorkflowHostingOptions” and the “WorkflowHostingOptions” has to come before the <system.serviceModel> section.

There is one other detail that needs mentioning. During testing it was found that even with matching site names the Workflow service sometimes failed with "The requested resource has moved to one of the following locations" exception. The type of this exception is a RedirectionException. This is the result of the instance we are trying to load being locked in the instance store.

This can be remedied by modifying the configuration file of the workflow service to have a "workflowIdle" behavior with values of timeToPersist and timeToUnload both set to "0:0:0". This way, as soon as an instance goes idle it will be unloaded. This is what that behavior looks like in the web.config file: 

<workflowIdle timeToUnload="0:0:0" timeToPersist="0:0:0"/>

If the timeToUnload value is greater than 0:0:0, it is possible for a subsequent request to be received on one of the nodes while the instance is still loaded/locked by the other node. This scenario will result in the RedirectionException.

Once you have configured the workflowHostingOptions and the workflowIdle behavior your Workflow services should scale out in Azure without any issues.

To find out more information around the installation error first check the installation logs.
The Setup log is found at C:\ProgramData\Microsoft\AppFabric.
The installation logs are found in c:\users\<logon>\appdata\local\temp.
The name of the AppFabric installation log file will be of the form AppServerSetup(yyyy-MM-ddH-mm-ss).log.
There may also be logs named AppServerSetup1_1 or AppServerSet1_1_CustomActions.
Other logs may be located at C:\ProgramData\Microsoft\Windows\WER\ReportArchive.

AppFabric Stand-Alone Installation

Please note that Sharepoint has its own particular way to
install AppFabric and thus it has its own set of installation issues. The chart
below is specific to Sharepoint so it is not recommended to use these resolutions
for a standalone AppFabric installation.

AppFabric Sharepoint Installation

Product: AppFabric 1.1 for Windows Server -- Error 1722. There is a problem with this Windows Installer package. A program run as part of the setup did not finish as expected. Contact your support personnel or package vendor.  Action Env_PSModulePath_powershell_i, location: c:\Program Files\AppFabric 1.1 for Windows Server\Microsoft.ApplicationServer.InstallHelper.exe, command: powershell.exe "-command \"$str = [System.Environment]::GetEnvironmentVariable( \\\"PSModulePath\\\ ", [System.EnvironmentVariableTarget]::Machine); $str = $str+\\\";c:\Program Files\AppFabric 1.1 for Windows Server\PowershellModules\\\"; c:\Windows\system32\setx.exe /M PSModulePath \\\"$str\\\"\ "" "C:\Users\Administrator\AppData\Local\Temp\AppServerSetup1_1_CustomActions(2014-5-15 11-08-45).log"

CAUSE
The PSModulePath environment variable is corrupted. There is a quotation mark at the end.

C:\Windows\system32\WindowsPowerShell\v1.0\Modules\;C:\Program Files (x86)\Microsoft SQL Server\110\Tools\PowerShell\Modules;c:\Program Files\AppFabric 1.1 for Windows Server\PowershellModules"

RESOLUTION
Delete the ending quotation mark and re-run the setup.

If you are using a third party firewall, you must manually configure the firewall for the caching features to work properly. What additional protocols or ports do you need to open besides cache/replication/arbitration/cluster TCP port? The additional protocols used by the caching features are SMB, SMB2 and ICMP. The AppFabric cache uses SMB and SMB2 to read the cluster config file on a share when the XML store provider is being used and to access the remote registry service regardless of the provider type. The AppFabric cache also uses the ICMP protocol. So you need to open a firewall for SMB, SMB2 and ICMP protocols or for port 445 and the ICMP protocol in addition to those cache TCP ports (22233, 22234, 22235 and 22236 by default.)

Cumulative update package 1 for Microsoft AppFabric 1.1 for Windows Server
Cumulative update package 2 for Microsoft AppFabric 1.1 for Windows Server
Cumulative update package 3 for Microsoft AppFabric 1.1 for Windows Server     
Cumulative update package 4 for Microsoft AppFabric 1.1 for Windows Server
Cumulative update package 5 for Microsoft AppFabric 1.1 for Windows Server
Cumulative update package 6 for Microsoft AppFabric 1.1 for Windows Server
Cumulative update package 7 for Microsoft AppFabric 1.1 for Windows Server

These are the AppFabric 1.1 cache dll versions Microsoft has released:

RTM: 1.0.4632   (29-Nov-2011)
CU1: 1.0.4639    (20-Feb-2012)
CU2: 1.0.4644    (29-May-12)
CU3: 1.0.4652.2 (27-Nov-2012)       
CU4: 1.0.4653.2 (28-Mar-2013)
CU5: 1.0.4655.2 (03-Mar-2014)
CU6: 1.0.4656.2 (27-Feb-2015)
CU7: 1.0.4657.2 (01-Sept-2015)

CU3 and later versions including CU7 require .NET Framework 4.5 on cache servers as documented on the KBs. However .NET 4.5 is not a prerequisite on cache clients to use client binaries in CU3, CU4, CU5, CU6 and CU7.

You can search for AppFabric cumulative update packages at https://support.microsoft.com for the phrase (including the quotes): 

"cumulative update package" appfabric

The following query does this search for you. Choose the latest one:

https://search.microsoft.com/en-us/supportresults.aspx?form=mssupport&q=%22cumulative%20update%20package%22%20appfabric

System.UnauthorizedAccessException: Access to the registry key 'Global' is denied.
   at Microsoft.Win32.RegistryKey.Win32Error(Int32 errorCode, String str)
   at Microsoft.Win32.RegistryKey.InternalGetValue(String name, Object defaultValue, Boolean doNotExpand, Boolean checkSecurity)
   at Microsoft.Win32.RegistryKey.GetValue(String name)
   at System.Diagnostics.PerformanceMonitor.GetData(String item)
   at System.Diagnostics.PerformanceCounterLib.GetPerformanceData(String item)
   at System.Diagnostics.PerformanceCounterLib.get_CategoryTable()
   at System.Diagnostics.PerformanceCounterLib.CategoryExists(String machine, String category)
   at System.Diagnostics.PerformanceCounterCategory.Exists(String categoryName, String machineName)
   at Microsoft.ApplicationServer.Caching.ClientPerformanceCounters.InitializeClientPerformanceCounters()
   at Microsoft.ApplicationServer.Caching.ClientPerformanceCounters.IsPerfCountersEnabled()
   at Microsoft.ApplicationServer.Caching.DataCacheFactory..ctor(DataCacheFactoryConfiguration configuration)

It is a registry permission issue. The check for performance monitor counters (ClientPerformanceCounters.IsPerfCountersEnabled) was first introduced in the CU1 release. You won't see the issue in AppFabric 1.0 and 1.1 RTM.

Workaround:

• Add the cache client application account to the following groups: Performance Monitor Users group and Performance Log Users group, and
• Restart the cache client application

As of today, these are the AppFabric 1.1 cache client versions Microsoft has released:

RTM: 1.0.4632 (29-Nov-2011)
CU1: 1.0.4639 (20-Feb-2012)
CU2: 1.0.4644 (29-May-12)
CU3: 1.0.4652.2 (27-Nov-2012)
CU4: 1.0.4653.2 (28-Mar-2013)

CU=cumulative update

        IGlobalOptions* pGlobalOptions = NULL;
        HRESULT hr = CoCreateInstance(CLSID_GlobalOptions, NULL, CLSCTX_INPROC_SERVER, IID_PPV_ARGS(&pGlobalOptions));
        if (SUCCEEDED(hr))
        {
            pGlobalOptions->Set(COMGLB_RO_SETTINGS, COMGLB_FAST_RUNDOWN);
        }

        pGlobalOptions->Release();
        pGlobalOptions = NULL;

A rundown is the process by with the RpcSs (RPC Server Service) calls release on COM stubs within a COM server application to release resources maintained by that COM stub.  See this documentation for more information on COM stubs.  By setting the global flag COMGLB_FAST_RUNDOWN all of the COM stubs in this COM server will be subject to the fast rundown behavior which means that stubs are run down on termination of the client process.  This can be useful particularly in a case where a COM client might terminate abnormally and not call release on any COM proxies that it still references, e.g. a case where a COM client has crashed due to an unhandled exception.

It should be noted that the fast rundown path only works if the COM client and server are on the same machine.  For remote COM stubs this will not work and rundown will still be governed by the normal DCOM garbage collection process.  Just as I did, you might wonder why this won’t work for remote COM servers.  My best guess is this.  DCOM Garbage Collection is performed through a pinging process where the client machine’s RpcSs service will ping the remote server’s RpcSs service to let it know the client is still alive. In the remote case the remote RpcSs service (running on the COM server machine) is responsible for running down stubs in the remote COM server, but the remote RpcSs service has no idea of when the client will exit since the client executable is running on a different machine.

What if you are not able to set this global flag, i.e. your have a COM DLL that is acts as a plug-in to another application?  In this case you are not the owner of the COM server code and can’t set this global flag during process initialization.  Even if you could set this global flag I don’t think the application owner would want you setting a global flag that would impact other plug-ins.  In this case you can opt-in to the fast rundown behavior for just your COM objects by implementing the IFastRundown marker interface.