====================================

First make sure that your Cache Cluster is configured following best practices, you can use the script:

https://blogs.technet.com/b/filipbosmans/archive/2015/01/04/troubleshooting-distributed-cache-for-sharepoint-2013-on-premise.aspx

Reduce complexity of your setup:

=====================

Revert to 1 Cache Host if possible.

If you have intermittent issues, or the Cluster is not down for instance, reduce Cache Hosts to maximum 2.

If you can trigger the issue, you can go all the way with increasing log levels and so on.

If you can't, you need to take into account that logs will become massive over time, so finetuning is recommended. 

There are too many scenarios to list here that might need different log sets, so adjust accordingly, I added things that you might not need at all or you can combine both.

Able to trigger the issue:

================

• Set up Performance Counters for AppFabric, take a baseline set from morning till evening.
• Take another set from the day you trigger.
• Increase Tracing to VerboseEx and Events to Verbose through PowerShell.

Set-SPLogLevel -TraceSeverity VerboseEx -EventSeverity Verbose

• Check if AppFabric logs are enabled:

“Event Viewer" - "Applications and Services Logs" - "Microsoft" - "Windows" - "Application Server–System Services" - "Microsoft-Windows-Application Server–System Services”:

Admin, Analytic, Operational, Debug. 

To get Debug and Analytic logs, choose “Microsoft-Windows-Application Server–System Services”, and from the top menu, chose "Show" - "Analytic and Debug logs".

Once all the 4 logs are showing, you can disable them, increase the size from default 1 MB to perhaps 5 or 10 through Right click - "Properties" and "General" tab, and then enable them again.

• Setup Network Monitor on all servers you think are involved.

For instance with a communication issue, run it on all Cache Hosts and perhaps even your SQL.

• Enable logman for Appfabric:

logman create trace CacheETWTracing -p "Microsoft-Windows Server AppFabric Caching" -o c:\temp\%computername%sAppFabricCacheTrace.etl -ow -f bin

logman start CacheETWTracing

Reproduce issue

logman stop CacheETWTracing

• If you can trigger the issue with accessing a site for instance, enable Developer Dashboard if allowed, if not, take a Fiddler or enable IE12 instead and lookup the CorrelationID.
• Reproduce.
• Take limited merged ULS logs, make sure to include at least a couple of minutes before repro.

Merge-SPLogFile -Path "C:\FarmMergedLog.log" -Overwrite -StartTime (Get-Date).AddMinutes(-3)

AddMinutes(-3) will take logs from the moment you run the Merge command and goes back until 3 minutes before.

• Take the Application and AppFabric Events, filter on "Last Hour", make sure to analyze them on a SharePoint Server test machine.
• Take the logman trace and use Message Analyzer on a SharePoint Server test machine (otherwise it wont find the libraries) or save as .csv file.

Not able to trigger the issue:

===================

• Increase tracing to Verbose(Ex) and Events to Verbose only for "SharePoint Server" - "Distributed Cache". (if you expect logs related to the issue will surface there)

Set-SPLogLevel -TraceSeverity Verbose -Identity "Distributed Cache" -EventSeverity Verbose

• Check if AppFabric logs are enabled.
• Set up Performance Counters for AppFabric, take a baseline for a prolonged period based on when the issue occurs usually.
• After the baseline counters, start fresh batch of performance counters to compare to the set from when the issue occurred.
• If you encounter the error/issue, look in ULS and Events and take a merged log with filtered Events from that time. (If possible only from relevant servers)

Merge-SPLogFile -Path "C:\FarmMergedLog.log" -Overwrite -StartTime "06/09/2008 16:00" -EndTime "06/09/2008 16:15"

If you have too many servers, you can possibly set up a Subscription on 1 server to collect all the Event logs from all your servers there.

Event Subscriptions

https://technet.microsoft.com/en-us/library/cc749183.aspx

Documentation and downloads:

=====================

Network Monitor

https://www.microsoft.com/en-us/download/details.aspx?id=4865

Message Analyzer

https://www.microsoft.com/en-us/download/details.aspx?id=44226

ULS Viewer

https://www.microsoft.com/en-us/download/details.aspx?id=44020

AppFabric Performance Counters

https://msdn.microsoft.com/en-us/library/ff637725(v=azure.10).aspx

Health Monitoring AppFabric

https://msdn.microsoft.com/en-us/library/ff921010(v=azure.10).aspx

Troubleshoot AppFabric

https://msdn.microsoft.com/en-us/library/ee790821.aspx

Keep in mind not all the actions listed here are allowed in a SharePoint setup.

Logging and Counters in App Fabric Cache

https://blogs.msdn.com/b/appfabriccat/archive/2010/12/14/logging-in-app-fabric-cache.aspx

Logging and Counters in App Fabric Cache

https://blogs.msdn.com/b/appfabriccat/archive/2010/12/14/logging-in-app-fabric-cache.aspx

Logman

https://technet.microsoft.com/en-us/library/bb490956.aspx

• Create a folder @ C:\Distributed Cache [ServerName]
• Create files which contain things like local event logs for DC, DC commands, exported configuration XML for the Cache Cluster etc.

• Might sound strange, but don't disable/remove the Distributed Cache Cluster on your Farm even if it for instance performs better without, solve the issue instead.
• Before disconnecting a server from the farm, make sure that it is not part of the AppFabric Cluster, as it will break it.
• Install the latest AppFabric CU and make sure the Garbage Collection section is added to the config file.

Latest at time of writing (CU7):
https://support.microsoft.com/en-us/kb/3092423
GC:
https://blogs.msdn.com/b/calvarro/archive/2014/03/20/points-to-consider-with-distributed-cache-on-sharepoint-2013.aspx

• There are articles out there that tell you it is ok to perform unsupported procedures.

If you want to use AppFabric for custom caches, don't use the SharePoint Cache Cluster, build a separate one for that.
Don't run AppFabric Tools for managing your AppFabric Cluster or Hosts, only use Distributed Cache PS commands.
The AppFabric "Windows" service should not be touched at all, meaning, don't change the account manually, nor start/stop the service or change the password for the account. Only exception would be that someone changed the account manually before, and you only want to revert back to the one AppFabric is aware of, but even then it might be better to simply remove and add the server as a Cache Host instead.
As with almost everything on DB level in SharePoint, don't touch the Cluster Details in the config DB.
The AppFabric Export-CacheClusterConfig functionality should not be used to manipulate the Cluster nor Hosts, unless verified by MS Support, there are "very" few reasons to do so anyway, as most can be done through other ways.

• Change the values for the caches from day 1 as recommended, if need be increase even more.

https://technet.microsoft.com/en-us/library/jj219613.aspx#finetune

• Run AppFabric on WFE's (unless they are struggling to keep up), not Application servers, if possible use dedicated Cache Hosts.
• Don't run AppFabric on servers that have services running that will compete for HW resources.

Can contradict with the previous statement, but if you are in a situation where WFE's are overloaded and you have a limited amount of App servers, I would really consider adding dedicated servers.
https://technet.microsoft.com/en-us/library/jj219572.aspx

• Increase memory allocation sufficiently based on MS recommendation or even more if you see the Caches are overloaded.

https://technet.microsoft.com/en-us/library/jj219613.aspx#memory

• Don't go overboard with the number of Cache Hosts, having 10 WFE's does not mean you need 10 Cache Hosts, 2-3 will be enough in most cases.
• Use the script on Technet for a Graceful shutdown of a Host, don't forget to run Use-CacheCluster before ;)

https://technet.microsoft.com/en-us/library/jj219613.aspx#graceful

• After doing maintenance to Cache Hosts, check that all are healthy and match memory size and service acccount.

Use-CacheCluster
Get-CacheHost
Export-CacheClusterConfig -Path C:\export.txt

• When virtualizing your SharePoint servers make sure you are not using Virtual Memory.
• When you ran into an issue and the Cluster is in an unhealthy state from which you simply can't recover due to for instance unsupported actions, rebuild your Farm instead of trying to fix it manually.

https://technet.microsoft.com/en-us/library/jj219613.aspx
The Distributed Cache service can end up in a nonfunctioning or unrecoverable state if you do not follow the procedures that are listed in this article.
In extreme scenarios, you might have to rebuild the server farm.
The Distributed Cache depends on Windows Server AppFabric as a prerequisite.
Do not administer the AppFabric Caching Service from the Services window in Administrative Tools in Control Panel.
Do not use the applications in the folder named AppFabric for Windows Server on the Start menu.

• Solving an issue with AppFabric usually comes down to going over your list of recommendations, known issues and misconfiguration, if the issue persists after that, well ... Houston.
• I'd highly recommend to make sure your Cluster is configured best practice before starting to troubleshoot, it can save you a lot of pain and effort.

Related articles:

https://blogs.technet.com/b/filipbosmans/archive/2015/01/04/troubleshooting-distributed-cache-for-sharepoint-2013-on-premise.aspx

https://blogs.technet.com/b/filipbosmans/archive/2015/09/07/how-to-check-for-issues-with-distributed-cache-and-the-script.aspx

We will go through the logs you get via the script:

https://blogs.technet.com/b/filipbosmans/archive/2015/01/04/troubleshooting-distributed-cache-for-sharepoint-2013-on-premise.aspx

Recommendations post:

https://blogs.technet.com/b/filipbosmans/archive/2015/10/17/sharepoint-distributed-cache-recommendations.aspx

AppFabric.txt

=======

==================
AppFabric Version
==================

This will tell you which CU for AppFabric is installed.

For instance:

CU 3 and onwards, you need to add a section in the config file to let Garbage Collection work correctly, you can check in the "AppFabric Configuration File" in the AppFabric.txt

Look for:

<appSettings>
<add key="backgroundGC" value="true"/>
</appSettings>

Add Garbage Collection Setting

Recommendation is to go to at least to CU 6 at the time of writing.

==================
AppFabric Service
==================

name : AppFabricCachingService
startname : CONTOSO\spfarm
startmode : Auto
state : Running

Compare the account name with the output from the hosts section.

Sometimes someone changed it manually (not supported) in the service itself, this will not be picked up by SharePoint nor AppFabric.

==================
Registry
==================

AdminConfigured : 1
ServiceConfigured : 1
ConnectionString : Data Source=spalias;Initial Catalog=SharePoint_Config;Integrated Security=True;Enlist=False
Provider : SPDistributedCacheClusterProvider
PSPath : Microsoft.PowerShell.Core\Registry::HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\AppFabric\V1.0\Configurat
ion
PSParentPath : Microsoft.PowerShell.Core\Registry::HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\AppFabric\V1.0
PSChildName : Configuration
PSDrive : HKLM
PSProvider : Microsoft.PowerShell.Core\Registry

If the connection string is empty, the system has no way determining the Cache Cluster, in our case (SharePoint) stored in the Configuration Database.

In this example I used a SQL alias during configuration, check if needed if it is referencing correctly.

Export.txt

=======

</cache>
<cache consistency="StrongConsistency" name="DistributedLogonTokenCache_49ef67e4-a8c6-44e4-b276-d51c9eb01b93"
minSecondaries="0">
<policy>
<eviction type="Lru" />
<expiration defaultTTL="10" isExpirable="true" />
</policy>
</cache>

Rarely happens, but for all the caches, it needs to have the FarmID at the end of the name like 49ef67e4-a8c6-44e4-b276-d51c9eb01b93

If it is not there (except for "default cache" that "can" be mentioned as number one), that means somebody added a custom cache which is not supported.

<hosts>
<host replicationPort="22236" arbitrationPort="22235" clusterPort="22234"
hostId="203503434" size="614" leadHost="true" account="CONTOSO\spfarm"
cacheHostName="AppFabricCachingService" name="SPCA.contoso.com"
cachePort="22233" />
<host replicationPort="22236" arbitrationPort="22235" clusterPort="22234"
hostId="390812545" size="614" leadHost="true" account="CONTOSO\spfarm"
cacheHostName="AppFabricCachingService" name="SPWFE.contoso.com"
cachePort="22233" />
</hosts>

This is important, size, ports and account need to be the same on all Cache Hosts, when you add/remove a host, the size and account will be back to the default, always check.

Change memory

Change account

Verify that all the host names are FQDN, only seen this once long time ago but just to be sure ...

Size needs to be sufficient for running the service, if it is too low it can lead to eviction of cached items and even crashes of the service.

GetInfo.txt

========

==================
Get-CacheClusterHealth
==================

HostName = SPCA.contoso.com
-------------------------

NamedCache = DistributedViewStateCache_49ef67e4-a8c6-44e4-b276-d51c9eb01b93
Healthy = 4.55
UnderReconfiguration = 0.00
NotPrimary = 0.00
InadequateSecondaries = 0.00
Throttled = 0.00

Unallocated named cache fractions
---------------------------------

NamedCache = default
Unallocated fraction = 0.07

NamedCache = DistributedActivityFeedCache_49ef67e4-a8c6-44e4-b276-d51c9eb01b93
Unallocated fraction = 0.14

NamedCache = DistributedDefaultCache_49ef67e4-a8c6-44e4-b276-d51c9eb01b93
Unallocated fraction = 0.07

If you see anything mentioned under "Unallocated" that means the caches are still building up, for instance after adding a Cache Host, nothing to worry, just be sure it is empty after like 15 minutes.

The "split" between Cache Hosts should always reflect how many Hosts there are, 3 hosts will divide content in 3 so every host will have a value of around 3, 2 will have around 4.5.

"Throttled" is self explanatory.

==================
Get-CacheHost
==================

HostName    : SPCA.contoso.com
PortNo      : 22233
ServiceName : AppFabricCachingService
Status      : Up
VersionInfo : 3[3,3][1,3]

HostName    : SPWFE.contoso.com
PortNo      : 22233
ServiceName : AppFabricCachingService
Status      : Up
VersionInfo : 3[3,3][1,3]

This is AppFabric side.

==================
Get-SPServiceInstance
==================

TypeName : Distributed Cache
Status   : Online
Id       : 8dde3fe4-b745-445c-ae1d-7f8a5497fe64
Server   : SPServer Name=SPCA

TypeName : Distributed Cache
Status   : Online
Id       : 04bf5462-3089-44ff-9760-90b7945b3980
Server   : SPServer Name=SPWFE

This is SharePoint side.

Usually a mismatch is the culprit when it comes to crashes of the AppFabric service for instance, AppFabric and SharePoint have different information regarding which SharePoint Servers are supposed to run as a Cache Host.

You can use Sam's blog to fix this:

https://blogs.msdn.com/b/sambetts/archive/2014/03/19/sharepoint-2013-distributed-cache-appfabric-troubleshooting.aspx

==================
Cache Settings
==================

DistributedLogonTokenCache

ChannelInitializationTimeout : 60000
ConnectionBufferSize         : 131072
MaxBufferPoolSize            : 268435456
MaxBufferSize                : 8388608
MaxOutputDelay               : 2
ReceiveTimeout               : 60000
ChannelOpenTimeOut           : 20
RequestTimeout               : 20
MaxConnectionsToServer       : 4

Check if these settings are increased/decreased correctly, keep in mind that these are recommendations and you can change them even more, but this is usually not necessary.

Recommendation from below article is 3000/3000/1

Change values

==================
NetShell
==================

MIB-II TCP Connection Entry
Local Address Local Port Remote Address Remote Port State
-----------------------------------------------------------------------------
192.168.0.51 1077 192.168.0.50 22234 Established
192.168.0.51 1098 192.168.0.50 22234 Established
192.168.0.51 3047 192.168.0.50 22233 Established
0.0.0.0 22233 0.0.0.0 0 Listen
0.0.0.0 22234 0.0.0.0 0 Listen
192.168.0.51 22234 192.168.0.50 21969 Established
192.168.0.51 22234 192.168.0.50 21996 Established
0.0.0.0 22236 0.0.0.0 0 Listen

As you can imagine, check if 22233/22234/22236 ports are established or listening. (22235 should not show up)

MIB-II TCP Statistics
------------------------------------------------------
Timeout Algorithm: Van Jacobson's Algorithm
Minimum Timeout: 10
Maximum Timeout: 4294967295
Maximum Connections: Dynamic
Active Opens: 1840
Passive Opens: 291
Attempts Failed: 0
Established Resets: 648
Currently Established: 27
In Segments: 2042415
Out Segments: 1990285
Retransmitted Segments: 14201
In Errors: 0
Out Resets: 530

A huge number of connections might cause issues.

The netstat part I added so you can check which servers (by name) are communicating.

==================
Memory and Manufacturer
==================

12287.55078125 MB Total Physical Memory
8733.18359375 MB Total Free Memory
Microsoft Corporation System Manufacturer
VMWare BIOS Manufacturer

Did not find a sure way to always be able to tell if the Server is virtualized or not, if in doubt ask your Server Administrator, have them check if the SharePoint servers are using a Hyper-Visor and more specifically if Virtual Memory is configured, which is not supported.

Microsoft Corporation indicates Hyper-V.

VMWare BIOS indicates ...

Still see customers using virtual memory, most likely due to non-SharePoint admins managing these environments.

Check if your free memory is getting low.

==================
Local running Services
==================

TypeName
--------
Microsoft SharePoint Foundation Workflow Timer Service
Microsoft SharePoint Foundation Web Application
Central Administration
Microsoft SharePoint Foundation Incoming E-Mail
Distributed Cache

It is not recommended to run Distributed Cache on SharePoint servers that have:

SQL Server 2008 or SQL Server 2012
Search service
Excel Services in SharePoint
Project Server services

Something that is not mentioned in the article is that the User Profile Sync Service "can" also compete with DC.

Did not add a check for SQL, simply open Start and check if Management Studio is listed, if so, you can check if the /DATA location in the File System contains databases if you don't have access to SQL.

SQL DB location

Check "Capacity planning for the Distributed Cache service" section if you're using the recommended memory sizes based on Farm size.

Memory Allocation

==================
Accounts + Local Groups
==================

WSS_ADMIN_WPG
WSS_WPG

Alias name WSS_WPG
Comment Members of this group have read access to system resources used by Microsoft SharePoint Foundation.

Members

-------------------------------------------------------------------------------
CONTOSO\spfarm
CONTOSO\sppool
NT AUTHORITY\LOCAL SERVICE
NT AUTHORITY\SYSTEM
The command completed successfully.

Alias name WSS_ADMIN_WPG
Comment Members of this group have write access to system resources used by Microsoft SharePoint Foundation.

Members

-------------------------------------------------------------------------------
Administrators
CONTOSO\spfarm
CONTOSO\spinstaller
CONTOSO\test1
The command completed successfully.

Alias name Administrators
Comment Administrators have complete and unrestricted access to the computer/domain

Members

-------------------------------------------------------------------------------
Administrator
CONTOSO\Domain Admins
CONTOSO\spfarm
CONTOSO\spinstaller
The command completed successfully.

==================
Memory used by AppFabric Process
==================

Handles NPM(K) PM(K) WS(K) VM(M) CPU(s) Id ProcessName
------- ------ ----- ----- ----- ------ -- -----------
870 682 1421696 636972 1766 5,694.95 1364 DistributedCacheService

Did not add code to check inside nested Groups, so if you see those open them manually to check the Members.

If you have issues with for instance Social features, check if the User Profile Account has access to Distributed Cache through any of the groups.

==================
FireWall Rules
==================

Name : FPS-ICMP4-ERQ-In
DisplayName : File and Printer Sharing (Echo Request - ICMPv4-In)
Description : Echo Request messages are sent as ping requests to other nodes.
DisplayGroup : File and Printer Sharing
Group : @FirewallAPI.dll,-28502
Enabled : True
Profile : Any
Platform : {}
Direction : Inbound
Action : Allow
EdgeTraversalPolicy : Block
LooseSourceMapping : False
LocalOnlyMapping : False
Owner :
PrimaryStatus : OK
Status : The rule was parsed successfully from the store. (65536)
EnforcementStatus : NotApplicable
PolicyStoreSource : PersistentStore
PolicyStoreSourceType : Local

Had some issues with pulling specific FireWall Rules, so there might be more than needed.

Check if they are enabled, they are mandatory, keep in mind that they can be disabled due to other Firewalls managing ports, hence if you see they are not present or disabled, a simple ping from all hosts in the cluster to the rest of the cluster will tell you if ICMP is allowed.

For Remote Management check these specific ports via Telnet/PSPing or ask your Network team.

ICMP

Remote management

Might add Test-Connection to ping all Cache Hosts from all Cache Hosts.

Some things you are not supposed to change manually unless told to do so by MS Support, if in doubt, open a support case.

More to follow if needed.