$manager = Get-SPClaimProviderManager
$manager.ClaimProviders

DisplayName : System
 Description : Provides system claim data.
 IsEnabled : True
 IsUsedByDefault : True
 IsVisible : True
 AssemblyName : Microsoft.SharePoint, Version=15.0.0.0,
 Culture=neutral, PublicKeyToken=71e9bce111e9429c
 TypeName : Microsoft.SharePoint.Administration.Claims.SPSyst
 emClaimProvider
 ClaimProvider : Microsoft.SharePoint.Administration.Claims.SPSyst
 emClaimProvider
 ClaimProviderType : Microsoft.SharePoint.Administration.Claims.SPSyst
 emClaimProvider
 IsValid : True
 UpgradedPersistedProperties : {}

DisplayName : Active Directory
 Description : Active Directory claim data.
 IsEnabled : True
 IsUsedByDefault : False
 IsVisible : True
 AssemblyName : Microsoft.SharePoint, Version=15.0.0.0,
 Culture=neutral, PublicKeyToken=71e9bce111e9429c
 TypeName : Microsoft.SharePoint.Administration.Claims.SPActi
 veDirectoryClaimProvider
 ClaimProvider : Microsoft.SharePoint.Administration.Claims.SPActi
 veDirectoryClaimProvider
 ClaimProviderType : Microsoft.SharePoint.Administration.Claims.SPActi
 veDirectoryClaimProvider
 IsValid : True
 UpgradedPersistedProperties : {}

DisplayName : All Users
 Description : Provides identity provider claim data.
 IsEnabled : True
 IsUsedByDefault : True
 IsVisible : True
 AssemblyName : Microsoft.SharePoint, Version=15.0.0.0,
 Culture=neutral, PublicKeyToken=71e9bce111e9429c
 TypeName : Microsoft.SharePoint.Administration.Claims.SPAllU
 serClaimProvider
 ClaimProvider : Microsoft.SharePoint.Administration.Claims.SPAllU
 serClaimProvider
 ClaimProviderType : Microsoft.SharePoint.Administration.Claims.SPAllU
 serClaimProvider
 IsValid : True
 UpgradedPersistedProperties : {}

DisplayName : Forms Auth
 Description : Forms Based Authentication claim data.
 IsEnabled : True
 IsUsedByDefault : False
 IsVisible : True
 AssemblyName : Microsoft.SharePoint, Version=15.0.0.0,
 Culture=neutral, PublicKeyToken=71e9bce111e9429c
 TypeName : Microsoft.SharePoint.Administration.Claims.SPForm
 sClaimProvider
 ClaimProvider : Microsoft.SharePoint.Administration.Claims.SPForm
 sClaimProvider
 ClaimProviderType : Microsoft.SharePoint.Administration.Claims.SPForm
 sClaimProvider
 IsValid : True
 UpgradedPersistedProperties : {}

DisplayName : User Profile Claim Provider
 Description : User Profile Claim Provider
 IsEnabled : True
 IsUsedByDefault : True
 IsVisible : True
 AssemblyName : Microsoft.Office.Server.UserProfiles,
 Version=15.0.0.0, Culture=neutral,
 PublicKeyToken=71e9bce111e9429c
 TypeName : Microsoft.Office.Server.Security.UserProfileClaim
 Provider
 ClaimProvider : Microsoft.Office.Server.Security.UserProfileClaim
 Provider
 ClaimProviderType : Microsoft.Office.Server.Security.UserProfileClaim
 Provider
 IsValid : True
 UpgradedPersistedProperties : {}

DisplayName : Customer Ldap
 Description : Claims Provider for getting claims from
 REST-Service with CustomerLdap-Source.
 IsEnabled : True
 IsUsedByDefault : True
 IsVisible : True
 AssemblyName : Customer.ClaimsProvider.LDAP,
 Version=1.0.0.0, Culture=neutral,
 PublicKeyToken=9512c55e4a68ef80
 TypeName : Customer.ClaimsProvider.LDAP.CustomerProvider
 ClaimProvider :
 ClaimProviderType :
 IsValid :
 UpgradedPersistedProperties : {}

$manager.ClaimProviders.RemoveAt(5)

$manager.Update()

The search service application administration in the Central Administration displayed the message "Unable to retrieve topology component health states. This may be because the admin component is not up and running".

Then I started to check the ULS logs. There were a lot of critical messages in the ULS log regarding CSS addresses:

Content Plugin can not be initialized - list of CSS addresses is not set.

I assumed that this message was the result of the unfinished search provisioning and not the cause of the problem so I filtered the entries in the ULS log by the "Product" column containing the word "search". This gave me an overview of the search provisioning process and I found entries of the process hostcontrollerservice.exe which reappears every 20 - 30 seconds:

HostController: Starting node AdminComponent1 on system 8C5512 marked for automatic startup - threadId: 12

HostController: Restarting node AdminComponent1 on system 8C5512 marked for automatic startup - threadId: 18

HostController: Restarting node AdminComponent1 on system 8C5512 marked for automatic startup - threadId: 18

It looks like the SharePoint system was trying to start the AdminComponent1 process but the process stops after a couple of seconds.The AdminComponent is one of the processes that is started as a child process of the hostcontrollerservice.exe. So I checked the running tasks with the Sysinternals Process Explorer (in tree mode) and found the following:

As you can see there are no child processes beneath the hostcontrollerservice.exe process. In a working SharePoint 2013 search system the are normally five or six child processes.

So the question was why do I not have any running child process? I checked the command line of the child processes on a SharePoint system with working search with the Process Explorer. The noderunner process of the admin component was started there with this command line:

"C:\Program Files\Common Files\Microsoft Office Servers\15.0\Search\Runtime\1.0\NodeRunner.exe" --noderoot "C:\Program Files\Microsoft Office Servers\15.0\Data\Office Server\Applications\Search\Nodes\C5539A\AdminComponent1" --addfrom "C:\Program Files\Microsoft Office Servers\15.0\Data\Office Server\Applications\Search\Nodes\C5539A\AdminComponent1\Configuration\Local\Node.ini" --tracelog "C:\Program Files\Microsoft Office Servers\15.0\Data\Office Server\Applications\Search\Nodes\C5539A\AdminComponent1\Logs\NodeRunner.log"

As you can see the command line contains a parameter tracelog:

--tracelog "C:\Program Files\Microsoft Office Servers\15.0\Data\Office Server\Applications\Search\Nodes\C5539A\AdminComponent1\Logs\NodeRunner.log

I opened the path "C:\Program Files\Microsoft Office Servers\15.0\Data\Office Server\Applications\Search\Nodes" on the not working SharePoint system and found a six digit sub folder with another folder named AdminComponent1 and a subfolder named Logs. In the logs folder there was a file NodeRunner.log which contained a lot of exception messages:

Caught exception in node activator: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at Microsoft.Ceres.CoreServices.Node.NodeActivator.ActivateNode(IDictionary`2 configuration)
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters)
   at Microsoft.Ceres.CoreServices.NodeRunner.Runner.StartNodeController(EntryPoint entryPoint)
   at Microsoft.Ceres.CoreServices.NodeRunner.Runner.Start()

It looks like I have a memory problem which prevents the noderunner processes from starting correctly. This was very odd because the system had 32 GB RAM. I searched for the exception message and found some results pointing me to the solution. It is possible to limit the memory consumption of the noderunner processes with an entry in the file noderunner.exe.config in the folder "C:\Program Files\Common Files\Microsoft Office Servers\15.0\Search\Runtime\1.0". I found the following entry in my noderunner config file:

I changed the memoryLimitMegabytes value of the tag <nodeRunnerSettings memoryLimitMegabytes="100" /> to 0 (which means no restriction) and restarted the Search Host Controller Windows service. After a couple of seconds the hostcontrollerservice.exe in the Process Explorer showed the expected child processes:

To verify that the search service application was really working I went to the Central Administration and the search administration and it showed me the wanted status:

Problem solved!

Sorry, something went wrong

The User Profile Application requires domain credentials for connection access

I was not able to find the root cause of the problem but I was able to add the Windows group through a PowerShell script I found at Waldek Mastykarz website:

https://blog.mastykarz.nl/granting-permissions-user-profile-service-application-powershell

The script was original written for SharePoint 2010 but it also works with SharePoint 2013. It modified the original script a little bit and attached it to this post.

Grant-UserProfileConnectionPermission.zip

The problem

If a user adds a post to his MySite blog the Newsfeed of the user is updated with a link to the new blog post:

The URL to this blog post ("Test Eintrag") looks like this:

https://mysite.customer.com/my/de_bspi0102/Blog/Lists/Posts/ViewPost.aspx?ID=8

 If I click on the link my browser opens ViewPost.aspx. ViewPost.aspx starts a redirect to the following URL (traced with Internet Explorer Developer Tools):

 https://mysite.customer.com/my/de_bspi0102/Blog/Lists/Beitraege/Post.aspx?List=7bfeb6ee%2Deacf%2D4295%2Da542%2D605d7a21a8df&ID=8&Web=f1bbf158%2Dfb30%2D48f0%2D8e29%2D5862c92a5a07

That is the point where the 404 error occurs. The cause of the 404 is the name of the blog list. The real name of the blog list is "Posts" and not "Beitraege". It seems that the ViewPost.aspx is using the display language value of the MySite profile settings to use a localized name of the blog list, and that is wrong. If I change the link to the blog post from

https://mysite.customer.com/my/de_bspi0102/Blog/Lists/Posts/ViewPost.aspx?ID=8

to

https://mysite.customer.com/my/de_bspi0102/Blog/Lists/Posts/Post.aspx?ID=8

the URL is working without any problems. If I change the display language in my MySite profile back to English, create a new blog entry and click on the link, the error persists, the redirected URL still points to the list with the name "Beitraege".

The solution

The content type for the blog list has three configuration fields to set the URL for the display form, the edit form and the new form. The picture below shows the 3 fields of the content type in SharePoint Designer:

The users with the 404 problem have the URL of the display form set to "Lists/Beitrage/Post.aspx" despite the fact that the name of the list is "Posts" and not "Beitrage".  I changed the URL of the display form of the blog list content type to "Lists/Posts/Post.aspx" and the problem disappeared. I had a lot of users with that problem so I decided to write a PowerShell script that identifies the affected MySites and sets the display form of the content type to the correct url.

Surprisingly when I checked the content type from the PowerShell script it was always showing the correct URL but it was wrong in fact. It seems that the only way to see the wrong URL is by using SharePoint Designer.  So I decided to always set the URL to the correct value with the PowerShell script even if it looks like it is already correct. That fixed the problem for all affected users.

I attached the PowerShell script to this blog post

Change-ListContentTypeDisplayFormURL.zip

After some testing I found out that the MySite newsfeed only contained information that was created after the server reboot. All entries that were created before the crash disappeared.
My first idea was that there must be a problem with the microfeed list in the MySite of the users so I checked the list (MySite/Lists/PublishedFeed) on the MySites of some users and found all old entries are still there, they just do not show up in the newsfeed. So my second idea was to check the distributed cache service and the cache cluster. I executed the following PowerShell command on every SharePoint server that has the distributed cache service running:

Use-CacheCluster

Get-CacheHostConfig –ComputerName $env:Computername -CachePort 22233

Get-CacheHost

The cmdlets were executed without any errors, both cache server instances were up and running. I recreated the cache cluster just to make sure that it was possible to create a new cache cluster without any problems. That also succeeded.

The next idea was to look for a PowerShell cmdlet that re-caches the news feed from the micro feed list and I found the following two cmdlets:

Update-SPRepopulateMicroblogLMTCache

Update-SPRepopulateMicroblogFeedCache

I started Update-SPRepopulateMicroblogLMTCache with the id of the User Profile Service application proxy one time like this:

$upp = Get-SPServiceApplicationProxy | where {$_.TypeName -like "User Profile Service Application Proxy"}

$uppID = $upp.Id

# Force a cache time stamp refresh

Update-SPRepopulateMicroblogLMTCache -ProfileServiceApplicationProxy $uppID

After that I executed the second command with the id of the User Profile Service application proxy and my own account name (domain\username):

$upp = Get-SPServiceApplicationProxy | where {$_.TypeName -like "User Profile Service Application Proxy"}

$uppID = $upp.Id

Update-SPRepopulateMicroblogFeedCache -ProfileServiceApplicationProxy $uppID -AccountName $domainAndUserName

And voila: After a couple of seconds and a reload of my MySite page all my old newsfeed entries were back again!

I wrote a PowerShell script that iterates through all of the users with a MySite and executes the Update-SPRepopulateMicroblogFeedCache cmdlet with the account name of the user as parameter. After some minutes all news feeds were repopulated with the old news feed entries.

The cause for this problem was the ungraceful shutdown of all distributed cache instances. This crash killed the in memory cache with the news feed entries.

Repopulate-Newsfeed.zip

The PowerShell script will create a new incoming SMTP alias domain and then configure selected properties like max. message size, etc. It uses WMI to create the domain and configure the properties.

This should work on all Windows versions that support IIS 6 and PowerShell, but I have only tested it on Server 2008 R2.

Enjoy!