SharePoint Administration

SharePoint 2013 Search Topology Activation Error: "Unable to retrieve topology component health states. This may be because of the admin component is not up and running"

BK_R — Mon, 09 Jun 2014 00:47:00 GMT

Scenario

In a newly deployed farm, creating new search service application (SSA) via PowerShell or GUI of SharePoint 2013.
The SSA gets created, however, topology in SSA will show

"Unable to retrieve topology component health states. This may be because of the admin component is not up and running."

The trial to create and activate topology fails immediately and sends an event ID to application event as follows

Event ID: 6482

Description:
Application Server Administration job failed for service instance Microsoft.Office.Server.Search.Administration.SearchServiceInstance
Reason: A call to SSPI failed, see inner exception.
Technical Support Details:
System.ServiceModel.Security.SecurityNegotiationException: A call to SSPI failed, see inner exception. ---> System.Security.Authentication.AuthenticationException: A call to SSPI failed, see inner exception. ---> System.ComponentModel.Win32Exception: No authority could be contacted for authentication.

Possible Reasons:

1. The timer service account is trying to communicate with HostController service and generates 'SSPI Connect' call demanding SPN to be created for target services Identity, if no SPN is found.

2. The Hostcontroller service's NodeRunner process is limited to use restricted amount of RAM

Possible Solutions:

Check which account is running the Timer Service account.
Check which account is running HostController Service (From Services.msc or Central Administration > Security > manage Service Accounts)

Use SetSPN -L to list active SPNs for these two accounts
We have to set a dummy SPN for Timer Service and HostController service account.
Set an SPN SP/SPTEST -- this is a dummy SPN for Timer Service account.
Set an SPN SP/SPTEST1 -- this is a dummy SPN, for service account of Host Controller Service also.
Do IIS Reset.

About the above issue:

During activation of topology via PowerShell, you may observe error,

System.Management.Automation.MethodInvocationException:
Exception calling "Activate" with "0" argument(s):
"Could not connect to the HostController service on server Topology Activation could not be started"

During topology activation, If opened task manager and checked status of HostController.exe - It will be running always , during the activation of topology. However, NO NODERUNNER.EXE will be found in process list. No attempt to activate topology will be observed in ULS.

Application Event will show error below

Source: Microsoft-SharePoint Products-SharePoint Server
Event ID: 6482
Task Category: Shared Services
Level: Error
Keywords:
User: Timer Service Account
Computer:
Description:
Application Server Administration job failed for service instance Microsoft.Office.Server.Search.Administration.SearchServiceInstance (Service instance GUID).

Reason: A call to SSPI failed, see inner exception.

Technical Support Details:
System.ServiceModel.Security.SecurityNegotiationException: A call to SSPI failed, see inner exception. ---> System.Security.Authentication.AuthenticationException: A call to SSPI failed, see inner exception. ---> System.ComponentModel.Win32Exception: No authority could be contacted for authentication
   --- End of inner exception stack trace ---
   at System.Net.Security.NegoState.StartSendAuthResetSignal(LazyAsyncResult lazyResult, Byte[] message, Exception exception)
   at System.Net.Security.NegoState.StartSendBlob(Byte[] message, LazyAsyncResult lazyResult)
   at System.Net.Security.NegoState.CheckCompletionBeforeNextSend(Byte[] message, LazyAsyncResult lazyResult)
   at System.Net.Security.NegoState.ProcessReceivedBlob(Byte[] message, LazyAsyncResult lazyResult)
   at System.Net.Security.NegoState.StartSendBlob(Byte[] message, LazyAsyncResult lazyResult)
   at System.Net.Security.NegoState.ProcessAuthentication(LazyAsyncResult lazyResult)
   at System.Net.Security.NegotiateStream.AuthenticateAsClient(NetworkCredential credential, String targetName, ProtectionLevel requiredProtectionLevel, TokenImpersonationLevel allowedImpersonationLevel)
   at System.ServiceModel.Channels.WindowsStreamSecurityUpgradeProvider.WindowsStreamSecurityUpgradeInitiator.OnInitiateUpgrade(Stream stream, SecurityMessageProperty& remoteSecurity)
   --- End of inner exception stack trace ---

Server stack trace:

   at System.ServiceModel.Channels.WindowsStreamSecurityUpgradeProvider.WindowsStreamSecurityUpgradeInitiator.OnInitiateUpgrade(Stream stream, SecurityMessageProperty& remoteSecurity)
   at System.ServiceModel.Channels.StreamSecurityUpgradeInitiatorBase.InitiateUpgrade(Stream stream)
   at System.ServiceModel.Channels.ConnectionUpgradeHelper.InitiateUpgrade(StreamUpgradeInitiator upgradeInitiator, IConnection& connection, ClientFramingDecoder decoder, IDefaultCommunicationTimeouts defaultTimeouts, TimeoutHelper& timeoutHelper)
   at System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.SendPreamble(IConnection connection, ArraySegment`1 preamble, TimeoutHelper& timeoutHelper)
   at System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.DuplexConnectionPoolHelper.AcceptPooledConnection(IConnection connection, TimeoutHelper& timeoutHelper)
   at System.ServiceModel.Channels.ConnectionPoolHelper.EstablishConnection(TimeSpan timeout)
   at System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.CallOnceManager.CallOnce(TimeSpan timeout, CallOnceManager cascade)
   at System.ServiceModel.Channels.ServiceChannel.EnsureOpened(TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]:

at Microsoft.Office.Server.Search.Administration.SearchServiceInstance.Synchronize()
at Microsoft.Office.Server.Administration.ApplicationServerJob.ProvisionLocalSharedServiceInstances(Boolean isAdministrationServiceJob)

ULS will have evidence for Timer Service account trying to contact HostController Service and fails trying to authenticate.

Once the activation of topology is failed, we can see detailed exception in ULS:

OWSTIMER.EXE SharePoint Server Search Administration aizcv Exception Failed to update hostcontroller primary repository version System.ServiceModel.Security.SecurityNegotiationException: A call to SSPI failed, see inner exception. ---> System.Security.Authentication.AuthenticationException: A call to SSPI failed, see inner exception. ---> System.ComponentModel.Win32Exception: No authority could be contacted for authentication --- End of inner exception stack trace ---

at System.Net.Security.NegoState.StartSendAuthResetSignal(LazyAsyncResult lazyResult, Byte[] message, Exception exception) at System.Net.Security.NegoState.StartSendBlob(Byte[] message, LazyAsyncResult lazyResult) System.Net.Security.NegoState.CheckCompletionBeforeNextSend(Byte[] message, LazyAsyncResult lazyResult) at System.Net.Security.NegoState.ProcessReceivedBlob(Byte[] message, LazyAsyncResult lazyResult) at System.Net.Security.NegoState.StartSendBlob(Byte[] message, LazyAsyncResult lazyResult) at System.Net.Security.NegoState.ProcessAuthentication(LazyAsyncResult lazyResult) at System.Net.Security.NegotiateStream.AuthenticateAsClient(NetworkCredential credential, String targetName, ProtectionLevel requiredProtectionLevel, TokenImpersonationLevel allowedImpersonationLevel) at System.ServiceModel.Channels.WindowsStreamSecurityUpgradeProvider.WindowsStreamSecurityUpgradeInitiator.OnInitiateUpgrade(Stream stream, SecurityMessageProperty& remoteSecurity)

SharePoint 2010: Resolving A UserProfile Service Application connecting to WCF endpoint: System.TimeoutException error

BK_R — Mon, 09 Jun 2014 00:41:00 GMT

Scenarios:

A) ... Every morning UPA worker process becomes unresponsive or web requests get queued for minutes and subsequently time out. Users of consuming farm experiences errors on their site pages while loading sites which require information from UPA.

B)... Users experiencing slowness while site load. SharePoint server with UPS SA service running, shows large number of connection requests queued in UPS SA worker process. These connection requests gets timeout after every 20 seconds.

C).... ULS shows timeout for any connection requests made to UPS SA (May be a request from client accessing sites, consuming farm or Newsgator social sites)

We see the following messages in the ULS logs when we try to access the site from the server that has the User Profile Service running.

Exception occurred while connecting to WCF endpoint: System.TimeoutException: The request channel timed out while waiting for a reply after 00:00:20. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout. ---> System.TimeoutException: The HTTP request to 'https://servername:32843/UPSSCEndPointGUID/ProfilePropertyService.svc' has exceeded the allotted timeout of 00:00:20. The time allotted to this operation may have been a portion of a longer timeout. ---> System.Net.WebException: The operation has timed out System.Net.HttpWebRequest.GetResponse()

System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout) --- End of inner exception stack trace ---

System.ServiceModel.Channels.HttpChannelUtilities.ProcessGetResponseWebException(WebException webException, HttpWebRequest request, HttpAbortReason abortReason)
System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)
System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)     --- End of inner exception stack trace ---    Server stack trace:
System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)

System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityRequestChannel.Request(Message message, TimeSpan timeout)     at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout)
System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)    Exception rethrown at [0]:      at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)    System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
Microsoft.Office.Server.UserProfiles.IProfilePropertyService.GetProfileProperties()
Microsoft.Office.Server.UserProfiles.ProfilePropertyServiceClient.<>c__DisplayClass1.b__0(IProfilePropertyService channel)

Microsoft.Office.Server.UserProfiles.MossClientBase`1.ExecuteOnChannel(String operationName, CodeBlock codeBlock)

UserProfileApplicationProxy.InitializePropertyCache: Microsoft.Office.Server.UserProfiles.UserProfileException: System.TimeoutException at Microsoft.Office.Server.UserProfiles.MossClientBase`1.ExecuteOnChannel(String operationName, CodeBlock codeBlock) Microsoft.Office.Server.UserProfiles.ProfilePropertyServiceClient.ExecuteOnChannel(String operationName, CodeBlock codeBlock) at Microsoft.Office.Server.UserProfiles.ProfilePropertyServiceClient.GetProfileProperties() Microsoft.Office.Server.Administration.UserProfileApplicationProxy.RefreshProperties(Guid applicationID)

Microsoft.Office.Server.Utilities.SPAsyncCache`2.GetValueNow(K key)

Microsoft.Office.Server.Utilities.SPAsyncCache`2.GetValue(K key, Boolean asynchronous)

Microsoft.Office.Server.Administration.UserProfileApplicationProxy.InitializePropertyCache()

Cause

We have added too many accounts (Users/Groups) to User Profile Service Application Administrators group. Each item in this ACL increases the time taken for ACL resolution and results in WCF endpoint timeout if ACL resolution takes more than 10-15 seconds. Default WCF end point timeout is 20 seconds.

Customer has multiple domains (child + Parent) and has to add too many users to User Profile Service Application Administrators group to grant required permissions for consuming farm.

Resolution

WorkAround 1

Use universal group in UPA Admin ACL in publishing farm. Nest all other groups and accounts in this Universal group. Add the universal group to User Profile Service application ACL.

Details:

Go to SharePoint Central administration, Manage Service Applications > Highlight UserProfile Service Apllication > Select Administrators from ribbon > Remove users/groups found in this ACL.

Create universal group (UG) in parent domain of SharePoint server. Add all removed users/groups to this UG. Add this Universal group to UPA SA Admin ACL in SharePoint.

Workaround 2

Increase WCF EndPoint timeout from 20 seconds to 60 seconds. Also, start User profile Service on multiple servers to load balance WCF requests. This should help reducing connection timeouts. For permanent fix, customer should remove user/group items and add few Universal groups only to UPS SA administratot ACL.

More Information

Use PowerShell script to check timeout when resolving UPS SA ACL on problem app servers . This script will tell the time taken for validating items in UPS SA Administrators ACL. If the value is more than 4-6 seconds then it can cause WCF timeouts at large connection requests from client.

The script as given below

# $upa must have your desired service application name.

Add-PSSnapin microsoft.sharepoint.powershell -erroraction silentlycontinue
$upa = get-spserviceapplication | ? {$_.TypeName.ToString().Equals("User Profile Service Application")}
$starttime = [system.DateTime]::Now
$acl = $upa.GetAdministrationAccessControl()
$accessRules = $acl.accessRules
$endTime = [system.DateTime]::Now
$elapsedTime = $endTime.subtract($startTime).seconds

Write-Host "Start Time: $StartTime End Time:$endTime Duration: $elapsedTime Seconds"
Write-Host ""

Resolving 'Duplicate mapping' error (Event 6301) for 'Title' property in SharePoint user Profile Service Application property Mapping

BK_R — Sun, 08 Jun 2014 23:47:00 GMT

How to edit Sharepoint user profile property mapping alias: Changing 'SPS-JobTitle' to 'Title' property mapping

Consider the following scenario:

In an existing User Profile Service Application (UPSA) with a synchronization connection to AD, you map 'SPS-JobTitle' to import 'Employee Type' (or any other AD attribute) and then map 'Title' property in UPSA to import an attribute in AD (for example 'Title').

You then run a full profile synchronization and wait for it to complete. The expected result is to get new values as per mapped attributes imported for 'Title' property in the SharePoint User Profiles. However, you observe that the 'Title' value of existing users is never changed and newly added users get a 'Duplicate mapping' error (Event 6301) for 'Title' property in MIISCLIENT.exe during 'SD_DeltaSync' and 'MOSS_DeltaSync' stages. Existing users having 'Title' values show no errors in FIM or ULS. However, the new attribute mapping value will not be updated.

New users will get duplicate mapping errors in Event logs.

Log Name: Application
Source: FIMSynchronizationService
Date: [ Date and Time ]
Event ID: 6301
Task Category: Server
Level: Error
Keywords: Classic
User: N/A
Computer: < servername >
Description:
The server encountered an unexpected error in the synchronization engine:

"BAIL: MMS(24436): eafam.cpp(1298): 0x80230304 (The image or dimage already has an attribute with that name.)
BAIL: MMS(24436): eafam.cpp(946): 0x80230304 (The image or dimage already has an attribute with that name.)
BAIL: MMS(24436): eafam.cpp(1013): 0x80230304 (The image or dimage already has an attribute with that name.)
BAIL: MMS(24436): amexec.cpp(1701): 0x80230304 (The image or dimage already has an attribute with that name.)
BAIL: MMS(24436): amexec.cpp(2086): 0x80230304 (The image or dimage already has an attribute with that name.)
BAIL: MMS(24436): eaf.cpp(1417): 0x80230304 (The image or dimage already has an attribute with that name.)
BAIL: MMS(24436): eaf.cpp(657): 0x80230304 (The image or dimage already has an attribute with that name.)
ERR: MMS(24436): synccoreimp.cpp(5271): 0x80230304 - export-flow failed 0x80230304
BAIL: MMS(24436): synccoreimp.cpp(5272): 0x80230304 (The image or dimage already has an attribute with that name.)
BAIL: MMS(24436): synccoreimp.cpp(4863): 0x80230304 (The image or dimage already has an attribute with that name.)
BAIL: MMS(24436): synccoreimp.cpp(10889): 0x80230304 (The image or dimage already has an attribute with that name.)
BAIL: MMS(24436): synccoreimp.cpp(10573): 0x80230304 (The image or dimage already has an attribute with that name.)
BAIL: MMS(24436): synccoreimp.cpp(2547): 0x80230304 (The image or dimage already has an attribute with that name.)
BAIL: MMS(24436): synccoreimp.cpp(4375): 0x80230304 (The image or dimage already has an attribute with that name.)
BAIL: MMS(24436): synccoreimp.cpp(10072): 0x80230304 (The image or dimage already has an attribute with that name.)
BAIL: MMS(24436): synccoreimp.cpp(3837)

Note: Skipping additional xml data; this data will show 'Title' property as the duplicate.

Reason behind this error:

In UPSA, following the first full sync, the 'SPS-Title' imports 'Title' from AD. 'Title' in SharePoint uses the value provided by 'SPS-JobTitle'. So, both properties in SharePoint will show the same value. Once a value is assigned to 'JobTitle', any new import mapping without clearing the existing projection (in FIM metaverse) fails to edit the existing value in FIM. This results in Sync errors on user objects affected by the mapping change.

A workaround is to first run a full sync without any mapping for 'Title' and 'JobTitle', and then map the properties as desired to AD. Run a full sync again. The steps required are listed below:

To map the 'SPS-JobTitle' property and ‘Title’ to different attributes in AD. Follow the steps given below for a new UPA

A. Create a new UPA SA using new ProfileDB, SocialDB and SyncDB.
B. Create UPSA Sync connections.
C. Edit ‘SPS-JobTitle’ and remove import mapping to ‘Title’
D. Run a full sync without any mapping for the ‘Job Title and ‘Title’.
E. Add Title =Import Title
F. Run full sync again
G. Add SPS-JobTitle = Import ‘Employee Type’
H. Run Full Syc
I. Open test users profile and observe changes.

Follow the steps given below for an existing UPA

A. Remove existing mapping for ‘Job Title and ‘Title’.
B. Run a full sync without any mapping for the ‘Job Title and ‘Title’.
C. Add Title =Import Title
D. Run full sync again
E. Add SPS-JobTitle = Import ‘Employee Type’
F. Run Full Syc
G. Open test users profile and observe changes. The changes should take effect as required.

About ‘Title’ and Job Title:

In SharePoint 2010, the ‘Job Title (SPS-JobTitle)’ property imports ‘Title’ attribute value from AD by default. This import mapping is created by SharePoint immediately after the creation of AD sync connection in UPSA. The ‘Title’ property uses 'SPS-JobTitle' value as primary property in UPSA. If a full sync runs, both properties in User Profile get the ‘Title’ value from AD.

Similar issue [https://blogs.technet.com/b/steve_chen/archive/2010/10/11/user-profile-import-from-bdc-fails-on-sharepoint-server-2010-with-event-id-6301.aspx]

No XsltTransformTimeOut property found in SharePoint farm

BK_R — Sun, 08 Jun 2014 23:28:00 GMT

Scenario

In Production , you have an XSLT list view web part displaying a list This web part returns timeout errors when loaded page initially, when refreshed, error disappears and page loads successfully. An error "Unable to display this Web Part. To troubleshoot the problem open this Web page using Microsoft SharePoint Foundation-compatible HTML editor such as Microsoft SharePoint Designer" is displayed when error occurs on page load.

A Possible reason

if we have many columns added to the target form library, which when opened using XSLT web parts was hitting a timeout error which took more time than what is set for XSLT timeout . In my case, the timeout value was set to 1 second. Using developer dashboard, we can see the timeout error comes if page refresh takes more than 1 second.

A Possible Resolution

1. Install February 2012 CU for SharePoint if you do not have XsltTransformTimeOut setting in farm.

Or

2. Increase timeout to 5 in $farm = Get-SPFarm $farm.XsltTransformTimeOut using following PS commands.
You have to use DeveloperDashboard on problem pages and determine average time taken for page load. Then use that time to increase the timeout.

About XsltTransformTimeOut

I had tested XsltTransformTimeOut in test farm. This is set to 1 second by default. Increasing this timeout can affect performance during large form rendering. To use as a workaround for customer's issue, suggested to increase timeout to5 second. I had tested it and found working if timeout increased to 2 or 5.

Increased timeout to 5 in $farm = Get-SPFarm $farm.XsltTransformTimeOut using following PS commands.

Steps to follow
==========

Check Current Setting

$farm = Get-SPFarm
$farm.XsltTransformTimeOut

The default setting is 1 second. The PowerShell code below changes it to 2 seconds which should be sufficient for your farm. Developer dashboard returned average 1 to 1.3 seconds delay in your production farm. So, I’m choosing 5 seconds timeout.

Change Setting to 5 Seconds

$farm = Get-SPFarm
$farm.XsltTransformTimeOut = 5
$farm.Update()