02/19/2016 15:16:21.79         OWSTIMER.EXE (0x259C)       0x2D04    SharePoint Portal Server        User Profiles           9sip         High                UserProfileApplication.SynchronizeMIIS: Failed to configure MIIS post database, will attempt during next rerun. Exception: System.Runtime.InteropServices.SEHException (0x80004005): External component has thrown an exception.     at Microsoft.Office.Server.UserProfiles.Synchronization.ILMPostSetupConfiguration.ConfigureMiisStage2()     at Microsoft.Office.Server.Administration.UserProfileApplication.SetupSynchronizationService(ProfileSynchronizationServiceInstance profileSyncInstance).              eadf609d-1937-f04a-6fd0-052797a13efb

The exception seems to be pointing to the custom PowerShell profile and I have learnt from customer that they configured custom PowerShell profile previously which they later removed. So to eliminate any chance of custom profile role in this issue, I had to rename\backup of the windows profile for the timer service account which we use to start the UPS. As you know it should be done by logging in to server using a different account and again login to server using the timer service account so that it will create a new and clean profile. This time the User Profile Synchronization service started like a champ. 

Happy SharePointing!!!

w3wp.exe (0x3A94) 0x31A8 SharePoint Foundation General brz8 High Url validation failed for https://SiteURL/_vti_bin/OfficialFile.asmx during Record and document center connection configuration with exception The request failed with HTTP status 401: Unauthorized.: at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall) at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters) at Microsoft.SharePoint.OfficialFileSoap.GetServerInfo() at Microsoft.SharePoint.ApplicationPages.OfficialFileAdminPage.<>c__DisplayClass2.b__0() at Microsoft.SharePoint.SPSecurity.<>c__DisplayClass5.b__3() at Microsoft.SharePoint.Utilities.SecurityContext.RunAsProcess(CodeToRunElevated secureCode) at Microsoft.SharePoint.SPSecurity.RunWithElevatedPrivileges(WaitCallback secureCode, Object param) at Microsoft.SharePoint.SPSecurity.RunWithElevatedPrivileges(CodeToRunElevated secureCode) at Microsoft.SharePoint.ApplicationPages.OfficialFileAdminPage.IsValidUriToRouter(Uri uriToValidate) d5ff109d-0433-60a3-4840-479983a2a474 

The Loopbackcheck were already disabled on the box and after a troubleshooting this issue for some time on their CA box I found the Windows credentials were configured to cache on the servers (Control Panel\User Accounts\Credential Manager). We were able to configure 'Send to Connections' after removing the credentials from credential manager. 

I was working with a customer who has issue with SSRS reports which keeps in spinning for around 2 minutes and end up with throwing following error in SharePoint 2013 farm environment. But the normal reports are generating successfully which takes less than a minute to load.

"Sys.WebForms.PageRequestManagerServerErrorException: An unknown error occurred while processing the request on the server. The status code returned from the server was: 503"

and you may see following time out exception  in ULS Logs...

08/15/2014 10:59:49.66* w3wp.exe (0x1130) 0x2C78 SharePoint Foundation Service Connections ev2x Verbose ...om the socket did not complete within the allotted timeout of 00:01:00. The time allotted to this operation may have been a portion of a longer timeout. ---&amp;gt; System.Net.Sockets.SocketException: The I/O operation has been aborted because of either a thread exit or an application request at System.ServiceModel.Channels.SocketConnection.HandleReceiveAsyncCompleted() at System.ServiceModel.Channels.SocketConnection.OnReceiveAsync(Object sender, SocketAsyncEventArgs eventArgs) --- End of inner exception stack trace --- at System.Runtime.AsyncResult.End[TAsyncResult](IAsyncResult result) at System.ServiceModel.Channels.ConnectionStream.ReadAsyncResult.End(IAsyncResult result) at System.Net.FixedSizeReader.ReadCallback(IAsyncResult transportResult)</ExceptionString>...

08/15/2014 10:59:49.66* w3wp.exe (0x1130) 0x2C78 SharePoint Foundation Service Connections ev2x Verbose ...<InnerException><ExceptionType>System.Net.Sockets.SocketException, System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</ExceptionType><Message>The I/O operation has been aborted because of either a thread exit or an application request</Message><StackTrace> at System.ServiceModel.Channels.SocketConnection.HandleReceiveAsyncCompleted() at System.ServiceModel.Channels.SocketConnection.OnReceiveAsync(Object sender, SocketAsyncEventArgs eventArgs)</StackTrace><ExceptionString>System.Net.Sockets.SocketException (0x80004005): The I/O operation has been aborted because of either a thread exit or an application request at System.ServiceModel.Channels.SocketConnection.HandleReceiveAsyncCompleted() at System.ServiceModel.Channels.SocketConnection.OnReceiveAsync(Ob...

I had a deep drive in to research on this issue by making use of articles and my energetic lab and found this may due to the default timeout value in Request Management Service..

Basically, If the Request Management service is enabled, it manages all incoming requests by evaluating logic rules against the user requests in order to determine what\which action to take and which machine or machines (targets) in the farm should handle the request.

By default the Request time out value for Request Management service are as follows,

RequestTimeout                    : 00:01:40
RequestExecutionTimeout       : 01:00:00

So the error throws from RM service for those reports who take more than 1 minute to generate the report.

If you do not have any defined rules in Request management Service other than the by default value, you may keep stop the RM service in your farm. If you would like to keep the RM Service running, then go ahead and increase the time out value by making use of following PowerShell  script.

$waUrl = "URLofWEbApp"

$wa = Get-SPWebApplication $waUrl

$rmSettings = $wa | Get-SPRequestManagementSettings

$req=$wa.RequestManagementSettings

$timeout2= New-TimeSpan -minutes 5

$req.Requesttimeout=$timeout2

$req.update()

Happy Troubleshooting..

Using the following characters anywhere in a site name, in a subsite name, or in a site or Active Directory group names are not supported:

Tilde (~)
Number sign (#)
Percent (%)
Ampersand (&)
Asterisk (*)
Braces ({ })
Backslash (\)
Colon (:)
Angle brackets (< >)
Question mark (?)
Slash (/)
Plus sign (+)
Pipe (|)
Quotation mark (")

Ref: https://support.microsoft.com/en-us/kb/905231 

By looking at the Fiddler I understand that the SAML token expiration value was 8:05 hrs. 

Created xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd">2015-04-28T20:44:00.307Z</wsu:Created>

<wsu:Expires xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd">2015-04-29T04:49:00.307Z</wsu:Expires>

Had a look at the STSConfiguration (Get-SPTrustedIdentityToekIssuer) and found the STS token expiration (LogonTokenCacheExpirationWindow) set to 2 minutes. This means the sliding session should be every 8:03 hrs, which means the client will try to re-authenticate with the SAML provider only every 8:03 hrs.  

Basically the value of the LogonTokenCacheExpirationWindow property is always less than the SAML token lifetime; otherwise, you'll see a loop whenever a user tries to access your SharePoint web application and keeps being redirected back to the token issuer. Here are the details on sliding session and its logic: https://msdn.microsoft.com/en-us/library/hh446526.aspx 

So here the questions is why the users being authenticates very frequently?  Customer already informed me that they played a lot with the values of DistributedLogonTokenCache parameters. I collected the SharePoint ULS logs in VerboseEx for an hour from their servers using Merge-SPLogFile command and found following,

-        Here the request started where the user tried to access a resource in SharePoint.

w3wp.exe (ServerName:0x057C)                   0x03C4      SharePoint Foundation                    Logging Correlation Data           xmnv         Medium   Name=Request (GET:https://SiteURL/_api/sp.userprofiles.PeopleManager/GetMyProperties)        91d7039d-55e9-7036-63d4-13bd0943287a

-        Here the exception thrown on ‘Distributed logon token cache’ complaining about the value of MaxBufferSize

w3wp.exe (ServerName:0x057C)                   0x03C4      SharePoint Foundation                    DistributedCache     agyfv         Unexpected               Unexpected error occurred in method 'GetObject' , usage 'Distributed Logon Token Cache' - Exception 'Microsoft.ApplicationServer.Caching.DataCacheException: ErrorCode<ERRCA0016>:SubStatus<ES0001>:The connection was terminated, possibly due to server or network problems or serialized Object size is greater than MaxBufferSize on server. Result of the request is unknown. ---> System.TimeoutException: The socket was aborted because an asynchronous receive from the socket did not complete within the allotted timeout of 00:01:00. The time allotted to this operation may have been a portion of a longer timeout. ---> System.IO.IOException: The read operation failed, see inner exception. ---> System.TimeoutException: The socket was aborted because an asynchronous receive from the socket did not complete within the allotted timeout of 00:01:00. The time allotted to this operation may have been a portion of a longer timeout. ---> System.Net.Sockets.SocketException: The I/O operation has been aborted because of either a thread exit or an application request     at System.ServiceModel.Channels.SocketConnection.HandleReceiveAsyncCompleted()     at System.ServiceModel.Channels.SocketConnection.OnReceiveAsync(Object sender, SocketAsyncEventArgs eventArgs)     --- End of inner exception stack trace ---     at System.Runtime.AsyncResult.End[TAsyncResult](IAsyncResult result)     at System.ServiceModel.Channels.ConnectionStream.ReadAsyncResult.End(IAsyncResult result)     at System.Net.FixedSizeReader.ReadCallback(IAsyncResult transportResult)     --- End of inner exception stack trace ---     at System.Net.Security.NegotiateStream.EndRead(IAsyncResult asyncResult)     at System.ServiceModel.Channels.StreamConnection.EndRead()     --- End of inner exception stack trace ---     at System.Runtime.AsyncResult.End[TAsyncResult](IAsyncResult result)     at System.ServiceModel.Channels.TransportDuplexSessionChannel.EndReceive(IAsyncResult result)     at Microsoft.ApplicationServer.Caching.WcfClientChannel.CompleteProcessing(IAsyncResult result)     --- End of inner exception stack trace ---     at Microsoft.ApplicationServer.Caching.DataCache.ThrowException(ResponseBody respBody, RequestBody reqBody)     at Microsoft.ApplicationServer.Caching.DataCache.InternalGet(String key, DataCacheItemVersion& version, String region, IMonitoringListener listener)     at Microsoft.ApplicationServer.Caching.DataCache.<>c__DisplayClass49.<Get>b__48()     at Microsoft.SharePoint.DistributedCaching.SPDistributedCache.GetObject(String key)'.       91d7039d-55e9-7036-63d4-13bd0943287a

-        The client failed to get the token for user in question from Distributed cache.

w3wp.exe (ServerName:0x057C)                   0x03C4      SharePoint Foundation                    DistributedCache     air4c           Medium   Token Cache: Failed to get token from distributed cache for '05.t|okta|user@domain.com'.(This is expected during the process warm up or if data cache Initialization is getting done by some other thread). Exception: 'Microsoft.SharePoint.DistributedCaching.SPDistributedCacheConnectionTerminatedException: Connection to the server terminated,check if the cache host(s) is running  ---> Microsoft.ApplicationServer.Caching.DataCacheException: ErrorCode<ERRCA0016>:SubStatus<ES0001>:The connection was terminated, possibly due to server or network problems or serialized Object size is greater than MaxBufferSize on server. Result of the request is unknown. ---> System.TimeoutException: The socket was aborted because an asynchronous receive from the socket did not complete within the allotted timeout of 00:01:00. The time allotted to this operation may have been a portion of a longer timeout. ---> System.IO.IOException: The read operation failed, see inner exception. ---> System.TimeoutException: The socket was aborted because an asynchronous receive from the socket did not complete within the allotted timeout of 00:01:00. The time allotted to this operation may have been a portion of a longer timeout. ---> System.Net.Sockets.SocketException: The I/O operation has been aborted because of either a thread exit or an application request     at System.ServiceModel.Channels.SocketConnection.HandleReceiveAsyncCompleted()     at System.ServiceModel.Channels.SocketConnection.OnReceiveAsync(Object sender, SocketAsyncEventArgs eventArgs)     --- End of inner exception stack trace ---     at System.Runtime.AsyncResult.End[TAsyncResult](IAsyncResult result)     at System.ServiceModel.Channels.ConnectionStream.ReadAsyncResult.End(IAsyncResult result)     at System.Net.FixedSizeReader.ReadCallback(IAsyncResult transportResult)     --- End of inner exception stack trace ---     at System.Net.Security.NegotiateStream.EndRead(IAsyncResult asyncResult)     at System.ServiceModel.Channels.StreamConnection.EndRead()     --- End of inner exception stack trace ---     at System.Runtime.AsyncResult.End[TAsyncResult](IAsyncResult result)     at System.ServiceModel.Channels.TransportDuplexSessionChannel.EndReceive(IAsyncResult result)     at Microsoft.ApplicationServer.Caching.WcfClientChannel.CompleteProcessing(IAsyncResult result)     --- End of inner exception stack trace ---     at Microsoft.ApplicationServer.Caching.DataCache.ThrowException(ResponseBody respBody, RequestBody reqBody)     at Microsoft.ApplicationServer.Caching.DataCache.InternalGet(String key, DataCacheItemVersion& version, String region, IMonitoringListener listener)     at Microsoft.ApplicationServer.Caching.DataCache.<>c__DisplayClass49.<Get>b__48()     at Microsoft.SharePoint.DistributedCaching.SPDistributedCache.GetObject(String key)     --- End of inner exception stack trace ---     at Microsoft.SharePoint.DistributedCaching.SPDistributedCache.GetObject(String key)     at Microsoft.SharePoint.IdentityModel.SPDistributedSecurityTokenCache.GetObject(String key)     at Microsoft.SharePoint.IdentityModel.SPTokenCache.TryGetCachedToken(String cacheKey)'.              91d7039d-55e9-7036-63d4-13bd0943287a

w3wp.exe (ServerName:0x057C)                   0x03C4      SharePoint Foundation                    DistributedCache     air4d          Medium   Token Cache: Reverting to local cache to get the token for '05.t|okta|user@domain.com'.             91d7039d-55e9-7036-63d4-13bd0943287a

w3wp.exe (ServerName:0x057C)                   0x03C4      SharePoint Foundation                    Claims Authentication                fr16            Verbose   Token Cache: Entry missing for user '05.t|okta|user@domain.com'.             91d7039d-55e9-7036-63d4-13bd0943287a

-        The user forcefully signed out to get a new token.

w3wp.exe (ServerName:0x057C)                   0x03C4      SharePoint Foundation                    Claims Authentication                af30i          Verbose   Token Cache: Failed to find token for user '05.t|okta|user@domain.com' for cookie so signing out the user.     91d7039d-55e9-7036-63d4-13bd0943287a

Basically in this scenario, SharePoint stores the user’s token in the user's browser session and in the DistributedCacheLogonTokenCache container. When SharePoint tried to retrieve the token from distributed cache, the connection would time out or a connection would be unavailable or the Distributed cache not able to serve the cookie detail due to some reason. Since it couldn't validate the presented token SharePoint had no choice but to log the user out and redirect them to the sign in page.

I shared one of my Blog to the customer to make the DistributedLogonTokenCache values to standardize and to increase the  distributed cache timeout values to 10 seconds. 

After changing the Distributed cache value we noticed a change in authentication behavior that it re-authenticates very frequently than before like in every few minutes.

I gone through the current DistributedLogonTokenCache value sand found "MaxConnectionsToServer" was set to "100".

The number of shared connections is determined by the maxConnectionsToServer configuration setting, which defaults to the number of processor cores on the client machine. Ref: https://msdn.microsoft.com/en-us/library/azure/dn386109.aspx. I shared following script with customer which to set “MaxConnectionsToServer” back to the recommended value of “1” for all  containers--note that if “MaxConnectionsToServer” is set too high, we will see high memory usage from the DistributedCache process, and possible intermittent hangs. 

Add-PSSnapin Microsoft.Sharepoint.Powershell

#DistributedLogonTokenCache
$DLTC = Get-SPDistributedCacheClientSetting -ContainerType DistributedLogonTokenCache
$DLTC.MaxConnectionsToServer = 1
Set-SPDistributedCacheClientSetting -ContainerType DistributedLogonTokenCache -DistributedCacheClientSettings $DLTC

#DistributedViewStateCache
$DVSC = Get-SPDistributedCacheClientSetting -ContainerType DistributedViewStateCache
$DVSC.MaxConnectionsToServer = 1
Set-SPDistributedCacheClientSetting -ContainerType DistributedViewStateCache $DVSC

#DistributedAccessCache
$DAC = Get-SPDistributedCacheClientSetting -ContainerType DistributedAccessCache
$DAC.MaxConnectionsToServer = 1
Set-SPDistributedCacheClientSetting -ContainerType DistributedAccessCache $DAC

#DistributedAccessCache
$DAF = Get-SPDistributedCacheClientSetting -ContainerType DistributedAccessCache
$DAF.MaxConnectionsToServer = 1
Set-SPDistributedCacheClientSetting -ContainerType DistributedActivityFeedCache $DAF

#DistributedActivityFeedLMTCache
$DAFC = Get-SPDistributedCacheClientSetting -ContainerType DistributedActivityFeedLMTCache
$DAFC.MaxConnectionsToServer = 1
Set-SPDistributedCacheClientSetting -ContainerType DistributedActivityFeedLMTCache $DAFC

#DistributedBouncerCache
$DBC = Get-SPDistributedCacheClientSetting -ContainerType DistributedBouncerCache
$DBC.MaxConnectionsToServer = 1
Set-SPDistributedCacheClientSetting -ContainerType DistributedBouncerCache $DBC

#DistributedDefaultCache
$DDC = Get-SPDistributedCacheClientSetting -ContainerType DistributedDefaultCache
$DDC.MaxConnectionsToServer = 1
Set-SPDistributedCacheClientSetting -ContainerType DistributedDefaultCache $DDC

#DistributedSearchCache
$DSC = Get-SPDistributedCacheClientSetting -ContainerType DistributedSearchCache
$DSC.MaxConnectionsToServer = 1
Set-SPDistributedCacheClientSetting -ContainerType DistributedSearchCache $DSC

#DistributedSecurityTrimmingCache
$DTC = Get-SPDistributedCacheClientSetting -ContainerType DistributedSecurityTrimmingCache
$DTC.MaxConnectionsToServer = 1
Set-SPDistributedCacheClientSetting -ContainerType DistributedSecurityTrimmingCache $DTC

#DistributedServerToAppServerAccessTokenCache
$DSTAC = Get-SPDistributedCacheClientSetting -ContainerType DistributedServerToAppServerAccessTokenCache
$DSTAC.MaxConnectionsToServer = 1
Set-SPDistributedCacheClientSetting -ContainerType DistributedServerToAppServerAccessTokenCache $DSTAC

 After running the script, you’ll need to do a Restart-CacheCluster to restart the AppFabric services.

We closely monitor the environment for next few days and found the web application doesnt try to re-authenticate frequently and the issue is resolved after changing the "MaxConnectionToServer" value to "1".

Recently the following technet article updated with script to fine tune the distributed cache service,

https://technet.microsoft.com/en-us/library/jj219613.aspx

 Happy SharePointing :) 

I had an interesting issue where customer used ADImport to fetch the profile from AD to SharePoint. They had requirement to use Target audience functionality and when they look the "Manage Audience" in user profile service application, it shows only 5 users for a AD group where they have 186 users are part of this group in AD. 

Had a look at the ULS while Full import and found following exception which talks about the account duplication in DNLookup table. 


OWSTIMER.EXE(SERVERNAME:0x192C) 0x12D4 SharePoint Portal Server User Profiles c8hz Verbose ProfileImportExportService.UpdateWithProfileChangeData: Begin Add 'Domain\User'  

OWSTIMER.EXE (SERVERNAME:0x192C)      0x12D4 SharePoint Portal Server       User Profiles                  aer4e Medium Exception while updating properties for 'Domain\User': Property: Url, Exception Microsoft.Office.Server.UserProfiles.PropertyNotDefinedException: Property Not Defined: Url.  An administrator must create this property in the Profile Administration tool.     at Microsoft.Office.Server.UserProfiles.UserProfile.get_Item(String strPropName)     at Microsoft.Office.Server.UserProfiles.UserProfile.GetProfileValueCollection(String propName)     at Microsoft.Office.Server.UserProfiles.UserProfile.BulkPropertiesUpdate(Int64 importExportId, Hashtable properties, String strAccountName).  

OWSTIMER.EXE (SERVERNAME:0x192C)      0x12D4 SharePoint Portal Server       User Profiles                  aer4e Medium Exception while updating properties for 'Domain\User': Property: Description, Exception Microsoft.Office.Server.UserProfiles.PropertyNotDefinedException: Property Not Defined: Description.  An administrator must create this property in the Profile Administration tool.     at Microsoft.Office.Server.UserProfiles.UserProfile.get_Item(String strPropName)     at Microsoft.Office.Server.UserProfiles.UserProfile.GetProfileValueCollection(String propName)     at Microsoft.Office.Server.UserProfiles.UserProfile.BulkPropertiesUpdate(Int64 importExportId, Hashtable properties, String strAccountName).  

OWSTIMER.EXE (SERVERNAME:0x192C)      0x12D4 SharePoint Portal Server       User Profiles                  aer4e Medium Exception while updating properties for 'Domain\User': Property: MailNickName, Exception Microsoft.Office.Server.UserProfiles.PropertyNotDefinedException: Property Not Defined: MailNickName.  An administrator must create this property in the Profile Administration tool.     at Microsoft.Office.Server.UserProfiles.UserProfile.get_Item(String strPropName)     at Microsoft.Office.Server.UserProfiles.UserProfile.GetProfileValueCollection(String propName)     at Microsoft.Office.Server.UserProfiles.UserProfile.BulkPropertiesUpdate(Int64 importExportId, Hashtable properties, String strAccountName).  

OWSTIMER.EXE (SERVERNAME:0x192C)      0x12D4 SharePoint Portal Server       User Profiles                  ogvr High User profile record of "FMI\013081" was changed by 'Domain\ServiceAccount'.  

OWSTIMER.EXE (SERVERNAME:0x192C)      0x12D4 SharePoint Portal Server       User Profiles                  c8i0 Verbose ProfileImportExportService.UpdateWithProfileChangeData: End Add 'Domain\User' 

OWSTIMER.EXE (SERVERNAME:0x192C)      0x12D4 SharePoint Portal Server       User Profiles                  aei8n High ProfileImportExportService.UpdateWithProfileChangeData: Uncaught exception for object '14307fbf-5f94-48c4-a1f4-db8c1b409b96 ('Domain\User')' - System.InvalidOperationException: More than one DN specified for the same profile.     at Microsoft.Office.Server.UserProfiles.ProfileImportExportService.AddDNLookupTable(UserProfileApplicationProxy upaProxy, Guid partitionID, Int64 recordId, String objectType, String distinguishedName)     at Microsoft.Office.Server.UserProfiles.ProfileImportExportService.<>c__DisplayClass26.<UpdateWithProfileChangeData>b__24(Int32 idx).

Possible Causes

1) They may haven't do a Full profile import via UPA after the UPA configuration. 
2) Most of the users may create by custom code, Search or by using My Sites. 
3) First they were using UPS and later changed to ADImport. For some reason the profiles got corrupted.

Resolution

1) Run following powershell command and make note of guid of UPSA. 
Get-SPServiceApplication 
2) Run following commands one by one. 
$u = Get-SPServiceApplication 
Set-SPProfileServiceApplication $u -GetNonImportedObjects $true # Here you will get the results for profiles 'IsImportedvalue = 0' from ProfileDB 
Ran Set-SPProfileServiceApplication $u -PurgeNonImportedObjects $true # This cmdlet will mark the profiles for deletion and need MySiteCleanup job to delete those which are marked for deletion. If MySite is part of one user that added in missing profile, the same job will delete those MySites as well. Here is the detailed implications while running My Site Cleanup job.
3) Run Incremental profile import. Query following against ProfileDB and we should get no results. 
select * from DNLookup where isimported !=1  # Make sure you have backup of profile database before running any queries against it. 
4) Go ahead and run Full profile Import, once the above query returns no result.

 Reference : http://technet.microsoft.com/en-us/library/ff608004(v=office.15).aspx 

Happy SharePointing :) 

Done many steps at least to get something in the ULS regarding this issue like flushing the config cache, Checking the permission level, deleting the one-time jobs etc..

While thinking logically, it seems something wrong with the timer job who broken in some point for some reason. To confirm that ran following powershell script..

( http://blogs.msdn.com/b/tehnoonr/archive/2011/09/07/sharepoint-server-2010-unable-to-provision-user-profile-synchronization-service-after-upgrade.aspx )

$farm = Get-SPFarm
$disabledTimers = $farm.TimerService.Instances | where {$_.Status -ne "Online"}
if ($disabledTimers -ne $null)
{
foreach ($timer in $disabledTimers)
{
Write-Host "Timer service instance on server " $timer.Server.Name " is not Online. Current status:" $timer.Status
#Write-Host "Attempting to set the status of the service instance to online"
#$timer.Status = [Microsoft.SharePoint.Administration.SPObjectStatus]::Online
#$timer.Update()
}
}
else
{
Write-Host "All Timer Service Instances in the farm are online! No problems found"
}

The result of above script was  >> "Timer service instance on server  SERVERNAMEis not Online. Current status: Disabled"

Following script we used to start the timer service instance,

$farm = Get-SPFarm
$disabledTimers = $farm.TimerService.Instances | where {$_.Status -ne "Online"}
if ($disabledTimers -ne $null)
{
foreach ($timer in $disabledTimers)
{
Write-Host "Timer service instance on server " $timer.Server.Name " is not Online. Current status:" $timer.Status
Write-Host "Attempting to set the status of the service instance to online"
$timer.Status = [Microsoft.SharePoint.Administration.SPObjectStatus]::Online
$timer.Update()
}
}
else
{
Write-Host "All Timer Service Instances in the farm are online! No problems found"
}

Once the timer service instance is up, restarted the timer service and tried to start the user profile synchronization service after deleting timer jobs which are stuck to run one time. But this time also we wasn't able to make it and made a progress on working the functionality to write user profile logs to SharePoint ULS.

Later while checking the permission levels based on the file system, I found that the the 'NETWORK SERVICE' read per mission is missing on '14' hive. After clearing this permission issue again tried starting the UPS and found in ULS that the  UPA web service was getting stopped with '503 Service Unavailable'. When checked the Application and found the web service is in stopped state for the identity as farm account with following error in system even viewer.

Description:
The identity of application pool 'GUIID' is invalid. The user name or password that is specified for the identity may be incorrect, or the user may not have batch logon rights. If the identity is not corrected, the application pool will be disabled when the application pool receives its first request. If batch logon rights are causing the problem, the identity in the IIS configuration store must be changed after rights have been granted before Windows Process Activation Service (WAS) can retry the logon. If the identity remains invalid after the first request for the application pool is processed, the application pool will be disabled. The data field contains the error number.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>

The user agreed that he reset the farm account password due to some inaccessibility issues. So we re-enter the password for failed app pool identity and the  User Profile Synchronization Service started successfully.

Hope this is helpful !

SharePoint 2013 layouts page breaks after one or more post backs and loses View State with error below.

"Unexpected error occurred in method 'Put' , usage 'SPViewStateCache' - Exception 'Microsoft.ApplicationServer.Caching.DataCacheException: ErrorCode:SubStatus:The request timed out.. Additional Information : The client was trying to communicate with the server : net.tcp://servername:22233" 

In the ULS viewer OWSTIMER.EXE process were throwing below error.

"UserProfileApplication.SynchronizeMIIS: Failed to configure MIIS post database, will attempt during next rerun. Exception: System.Configuration.ConfigurationErrorsException: ERR_CONFIG_DB     at Microsoft.Office.Server.UserProfiles.Synchronization.ILMPostSetupConfiguration.ValidateConfigurationResult(UInt32 result)     at Microsoft.Office.Server.UserProfiles.Synchronization.ILMPostSetupConfiguration.ConfigureMiisStage2()     at Microsoft.Office.Server.Administration.UserProfileApplication.SetupSynchronizationService(ProfileSynchronizationServiceInstance profileSyncInstance)"

Based on above error I made action plan which directed to resolution.

1) Ran 'Get-SPServiceInstance' in Sharepoint Powershell to get the status of Profile Synchronization and found its in 'Provisioning' status.

2) Stopped Profile Synchronization service by running the command 'Stop-SPServiceInstance'.

3) Database was running on SQL2012 and  checked SQL Native client in C:\Windows\System32 and found it was version11 (sqlncli11.dll).

4) Next step was to check the SharePoint box and found there are no SQL Native client deployed which is one of prerequisite for SharePoint 2013 deployment.

5) After deploying SQL Server 2008 R2 SP1 Native Client in all SharePoint boxes, clicked on Start 'User Profile Synchronisation Service' and it Started successfully.