Common Root Causes:

• SharePoint 2010 Usage Files

  • Issue:
    • We have a known issue for SharePoint 2010 that the usage files are not deleted which is making the timer job “Microsoft SharePoint Foundation Usage Data Import” never completes. This issue is shipped with the below symptoms:
    • Missing or incomplete Web Analytics data
    • The “Timer Service Recycle” timer job is never able to actually restart the timer service. It tries to pause the Usage Data Import job and cannot, so after ten minutes it skips the recycle.
    • Memory bloat (leak) within owstimer.exe. You may see the timer service consume much more memory than normal. Restarting the timer service will bring it back to normal, but it will build up again over time.
    • The "Diagnostic Data Provider: Trace Log" timer job intermittently does not import the ULS log data into the Logging database.
  • Solution:

The fix is in the December 2013 Cumulative Update for SharePoint 2010.

Reference: https://support.microsoft.com/en-gb/help/2916922/sharepoint-2010-usage-files-are-not-deleted--causing-timer-service-pro

• Anti-Virus

  • Issue:
    • Whatsoever the antivirus that you are using on your SharePoint servers, it has to exclude some directories for SharePoint's sake. Ignoring this fact will cause a tremendous performance issue for your SharePoint site loading time, and in some cases unexpected behaviors! Talking here about the antivirus that is installed on the OS level.
  • Solution:
    • Based on the installed SharePoint version there is a list with the required paths. For the complete list, please check this article.
    • To be Noted:
      • Take into consideration that if you have SharePoint server, you will need to add the directories for both SharePoint Server, and SharePoint Foundation. For instance, for SharePoint Server 2013, add the paths for SharePoint Foundation 2013 and SharePoint Server 2013.
      • The directories should be excluded along with their sub-directories.
      • For the path "C:\Users\ServiceAccount\AppData\Local\Temp\", this is not a path, rather you have to change the highlighted word ServiceAccount with your SharePoint service accounts, and application pool accounts. For instance, the below are excluded on my SharePoint lab.
        • C:\Users\spfarm\AppData\Local\Temp\
        • C:\Users\Default_AppPool\AppData\Local\Temp\
        • C:\Users\Default_sa_AppPool\AppData\Local\Temp\
      • Important! If your SharePoint servers are virtual machines, add the directories on both the hosting machines, and the SharePoint servers. Whereas, I saw in some cases the antivirus which is installed on the hosting machine is causing troubles to the SharePoint servers which they are hosting. (strange but true!).

• SCOM APM

  • Issue:
    • Many customers come with a severe performance issue on their SharePoint servers, and after investigating their issues, we notice that they have SCOM APM installed which is not supported.
    • In fact, there are several articles talking about this issue with detailed analysis, and one of them is Bugras' which can be found here along with other articles that can be found below:
      • https://technet.microsoft.com/en-us/library/jj614617.aspx?tduid=(1dfb939b69d4a5ed09b44f51992a8b97)(256380)(2459594)(TnL5HPStwNw-v0X_tBOK3jzpbtaadMW8RA)()
      • https://support.microsoft.com/en-us/help/3216459/system-center-2016-operations-manager-apm-agent-causing-heap-corruptio
  • Solution:
    • Uninstall the Operations Manager monitoring agent
    • Restart the computer to completely unload all .dll files.
  • • Reinstall the Operations Manager agent without APM. To do this, run the following command:

      msiexec /i MonAgent.msi NOAPM=1

      Note The NOAPM=1 parameter causes the Operations Manager agent to be installed without the .NET Framework application performance monitoring.

• Security Token Service Causing high CPU Utilization

  More information can be found here.

• Network Issue

  • Issue:
    • Many performance issues are caused by having a non consistent network latency. Based on your network status your performance issue may become severe (getting SharePoint errors with correlations because of timeout errors to SQL), or only taking longer time than normal to load any SharePoint page.
  • Solution:
    • For your SharePoint farm to work properly you have to ensure having a consistent (no timeouts!) intra-farm latency (< 1 ms). Specifically, there should be a highly consistent intra-farm latency of <1ms (one way from SharePoint servers to SQL servers), 99.9% of the time over a period of ten minutes.
    • How to figure this out, or identify this issue? - This can easily be done by using PING as shown below:
    • 
    • As shown from the above, the response is always <1ms and we don't have any timeouts which is what SharePoint loves :).
    • Reference: https://technet.microsoft.com/en-gb/library/cc748824.aspx#Anchor_6

• Servers' Specifications

  • You always have to check firstly if your SharePoint servers have the minimum required resources (RAM/Network/CPU), or not. You can verify this using the below links:
    • SharePoint Server 2010 Hardware Requirements: https://technet.microsoft.com/en-us/library/cc262485(v=office.14).aspx
    • SharePoint Server 2013 Hardware Requirements: https://technet.microsoft.com/library/a88d3f72-7ac3-4f08-b302-c4ca0a796268(v=office.16).aspx
    • SharePoint Server 2016 Hardware Requirements: https://technet.microsoft.com/en-us/library/cc262485(v=office.16).aspx
  • Certainly, you have Search Service Application, right? - If so, then please have a look on the Search hardware requirements:
    • SharePoint 2013 Search Hardware Requirements: https://technet.microsoft.com/en-us/library/dn342836.aspx
    • SharePoint 2016 Search Hardware Requirements: https://technet.microsoft.com/en-us/library/dn342836(v=office.16).aspx

• SharePoint Version Supportability

  In fact, this solution is not resolving the performance issue for most of the cases, but it's worth mentioning. Normally, our updates are shipped not only with new features, and fixes, but also with performance enhancements. Hence, it's always recommended to check and ensure that your SharePoint farm version has the least supported version. To get more information about this topic, and to know whether your farm is supported or not, please check this article. If installing the updates won't fix your performance issue, maybe restarting the servers or running the SharePoint Products Configuration wizard, as a part of the SharePoint patches installation, will :).

• SharePoint SQL Databases Outdated Statistics

  • Issue:
    • When database statistics become outdated, SharePoint Server installations may experience one or more of the following symptoms:
      • Slow load times and decreased performance that may generate an HTTP 500 error when you open a site page.
      • Slower performance that generates error messages such as the following:

        Service unavailable
        Unknown SQL Exception 53

        Server Error in '/' Application Runtime Error

      • Performing search crawls causes unexpected SQL Server performance, database locking, and blocking.
      • Long-running timer jobs such as the "Microsoft SharePoint Foundation Usage Data Processing" jobs that take progressively longer to finish during each iteration.
      • The inability to open a SharePoint site, and an error message that resembles the following:

        Unexpected System.Web.HttpException: Request timed out

      • Site rendering time-outs when you load navigation, and the following error message:

        PortalSiteMapProvider was unable to fetch children for node

      • High CPU usage on the server that's running SQL Server when it processes SharePoint queries.
  • Solution:
    • Although SharePoint 2010 and going forward has timer job for updating Index statistics; however, in some circumstances, we noticed that the timer job is not always successfully updating the statistics. For more information about this, check this article https://blogs.msdn.microsoft.com/erica/2012/10/31/sharepoint-and-database-statistics-why-are-they-out-of-date-and-what-to-do-about-it/
    • Therefore, we recommend updating the SharePoint content database statistics daily using the FULLSCAN option from SQL Server. Although SharePoint does have a timer job to update statistics by calling proc_updatestatistics, we strongly recommend implementing a scheduled maintenance plan from SQL Server to ensure database statistics are reliably updated on a daily basis.
    • When implementing the SQL Server maintenance plan to update the statistics on your SharePoint databases, it is not required to disable the job from SharePoint. However, because these maintenance tasks perform similar functions from both locations, it is permissible to disable the timer job from the SharePoint farm.
    • The timer job which updates the databases statistics relies beneath the health analyzer rule Databases used by SharePoint have outdated index statistics.
  • Reference: https://support.microsoft.com/en-us/help/3103194/outdated-database-statistics-decrease-sharepoint-server-performance--c

• Certificate Revocation List (CRL)

  All the needed information can be found here https://support.microsoft.com/en-us/help/2625048/site-slowness-because-of-sharepoint-sts-certificate-crl-checking

• Blob Cache

  • In some cases, we noticed that the root cause of the performance issue is the blob cache being enabled! shouldn't enabling blob cache resolve performance issues?!
  • In fact, if you don't follow the blob cache best practices, then you may experience a performance issue.
    • Make sure that you put the BLOB cache on a drive that has sufficient disk space available in which to store the cache.
    • select a drive that will be used by as few processes as possible so that the BLOB cache process does not encounter conflicts when it tries to access the drive. If too many processes compete for disk access on the drive where the BLOB cache is located, BLOB cache performance and other processes will be adversely affected. For instance, don't keep the blob cache in the same drive as the SharePoint trace logs.
    • If you plan to use the BLOB cache in a scenario with heavy cache use, such as serving videos in a high traffic environment, and if you will use ULS logging, consider placing the BLOB cache on a separate physical drive from the ULS log — not on a separate partition. Storing the BLOB cache and the ULS log on the same drive can result in poor server performance. If you place the BLOB cache and the ULS log on the same physical drive, make sure that you closely monitor the disk queue length for any performance effect.
    • If you have anti-virus already installed on your SharePoint servers, ensure excluding the blob cache directory and all its subdirectories from the AV scanning.
    • When you decide how large to make the BLOB cache, you must consider the number and size of the files to determine the total size of the data to be stored in the cache. By default, the BLOB cache is set to 10 gigabytes (GB). Allow at least 20 percent more space on the drive than the size of the cache. For example, if you have 100 GB of content, set the size of the cache to 120 GB on a drive that has at least 150 GB of space. If the BLOB cache is too small, serving files to users slows, reducing the performance of your site.
    • If you plan to use the image renditions feature on your site, you should account for each rendition being a separate BLOB in the cache. For example, if you plan to have five renditions per image, then you must allocate significantly more than the general estimate of 20 percent more space on the drive.
  • Reference: https://technet.microsoft.com/en-us/library/ee424404.aspx#Section1c

Not recommended SharePoint Data Structure/Configurations:

• Verbose/VerboseEx Logging

  • Issue:
    • Normally, you should not keep SharePoint Verbose logging, or VerboseEx logging enabled for long time, rather only to investigate a certain issue and to set it back to default right after. Whereas, Verbose/VerboseEx logging can quickly use drive space and affect drive and server performance.
    • You can verify the SharePoint log level from SharePoint Central Administration and via Monitoring >> Configure Diagnostic Logging. If the log categories are not in bold, then the log level is set to default, as shown below:
    • 
  • Solution
    • Just set the SharePoint log level back to default. Simply, you can do this either via Central Administration or by running the below command via SharePoint management shell.

    • Clear-SPLogLevel

    • It's worth mentioning that if you have a custom logging category from your dev team, ensure that the default log level is not Verbose :)

• Debug=true and Web Application Batch Compilation

  • Issue:
    • ASP.NET supports compiling applications in a special debug mode that facilitates developer troubleshooting. Debug mode causes ASP.NET to compile applications with extra information that enables a debugger to closely monitor and control the execution of an application.
    • Applications that are compiled in debug mode execute as expected. However, the performance of the application is affected. To avoid the effect on performance, it is a good idea to enable debugging only when a developer is doing interactive troubleshooting. By default, debugging is disabled.
    • How to verify if the web application debugging is enabled or not?
      • Open IIS Manager, and allocate your SharePoint site.
      • Access the site web.config file.
      • Check the value of the attributes debug and batch which should have  false value in order to avoid any performance issue. In other words, the web.config should contain the below tag.

      • <compilation batch="false" debug="false">

  • Solution:
    • Disable the SharePoint web application debugging by having the below tag in the web application web.config

    • <compilation batch="false" debug="false">

    • More information can be found here https://blogs.technet.microsoft.com/stefan_gossner/2012/07/20/sharepoint-memory-leak-caused-by-incorrect-web-config-settings/

• Audit Logs

  • Issue
    • I saw cases where customers enabled SharePoint Audit Logging without configuring automatic trimming which will make the Audit Logs increase dramatically without being monitored and will certainly eventually cause a site performance issue.
    • We recommend that you only select Opening or downloading documents, viewing items in lists, or viewing item properties for SharePoint sites when absolutely needed. This option is likely to generate a large number of events that will potentially degrading the performance and other aspects of the site collection.
    • How to determine the current number of Audit Log records that you have? - This can be achieved by any of the below methods
      • Running the below commands via SharePoint Management Shell

        • $site = Get-SPSite SiteURL
          

        • $site.Audit.GetEntries().count

      • Via SQL server and using SQL Server Management Studio, for your content database check the properties of the table "dbo.AuditData", as shown below:
      • 
    • If your site audit log records exceeds 5 Million records, then your users may experience performance issue when accessing this site.
    • Reference: https://support.office.com/en-us/article/Configure-audit-settings-for-a-site-collection-f5a346d0-ee0f-4412-a5e6-d9b5abaa1012
  • Solution:
    • If you are in this situation, you will need to enable Audit Log Trimming from the site collection settings page, and specify number of days to retain these logs. Hereafter, you will need to run the timer job "Audit Log Trimming" for your web application.
    • As the timer job "Audit Log Trimming" is by default scheduled to run monthly, I would recommend also scheduling it to run Daily or Weekly. Whereas, this can be achived as follows:
      • Access SharePoint Central Administration Site >> Monitoring >> Review Job Definitions.
      • Click on the timer job "Audit Log Trimming" for your web application.
      • 
      • Change its schedule as per your desire, or click Run Now button.

• Large Lists/Libraries

  • Issue
    • Do you have large lists/libraries in your farm? By large lists/libraries, we mean a list/library that has about 5,000 items or more. Yes, in our SharePoint boundaries and limits article we state that a single SharePoint list/library can hold up to 30 Million items/documents; however, this is a conditional supported number!
    • If you are really going to have a list/library that has more than 5,000 items/documents, then you have to structure your list/library in order not to query the SQL server with more than 5,000 items/documents, unless you keep the default list/library throttling values.
    • So, why only 5,000 although the list/library can support up to 30 Million?
      • To minimize database contention SQL Server, the back-end database for SharePoint, often uses row-level locking as a strategy to ensure accurate updates without adversely impacting other users who are accessing other rows. However, if a read or write database operation, such as a query, causes more than 5,000 rows to be locked at once, then it's more efficient for SQL Server to temporarily lock the entire table until the database operation is completed.

Note: The actual number is not always 5,000, and can vary depending on your site, the amount of activity in the database, and your site's configuration.

• • • • When the whole table is locked, it prevents other users from accessing the table. If this happens too often, then users will experience a degradation of system performance. Therefore, throttling and limits are essential to help minimize the impact of resource-intensive database operations and balance the needs of all users.
      • Event worse, this won't only impact this site collection, but all the site collections contained in the same content databases, as all the site collections inside a content database share the same items/documents tables.

Reference: https://support.office.com/en-us/article/Manage-large-lists-and-libraries-in-SharePoint-b8588dae-9387-48c2-9248-c24122f07c59

• • Solution
    • The solution here, in order to avoid encountering a performance issue due to large lists/libraries, is to ensure creating the items/documents inside folders; whereas, each folder should contain at most 5,000 items/documents. By doing so, we will be sure that SharePoint will never sent a SQL query with 5,000+ items/documents results.
    • Furthermore, ensure not using or creating any view that will retrieve more than 5,000 items/documents.
    • How to detect the large lists/libraries in your site?
      • This can be achieved via Site Settings >> Site Contents
      • 
      • Under each list/library, you will see the number of items, as highlighted below:
      • 
      • If for any reason you would like to get all the lists/libraries in yours farm which exceed 5,000 items, there are plenty of scripts online performing so...

• List/Library Throttling

  • Issue
    • • Whether you disabled the list view throttling intentionally or unintentionally (your dev team did), you have to know that SharePoint throttling is enabled by default for a valid reason and to save your back when a user is heavily using a list/library which eventually will impact all the users that are using this site collection and all the site collections included in the same content database.
      • As demonstrated in the Large Lists/Libraries section, if a read or write database operation, such as a query, causes more than 5,000 rows to be locked at once, then it's more efficient for SQL Server to temporarily lock the entire table until the database operation is completed, and that's why we have the view throttling configuring with 5,000 items.
      • Is my web application configured with list view throttling equals 5,000? - This can be known via Central Administration site >> Application Management >> Manage Web Applications, then select your web application and from the Ribbon choose General Settings >> Resource Throttling.
      • 
      • In fact, the web application throttling is not the only place to configure throttling; whereas, lists/libraries can be configured using PowerShell. So, how to determine the lists/libraries that have throttling disabled?!
      • Using the below code should do the trick for you.

    Add-PSSnapin Microsoft.SharePoint.PowerShell -ErrorAction SilentlyContinue
    
    $webApplicationURL = Read-Host "Enter Web Application URL"
    $reportPath = Read-Host "Enter full Path URL in the format (d:\temp\throttledLists.txt)"
    $webapplication = Get-SPWebApplication $webApplicationURL
    $sites = $webapplication.Sites
    Write-Output "List URL" | Out-File $reportPath
    
    foreach($site in $sites)
    {
        $webs = $site.AllWebs
        foreach($web in $webs)
        {
           $lists = $web.Lists
           foreach($list in $lists)
            {
              if(-not $list.EnableThrottling)
               {
                 $listUrl = "$($web.Url)$($list.DefaultViewUrl)"
                 Write-Host $listUrl -ForegroundColor Red
                  Write-Output $listUrl | Out-File $reportPath -append 
               }
            }
           $web.Dispose()
        }
        $site.Dispose()
    }
    Write-Host "Script is completed successfully!" -ForegroundColor Green 

  • Solutio
    • Just keep the web application throttling values as they are :).

• Workflow Tasks/History Lists

  • Issue:
    • SharePoint workflow tasks list, and history list can grow indefinitely which is going to cause a performance issue in your site. Here it comes our default enabled timer job "Workflow Auto Cleanup" which runs to delete the old workflow tasks, and workflow instances.
    • However, if you disable this timer job, or increase your workflow retention from 60 days, then you will have to keep an eye on your tasks list.
    • Workflow Tasks list, and History list are normal lists at the end; so, they should not contain much items there.
  • Solution:
    • Do not disable the timer job "Workflow Auto Cleanup". If you have to disable it, keep an eye on your workflow Tasks list, and ensure that you have an archival system on it.
    • Reference: https://technet.microsoft.com/en-us/library/ee662522(v=office.14).aspx
    • Ensure using a separate workflow Tasks list, and History list for each workflow association.
    • More information can be found here https://blogs.msdn.microsoft.com/ahmedamin/2017/10/01/sharepoint-workflow-auto-cleanup/

• Unique Granted Permissions(ACL)

  • Issue:
    • We have two configurations/limits here, as follows:

      Security scope

      50,000 per list

      Threshold

      The maximum number of unique security scopes set for a list cannot exceed 50,000. For most farms, we recommend that you consider lowering this limit to 5,000 unique scopes. For large lists, consider using a design that uses as few unique permissions as possible. When the number of unique security scopes for a list exceeds the value of the list view threshold (set by default at 5,000 list items), additional SQL Server round trips take place when the list is viewed, which can adversely affect list view performance. A scope is the security boundary for a securable object and any of its children that do not have a separate security boundary defined. A scope contains an Access Control List (ACL), but unlike NTFS ACLs, a scope can include security principals that are specific to SharePoint Server 2013. The members of an ACL for a scope can include Windows users, user accounts other than Windows users (such as forms-based accounts), Active Directory groups, or SharePoint groups.

      Security principal: size of the Security Scope

      5,000 per Access Control List (ACL)

      Supported

      The size of the scope affects the data that is used for a security check calculation. This calculation occurs every time that the scope changes. There is no hard limit, but the bigger the scope, the longer the calculation takes.

    • Hence, in order to identify the security scopes which are big already, you can run the below SQL query. However, as we in MS don't recommend running SQL queries, and i personally hate doing so, i would recommend taking a backup from your content database, restore it on another SQL server, and then run the below SQL query.

    • select COUNT(ra.PrincipalId) as [Count],p.ScopeUrl
      from RoleAssignment ra with(nolock)
      join Perms p with(nolock) on p.SiteId = ra.SiteId and p.ScopeId = ra.ScopeId
      group by p.ScopeUrl
      order by [Count] desc

    • Running the above query should display results, as shown below:
    • 
  • Solution:
    • The solution here is easy, but hard to be implemented.
      • Ensure not having many scope URLs. In other words, don't make many items have unique scopes. For instance, don't make each single item in a list has a unique scope. As doing so will eventually increase the number of unique scopes, and will eventually cause a severe performance issue.
      • Try not granting permissions to individual users, rather to groups. Even better, use AD groups for granting permissions instead of SharePoint groups which may also contains so many users that will impact your site performance.
      • More information can be found here on Bugra's article, although it's for SharePoint 2007, but the concept is similar.

• Custom Code

  Information about coding best practices can be found here https://msdn.microsoft.com/en-us/library/bb687949(v=office.12).aspx

xx/xx/xxxx xx:xx:xx.xx w3wp.exe 0x3394 SharePoint Foundation Runtime 6616 Critical Requested registry access is not allowed. 
xx/xx/xxxx xx:xx:xx.xx w3wp.exe 0x3394 SharePoint Foundation General aat87 Monitorable An unexpected error has occurred.

Log Name:      Application
Source:        Microsoft-SharePoint Products-SharePoint Foundation
Date:          xx/xx/xxxx xx:xx:xx
Event ID:      6616
Task Category: Runtime
Level:         Error
Keywords:      
User:          NT AUTHORITY\IUSR
Computer:      ahmed.amin.com
Description:
Requested registry access is not allowed.

Cause

Solution

1. Installed the missing SharePoint updates on all the farm servers.
2. Restarted all the SharePoint servers after the updates were installed.
3. Ran the SharePoint Products Configuration wizard starting with the Central Administration server, and sequentially on all the farm servers.

N.B., ensure that you perform the above steps using a farm administrator who is a local administrator on all the SharePoint servers in the farm.

psconfig.exe -cmd secureresources

<clientAppDisplayName>
You can't access this application
<clientAppDisplayName> needs permission to access resources in your organization that only an admin can grant. Please ask an admin to grant permission to this app before you can use it.

Cause

• Your SPA needs permissions to access the SharePoint data, or change it. Hence, you, as a developer/admin, have to grant it the needed permissions. Some of these permissions Requires Admin as shown below:

• You granted your application the needed permissions, the application is running, and a user is about to use it for the first time. If the application has not already acquired an access or refresh token, the application needs to go to Azure AD’s authorization endpoint to obtain an authorization code that can be used to acquire a new access and refresh token.
• If the user is not already authenticated, they’ll be asked to sign in to Azure AD.
• After the user has signed in, Azure AD will determine if the user needs to be shown a consent page. This determination is based on whether the user (or their organization’s administrator) has already granted the application consent. If consent has not already been granted, Azure AD will prompt the user for consent and will display the required permissions it needs to function. The set of permissions that is displayed in the consent dialog are the same as what was selected in the Delegated Permissions in the Azure portal.
• After the user grants consent, an authorization code is returned to your application, which can be redeemed to acquire an access token and refresh token.
• For delegated permission that requires admin, the consent have to be granted by the global tenant Administrator.

Solution

• Access the URL https://aad.portal.azure.com/
• From the left navigation click on More Services >> App registrations
• Select your App, and then click on Required Permissions.
• Click on Add button, and then Select an API.
• Select Office 365 SharePoint Online (Microsoft.SharePoint), and click on Select button.

• Tick the suitable permission(s) for your application under the category "Delegated Permissions", and save your changes.

N.B., if you select any permission that "Requires Admin", the below step has to be done by the tenant global admin. In addition, it has to be done by the administrator in all cases to be applied on all users.

• Click Grant Permissions button.

• Click Yes.

New-OfficeWebAppsFarm : The Office Online service failed to start. See the Windows Application Event Log for more details. At line:1 char:1 + New-OfficeWebAppsFarm -InternalURL "https://oos.ahmedamin.com" -Exter ... + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : OperationStopped: (:) [New-OfficeWebAppsFarm], ApplicationFailedException + FullyQualifiedErrorId : AgentManagerStartFailed,Microsoft.Office.Web.Apps.Administration.NewFarmCommand

Log Name: Application
Source: WACSM
Date: 21.09.2017 10:39:55
Event ID: 0
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: server
Description:
Service cannot be started. System.TypeInitializationException: The type initializer for 'Microsoft.Office.Web.Common.Log' threw an exception. ---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at System.String.ReplaceInternal(String oldValue, String newValue)
at System.String.Replace(String oldValue, String newValue)
at Microsoft.Office.Web.Common.Log.GetQuotedEscapedStringOrNull(Object input)
at Microsoft.Office.Web.Common.Log.GetMessageForExceptionLogging(String verb, Exception originalException, Tuple`2[] info)
at Microsoft.Office.Web.Common.Log.TraceWithoutRetraceOnException(UInt32 tag, String verb, Exception originalException, Tuple`2[] extraInfo)
at Microsoft.Office.Web.Common.Log.WriteEventTag(UInt32 tag, EventId id, Object[] insertionData)
at Microsoft.Office.Web.Common.Log.TraceWithoutRetraceOnException(UInt32 tag, String verb, Exception originalException, Tuple`2[] extraInfo)
at Microsoft.Office.Web.Common.Log.WriteEventTag(UInt32 tag, EventId id, O...

Log Name: Application
Source: .NET Runtime
Date: 21.09.2017 10:39:57
Event ID: 1026
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: server
Description:
Application: Microsoft.Office.Web.AgentManager.exe Framework Version: v4.0.30319 Description: The process was terminated due to an unhandled exception. Exception Info: System.OutOfMemoryException at System.String.ReplaceInternal(System.String, System.String) at System.String.Replace(System.String, System.String) at Microsoft.Office.Web.Common.Log.GetQuotedEscapedStringOrNull(System.Object) at Microsoft.Office.Web.Common.Log.GetMessageForExceptionLogging(System.String, System.Exception, System.Tuple`2<System.String,System.Object>[]) at Microsoft.Office.Web.Common.Log.TraceWithoutRetraceOnException(UInt32, System.String, System.Exception, System.Tuple`2<System.String,System.Object>[]) at Microsoft.Office.Web.Common.Log.WriteEventTag(UInt32, Microsoft.Office.Web.Common.EventId, System.Object[]) at Microsoft.Office.Web.Common.Log.TraceWithoutRetraceOnException(UInt32, System.String, System.Exception, System.Tuple`2<System.String,System.Object>[]) at Microsoft.Office.Web.Common.Log.WriteEventTag(UInt32, Microsoft.Office.Web.Common.EventId, System.Object[]) at Microsoft.Office.Web.Common.Log.TraceWithoutRetraceOnException(UInt32, System.String, System.Exception, System.Tuple`2<System.String,System.Object>[]) at Microsoft.Office.Web.Common.Log.WriteEventTag(UInt32, Microsoft.Office.Web.Common.EventId, System.Object[]) at Microsoft.Office.Web.Common.Log.TraceWithoutRetraceOnException(UInt32, System.String, System.Exception, System.Tuple`2<System.String,System.Object>[]) at Microsoft.Office.Web.Common.Log.WriteEventTag(UInt32, Microsoft.Office.Web.Common.EventId, System.Object[]) at Microsoft.Office.Web.Common.Log.TraceWithoutRetraceOnException(UInt32, System.String, System.Exception, System.Tuple`2<System.String,System.Object>[]) at Microsoft.Office.Web.Common.Log.WriteEventTag(UInt32, Microsoft.Office.Web.Common.EventId, System.Object[]) at Microsoft.Office.Web.Common.Log.TraceWithoutRetraceOnException(UInt32, System.String, System.Exception, System.Tuple`2<System.String,System.Object>[]) at Microsoft.Office.Web.Common.Log.WriteEventTag(UInt32, Microsoft.Office.Web.Common.EventId, System.Object[]) at Microsoft.Office.Web.Common.Log.TraceWithoutRetraceOnException(UInt32, System.String, System.Exception, System.Tuple`2<System.String,System.Object>[]) at Microsoft.Office.Web.Common.Log.WriteEventTag(UInt32, Microsoft.Office.Web.Common.EventId, System.Object[]) at Microsoft.Office.Web.Common.Log.TraceWithoutRetraceOnException(UInt32, System.String, System.Exception, System.Tuple`2<System.String,System.Object>[]) at Microsoft.Office.Web.Common.Log.WriteEventTag(UInt32, Microsoft.Office.Web.Common.EventId, System.Object[]) at Microsoft.Office.Web.Common.Log.TraceWithoutRetraceOnException(UInt32, System.String, System.Exception, System.Tuple`2<System.String,System.Object>[]) at Microsoft.Office.Web.Common.Log.WriteEventTag(UInt32, Microsoft.Office.Web.Common.EventId, System.Object[]) at Microsoft.Office.Web.Common.Log.TraceWithoutRetraceOnException(UInt32, System.String, System.Exception, System.Tuple`2<System.String,System.Object>[]) at Microsoft.Office.Web.Common.Log.WriteEventTag(UInt32, Microsoft.Office.Web.Common.EventId, System.Object[]) at Microsoft.Office.Web.Common.Log.TraceWithoutRetraceOnException(UInt32, System.String, System.Exception, System.Tuple`2<System.String,System.Object>[]) at Microsoft.Office.Web.Common.Log.WriteEventTag(UInt32, Microsoft.Office.Web.Common.EventId, System.Object[]) at Microsoft.Office.Web.Common.Log.TraceWithoutRetraceOnException(UInt32, System.String, System.Exception, System.Tuple`2<System.String,System.Object>[]) at Microsoft.Office.Web.Common.Log.WriteEventTag(UInt32, Microsoft.Office.Web.Common.EventId, System.Object[]) at Microsoft.Office.Web.Common.Log.TraceWithoutRetraceOnException(UInt32, System.String, System.Exception, System.Tuple`2<System.String,System.Object>[]) at Microsoft.Office.Web.Common.Log.WriteEventTag(UInt32, Microsoft.Office.Web.Common.EventId, System.Object[]) at Microsoft.Office.Web.Common.Log.TraceWithoutRetraceOnException(UInt32, System.String, System.Exception, System.Tuple`2<System.String,System.Object>[]) at Microsoft.Office.Web.Common.Log.WriteEventTag(UInt32, Microsoft.Office.Web.Common.EventId, System.Object[]) at Microsoft.Office.Web.Common.Log.TraceWithoutRetraceOnException(UInt32, System.String, System.Exception, System.Tuple`2<System.String,System.Object>[]) at Microsoft.Office.Web.Common.Log.WriteEventTag(UInt32, Microsoft.Office.Web.Common.EventId, System.Object[]) at Microsoft.Office.Web.Common.Log.TraceWithoutRetraceOnException(UInt32, System.String, System.Exception, System.Tuple`2<System.String,System.Object>[]) at Microsoft.Office.Web.Common.Log.WriteEventTag(UInt32, Microsoft.Office.Web.Common.EventId, System.Object[]) at Microsoft.Office.Web.Common.Log.TraceWithoutRetraceOnException(UInt32, System.String, System.Exception, System.Tuple`2<System.String,System.Object>[]) at Microsoft.Office.Web.Common.Log.WriteEventTag(UInt32, Microsoft.Office.Web.Common.EventId, System.Object[]) at Microsoft.Office.Web.Common.Log.TraceWithoutRetraceOnException(UInt32, System.String, System.Exception, System.Tuple`2<System.String,System.Object>[]) at Microsoft.Office.Web.Common.Log.WriteEventTag(UInt32, Microsoft.Office.Web.Common.EventId, System.Object[]) at Microsoft.Office.Web.Common.Log.TraceWithoutRetraceOnException(UInt32, System.String, System.Exception, System.Tuple`2<System.String,System.Object>[]) at Microsoft.Office.Web.Common.Log.WriteEventTag(UInt32, Microsoft.Office.Web.Common.EventId, System.Object[]) at Microsoft.Office.Web.Common.Log.TraceWithoutRetraceOnException(UInt32, System.String, System.Exception, System.Tuple`2<System.String,System.Object>[]) at Microsoft.Office.Web.Common.Log.WriteEventTag(UInt32, Microsoft.Office.Web.Common.EventId, System.Object[]) at Microsoft.Office.Web.Common.Log.TraceWithoutRetraceOnException(UInt32, System.String, System.Exception, System.Tuple`2<System.String,System.Object>[]) at Microsoft.Office.Web.Common.Log.WriteEventTag(UInt32, Microsoft.Office.Web.Common.EventId, System.Object[]) at Microsoft.Office.Web.Common.Log.TraceWithoutRetraceOnException(UInt32, System.String, System.Exception, System.Tuple`2<System.String,System.Object>[]) at Microsoft.Office.Web.Common.Log.WriteEventTag(UInt32, Microsoft.Office.Web.Common.EventId, System.Object[]) at Microsoft.Office.Web.Common.Log.TraceWithoutRetraceOnException(UInt32, System.String, System.Exception, System.Tuple`2<System.String,System.Object>[]) at Microsoft.Office.Web.Common.Log.WriteEventTag(UInt32, Microsoft.Office.Web.Common.EventId, System.Object[]) at Microsoft.Office.Web.Common.Log.TraceWithoutRetraceOnException(UInt32, System.String, System.Exception, System.Tuple`2<System.String,System.Object>[]) at Microsoft.Office.Web.Common.Log.WriteEventTag(UInt32, Microsoft.Office.Web.Common.EventId, System.Object[]) at Microsoft.Office.Web.Common.Log.TraceWithoutRetraceOnException(UInt32, System.String, System.Exception, System.Tuple`2<System.String,System.Object>[]) at Microsoft.Office.Web.Common.Log.WriteEventTag(UInt32, Microsoft.Office.Web.Common.EventId, System.Object[]) at Microsoft.Office.Web.Common.Log.TraceWithoutRetraceOnException(UInt32, System.String, System.Exception, System.Tuple`2<System.String,System.Object>[]) at Microsoft.Office.Web.Common.Log.WriteEventTag(UInt32, Microsoft.Office.Web.Common.EventId, System.Object[]) at Microsoft.Office.Web.Common.Log.TraceWithoutRetraceOnException(UInt32, System.String, System.Exception, System.Tuple`2<System.String,System.Object>[]) at Microsoft.Office.Web.Common.Log.WriteEventTag(UInt32, Microsoft.Office.Web.Common.EventId, System.Object[]) at Microsoft.Office.Web.Common.Log.TraceWithoutRetraceOnException(UInt32, System.String, System.Exception, System.Tuple`2<System.String,System.Object>[]) at Microsoft.Office.Web.Common.Log.WriteEventTag(UInt32, Microsoft.Office.Web.Common.EventId, System.Object[]) at Microsoft.Office.Web.Common.Log.TraceTag(UInt32, Microsoft.Office.Web.Common.LogCategory, Level, System.String, System.Object[]) at Microsoft.Office.Web.Common.Log..cctor() Exception Info: System.TypeInitializationException at Microsoft.Office.Web.Common.Log.WriteEventTag(UInt32, Microsoft.Office.Web.Common.EventId, System.Object[]) at Microsoft.Office.Web.Environment.WacServer.AgentManager.AgentManagerApplication.CurrentDomain_UnhandledException(System.Object, System.UnhandledExceptionEventArgs)

Log Name: Application
Source: Application Error
Date: 21.09.2017 10:39:59
Event ID: 1000
Task Category: (100)
Level: Error
Keywords: Classic
User: N/A
Computer: server
Description:
Faulting application name: Microsoft.Office.Web.AgentManager.exe, version: 16.0.7329.1023, time stamp: 0x58145a49
Faulting module name: KERNELBASE.dll, version: 10.0.14393.1715, time stamp: 0x59b0d106
Exception code: 0xe0434352
Fault offset: 0x0000000000033c58
Faulting process id: 0x2f0
Faulting application start time: 0x01d332b51759f9e6
Faulting application path: C:\Program Files\Microsoft Office Web Apps\AgentManager\Microsoft.Office.Web.AgentManager.exe
Faulting module path: C:\Windows\System32\KERNELBASE.dll
Report Id: a6a0f8c7-52d1-4be5-aae9-ae4d627a0d4d
Faulting package full name: 
Faulting package-relative application ID:

Cause

Solution

• Ran the below command to get domain\spadmin accounts that were existent on SharePoint.
  • COMMAND

    Get-SPManagedAccount "domain\spadmin" | select sid

  • RESULT Sid --- S-1-5-21-3855947664-3092534411-819001659-1127 S-1-5-21-3855947664-3092534411-819001659-2514

• Hence, we ran the below commands to delete the old duplicate account "domain\spadmin"
  • COMMAND

    $account = Get-SPManagedAccount | ?{$_.sid -eq "S-1-5-21-3855947664-3092534411-819001659-1127"}
    
    $account.Delete()

  • RESULT

Exception calling "Delete" with "0" argument(s): "The account domain\spadmin is still being used by these components:

Microsoft SharePoint Foundation Sandboxed Code Service

Security Token Service Application

Application Discovery and Load Balancer Service Application."

At line:1 char:1 + $account.Delete() + ~~~~~~~~~~~~~~~~~     + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException     + FullyQualifiedErrorId : InvalidOperationException

• Changed the SharePoint farm account by running the below command via SharePoint management shell.

  stsadm.exe -o updatefarmcredentials -userlogin "domain\spadmin" -password "Password"

  N.B., changed the above highlighted text with the farm account password

• Change the SharePoint Service Applications Application Pools:
  • We ran the below command in order to list all the SharePoint Service Applications Application pools
    • COMMAND

      Get-SPServiceApplicationPool | format-list

    • RESULT
      • Name                        : SecurityTokenServiceApplicationPool ProcessAccount              : S-1-5-21-3855947664-3092534411-819001659-1127 TypeName                    : Microsoft.SharePoint.Administration.SPIisWebServiceApplicationPool DisplayName                 : SecurityTokenServiceApplicationPool Id                          : ea6fdac5-f1d5-4284-8fc4-b70d633fac90 Status                      : Online Parent                      : SPIisWebServiceSettings Name=SharePoint Web Services Version                     : 2852 Properties                  : {} Farm                        : SPFarm Name=Sharepoint_Config UpgradedPersistedProperties : {} ProcessAccountName          :
      • Name                        : SharePoint Hosted Services ProcessAccount              : S-1-5-21-3855947664-3092534411-819001659-1128 TypeName                    : Microsoft.SharePoint.Administration.SPIisWebServiceApplicationPool DisplayName                 : SharePoint Hosted Services Id                          : 6183b7e5-8939-474b-bbed-40f04a8c925a Status                      : Online Parent                      : SPIisWebServiceSettings Name=SharePoint Web Services Version                     : 9772 Properties                  : {} Farm                        : SPFarm Name=Sharepoint_Config UpgradedPersistedProperties : {} ProcessAccountName          :
      • Name                        : SharePoint Search Application Pool ProcessAccount              : S-1-5-21-3855947664-3092534411-819001659-1128 TypeName                    : Microsoft.SharePoint.Administration.SPIisWebServiceApplicationPool DisplayName                 : SharePoint Search Application Pool Id                          : ad785e4f-d5f3-4ec1-81c2-9ef6850381a9 Status                      : Online Parent                      : SPIisWebServiceSettings Name=SharePoint Web Services Version                     : 10236 Properties                  : {} Farm                        : SPFarm Name=Sharepoint_Config UpgradedPersistedProperties : {} ProcessAccountName          :
      • Name                        : SharePoint Web Services System ProcessAccount              : S-1-5-21-3855947664-3092534411-819001659-1127 TypeName                    : Microsoft.SharePoint.Administration.SPIisWebServiceApplicationPool DisplayName                 : SharePoint Web Services System Id                          : a4007afd-aa3b-4a96-95be-7721cbf9357e Status                      : Online Parent                      : SPIisWebServiceSettings Name=SharePoint Web Services Version                     : 2883 Properties                  : {} Farm                        : SPFarm Name=Sharepoint_Config UpgradedPersistedProperties : {} ProcessAccountName          :
  • As we are interested in the SID "S-1-5-21-3855947664-3092534411-819001659-1127", we ran the below commands via SharePoint management shell in order to replace the old service accounts with the new ones.

    Set-SPServiceApplicationPool "SecurityTokenServiceApplicationPool" -Account "domain\spadmin"
    
    Set-SPServiceApplicationPool "SharePoint Web Services System" -Account "domain\spadmin"

• Change Sandboxed Code Service Service Account: we ran the below command via SharePoint management shell on the SharePoint server:

  Get-SPServiceInstance | %{if($_.TypeName -eq "Microsoft SharePoint Foundation Sandboxed Code Service") {$_.Service.ProcessIdentity.Username ="domain\spadmin"; $_.Service.ProcessIdentity.Update(); $_.Service.ProcessIdentity.Deploy();}}

$account = Get-SPManagedAccount | ?{$_.sid -eq "S-1-5-21-3855947664-3092534411-819001659-1127"}
$account.Delete()

Item has already been added. Key in dictionary: 'domain\spfarm'  Key being added: 'domain\spfarm'

• Ran the below command to get spfarm accounts that were existent on SharePoint.
  • COMMAND

    Get-SPManagedAccount "domain\spfarm" | select sid

  • RESULT Sid --- S-1-5-21-3855947664-3092534411-819001659-1128 S-1-5-21-3855947664-3092534411-819001659-2516

• Hence, we found out that the old account SID is "S-1-5-21-3855947664-3092534411-819001659-1128"; Therefore, we ran the below commands to delete the managed account:
  • COMMAND

    $account = Get-SPManagedAccount | ?{$_.sid -eq "S-1-5-21-3855947664-3092534411-819001659-1128"}
    
    $account.Delete()

  • RESULT Exception calling "Delete" with "0" argument(s): "The account domain\spfarm is still being used by these components: Microsoft SharePoint Foundation Application Pool: mysite.com User Profile Synchronization Service Microsoft SharePoint Foundation Tracing Distributed Cache Microsoft SharePoint Foundation Application Pool: SharePoint – 80 SharePoint Server Search Search Host Controller Service Secure Store Service PowerPoint Conversion Service Application PerformancePoint Service Excel Services Application Subscription Settings Service Business Data Connectivity Service Work Management Service Application Visio Graphics Service Managed Metadata Service Word Automation Services User Profile Service Application Search Administration Web Service for Search Service Application Search Service Application." At line:1 char:1 + $account.Delete() + ~~~~~~~~~~~~~~~~~ + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException + FullyQualifiedErrorId : InvalidOperationException

• Changing the Service Applications Application Pools was OK, and done using the same above commands Set-SPServiceApplicationPool.

• Change User Profile Service Account: we ran the below command via SharePoint management shell

  Get-SPServiceInstance | %{if($_.TypeName -eq "User Profile Synchronization Service") {$_.Service.ProcessIdentity.Username ="domain\spfarm"; $_.Service.ProcessIdentity.Update(); $_.Service.ProcessIdentity.Deploy();}}

• Change the Distributed Cache Service Account: , we ran the below commands:

  $farm = Get-SPFarm
  
  $cacheService = $farm.Services | where {$_.Name -eq "AppFabricCachingService"}
  
  $accnt = Get-SPManagedAccount | ?{$_.sid -eq "S-1-5-21-3855947664-3092534411-819001659-2516"}
  
  $cacheService.ProcessIdentity.CurrentIdentityType = "SpecificUser"
  
  $cacheService.ProcessIdentity.ManagedAccount = $accnt
  
  $cacheService.ProcessIdentity.Update()
  
  $cacheService.ProcessIdentity.Deploy()

N.B., the above highlighted SID is the new account's SID

• Change the Tracing Service Service Account: we ran the below commands via SharePoint management shell.

  $farm = Get-SPFarm
  
  $tracingService = $farm.Services | where {$_.Name -eq "SPTraceV4"}
  
  $managedAccount = Get-SPManagedAccount | ?{$_.sid -eq "S-1-5-21-3855947664-3092534411-819001659-2516"}
  
  $tracingService.ProcessIdentity.CurrentIdentityType = "SpecificUser"
  
  $tracingService.ProcessIdentity.ManagedAccount = $managedAccount
  
  $tracingService.ProcessIdentity.Update()
  
  $tracingService.ProcessIdentity.Deploy()

N.B., the above highlighted SID is the new account's SID

• Change Search Service Account: we ran the below commands via SharePoint management shell to replace the Search service account.

  $procId = (Get-SPEnterpriseSearchService).get_ProcessIdentity()
  
  $procId.CurrentIdentityType = "SpecificUser"
  
  $myManagedAcount = Get-SPManagedAccount | ?{$_.sid -eq "S-1-5-21-3855947664-3092534411-819001659-2516"}
  
  $procId.ManagedAccount = $myManagedAcount
  
  $procId.Update()
  

N.B., the above highlighted SID is the new account's SID

• Change the Search Host Controller Service Service Account: we ran the below commands via SharePoint management shell.

  Get-SPServiceInstance | %{if($_.TypeName -eq "Search Host Controller Service") {$_.Service.ProcessIdentity.Username ="domain\spfarm"; $_.Service.ProcessIdentity.Update(); $_.Service.ProcessIdentity.Deploy();}}

  

• Change the SharePoint Web Applications Application Pools: we ran the below commands via SharePoint management shell:

  $ManagedAccount = Get-SPManagedAccount | ?{$_.sid -eq "S-1-5-21-3855947664-3092534411-819001659-2516"}    
  
  $WebApplication = Get-SPWebApplication | %{$_.ApplicationPool.ManagedAccount = $ManagedAccount; $_.ApplicationPool.Update();$_.ProvisionGlobally();$_.Update()}

  

• Eventually, we deleted the old managed account by running the below commands via SharePoint management shell:

  $account = Get-SPManagedAccount | ?{$_.sid -eq "S-1-5-21-3855947664-3092534411-819001659-1128"}
  
  $account.Delete()

  

ALTER USER [domain\spadmin] WITH LOGIN = [domain\spadmin];

ALTER USER [domain\spfarm] WITH LOGIN = [domain\spfarm];

The permissions granted to user 'domain\user' are insufficient for performing this operation. ---> Microsoft.ReportingServices.Diagnostics.Utilities.AccessDeniedException: The permissions granted to user 'domain\user' are insufficient for performing this operation.

<detail><ErrorCode xmlns="https://www.microsoft.com/sql/reportingservices">rsAccessDenied</ErrorCode><HttpStatus xmlns="https://www.microsoft.com/sql/reportingservices">400</HttpStatus><Message xmlns="https://www.microsoft.com/sql/reportingservices">The permissions granted to user 'domain\user' are insufficient for performing this operation. ---&gt; Microsoft.ReportingServices.Diagnostics.Utilities.AccessDeniedException: The permissions granted to user 'domain\user' are insufficient for performing this operation.</Message><HelpLink xmlns="https://www.microsoft.com/sql/reportingservices">https://go.microsoft.com/fwlink/?LinkId=20476&amp;EvtSrc=Microsoft.ReportingServices.Diagnostics.Utilities.ErrorStrings&amp;EvtID=rsAccessDenied&amp;ProdName=Microsoft%20SQL%20Server%20Reporting%20Services&amp;ProdVer=11.0.3000.0</HelpLink><ProductName xmlns="https://www.microsoft.com/sql/reportingservices">Microsoft SQL Server Reporting Services</ProductName><ProductVersion xmlns="https://www.microsoft.com/sql/reportingservices">11.0.3000.0</ProductVersion><ProductLocaleId xmlns="https://www.microsoft.com/sql/reportingservices">127</ProductLocaleId><OperatingSystem xmlns="https://www.microsoft.com/sql/reportingservices">OsIndependent</OperatingSystem><CountryLocaleId xmlns="https://www.microsoft.com/sql/reportingservices">1033</CountryLocaleId><MoreInformation xmlns="https://www.microsoft.com/sql/reportingservices"><Source>Microsoft.ReportingServices.SharePoint.Server</Source><Message msrs:ErrorCode="rsAccessDenied" msrs:HelpLink="https://go.microsoft.com/fwlink/?LinkId=20476&amp;EvtSrc=Microsoft.ReportingServices.Diagnostics.Utilities.ErrorStrings&amp;EvtID=rsAccessDenied&amp;ProdName=Microsoft%20SQL%20Server%20Reporting%20Services&amp;ProdVer=11.0.3000.0" xmlns:msrs="https://www.microsoft.com/sql/reportingservices">The permissions granted to user 'domain\user' are insufficient for performing this operation.</Message></MoreInformation><Warnings xmlns="https://www.microsoft.com/sql/reportingservices" /></detail>

xx/xx/xxxx xx:xx:xx.xx w3wp.exe (ITSRVAMSPAPP2:0x0D60) 0x26B4 SharePoint Foundation General adyrv High Cannot find site lookup info for request Uri https://sp2013:32843/83dcde14cf16409295a8428c95c3573b/ReportStreaming.svc. ab231a9e-7b93-e018-3466-ecf670f397df

xx/xx/xxxx xx:xx:xx.xx             w3wp.exe (ITSRVAMSPAPP2:0x0D60)     0x26B4 SharePoint Foundation  General 8xfr       Verbose              PermissionMask check failed. asking for 0x00010000, have 0x00000000  ab231a9e-7b93-e018-3466-ecf670f397df

xx/xx/xxxx xx:xx:xx.xx             w3wp.exe (ITSRVAMSPAPP2:0x0D60)     0x26B4 SharePoint Foundation  General aix9j      High              SPRequest.GetSiteFlags: UserPrincipalName=, AppPrincipalName= ,bstrUrl=https://sp2013/sites/BI/PowerPivot Gallery/report.rdlx            ab231a9e-7b93-e018-3466-ecf670f397df

xx/xx/xxxx xx:xx:xx.xx             w3wp.exe (ITSRVAMSPAPP2:0x0D60)     0x26B4 SharePoint Foundation  General ai1wu   Medium              System.UnauthorizedAccessException: Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED)), StackTrace:    at Microsoft.SharePoint.SPSite.get_ReadLocked()     at Microsoft.ReportingServices.SharePoint.Server.SharePointServiceHelper.<>c__DisplayClass4.<GetAdditionalToken>b__3()     at Microsoft.SharePoint.SPSecurity.<>c__DisplayClass5.<RunWithElevatedPrivileges>b__3()     at Microsoft.SharePoint.Utilities.SecurityContext.RunAsProcess(CodeToRunElevated secureCode)     at Microsoft.SharePoint.SPSecurity.RunWithElevatedPrivileges(WaitCallback secureCode, Object param)     at Microsoft.SharePoint.SPSecurity.RunWithElevatedPrivileges(CodeToRunElevated secureCode)     at Microsoft.ReportingServices.SharePoint.Server.Utility.RunElevated(CodeToRunElevated code)     at Microsoft.ReportingServices.SharePoint.Server.SharePointServiceHelper.GetAdditionalToken(String wssUrl, String userName)     at Microsoft.ReportingServices.SharePoint.Server.SharePointServiceHelper.PopulateAdditionalToken(RSService service, String wssUrl)     at Microsoft.ReportingServices.Library.StreamRequestHandler.ExecuteCommand()     at Microsoft.ReportingServices.ServiceRuntime.ReportStreaming.<>c__DisplayClasse.<ExecuteStreamingOperation>b__d()     at Microsoft.ReportingServices.ServiceRuntime.ReportStreaming.ExecuteStreamingOperation[TResult](Func`1 func)     at Microsoft.ReportingServices.ServiceRuntime.StreamBodyWriter.OnWriteBodyContents(XmlDictionaryWriter writer)     at System.ServiceModel.Security.SecurityAppliedMessage.OnWriteMessage(XmlDictionaryWriter writer)     at System.ServiceModel.Channels.BinaryMessageEncoderFactory.BinaryMessageEncoder.WriteMessage(Message message, Stream stream)     at System.ServiceModel.Channels.HttpOutput.WriteStreamedMessage(TimeSpan timeout)     at System.ServiceModel.Channels.HttpOutput.Send(TimeSpan timeout)     at System.ServiceModel.Channels.HttpPipeline.EmptyHttpPipeline.SendReplyCore(Message message, TimeSpan timeout)     at System.ServiceModel.Channels.HttpRequestContext.OnReply(Message message, TimeSpan timeout)     at System.ServiceModel.Channels.RequestContextBase.Reply(Message message, TimeSpan timeout)     at System.ServiceModel.Channels.SecurityChannelListener`1.SecurityRequestContext.OnReply(Message message, TimeSpan timeout)     at System.ServiceModel.Channels.RequestContextBase.Reply(Message message, TimeSpan timeout)     at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.Reply(MessageRpc& rpc)     at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage9(MessageRpc& rpc)     at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage8(MessageRpc& rpc)     at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc)     at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage31(MessageRpc& rpc)     at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)     at System.ServiceModel.Dispatcher.ChannelHandler.DispatchAndReleasePump(RequestContext request, Boolean cleanThread, OperationContext currentOperationContext)     at System.ServiceModel.Dispatcher.ChannelHandler.HandleRequest(RequestContext request, OperationContext currentOperationContext)     at System.ServiceModel.Dispatcher.ChannelHandler.AsyncMessagePump(IAsyncResult result)     at System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)     at System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)     at System.ServiceModel.Channels.SecurityChannelListener`1.ReceiveItemAndVerifySecurityAsyncResult`2.InnerTryReceiveCompletedCallback(IAsyncResult result)     at System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)     at System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)     at System.Runtime.InputQueue`1.AsyncQueueReader.Set(Item item)     at System.Runtime.InputQueue`1.EnqueueAndDispatch(Item item, Boolean canDispatchOnThisThread)     at System.Runtime.InputQueue`1.EnqueueAndDispatch(T item, Action dequeuedCallback, Boolean canDispatchOnThisThread)     at System.ServiceModel.Channels.SingletonChannelAcceptor`3.Enqueue(QueueItemType item, Action dequeuedCallback, Boolean canDispatchOnThisThread)     at System.ServiceModel.Channels.HttpPipeline.EnqueueMessageAsyncResult.CompleteParseAndEnqueue(IAsyncResult result)     at System.ServiceModel.Channels.HttpPipeline.EnqueueMessageAsyncResult.HandleParseIncomingMessage(IAsyncResult result)     at System.Runtime.AsyncResult.SyncContinue(IAsyncResult result)     at System.ServiceModel.Channels.HttpPipeline.EmptyHttpPipeline.BeginProcessInboundRequest(ReplyChannelAcceptor replyChannelAcceptor, Action dequeuedCallback, AsyncCallback callback, Object state)     at System.ServiceModel.Channels.HttpChannelListener`1.HttpContextReceivedAsyncResult`1.ProcessHttpContextAsync()     at System.ServiceModel.Channels.HttpChannelListener`1.BeginHttpContextReceived(HttpRequestContext context, Action acceptorCallback, AsyncCallback callback, Object state)     at System.ServiceModel.Activation.HostedHttpTransportManager.HttpContextReceived(HostedHttpRequestAsyncResult result)     at System.ServiceModel.Activation.HostedHttpRequestAsyncResult.HandleRequest()     at System.ServiceModel.Activation.HostedHttpRequestAsyncResult.BeginRequest()     at System.ServiceModel.Activation.HostedHttpRequestAsyncResult.OnBeginRequest(Object state)     at System.ServiceModel.AspNetPartialTrustHelpers.PartialTrustInvoke(ContextCallback callback, Object state)     at System.ServiceModel.Activation.HostedHttpRequestAsyncResult.OnBeginRequestWithFlow(Object state)     at System.Runtime.IOThreadScheduler.ScheduledOverlapped.IOCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)     at System.Runtime.Fx.IOCompletionThunk.UnhandledExceptionFrame(UInt32 error, UInt32 bytesRead, NativeOverlapped* nativeOverlapped)     at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)              ab231a9e-7b93-e018-3466-ecf670f397df

xx/xx/xxxx xx:xx:xx.xx             w3wp.exe (ITSRVAMSPAPP2:0x0D60)     0x26B4 SQL Server Reporting Services    Report Server Catalog    00000              Verbose              UnauthorizedAccessException in GetAdditionalToken : System.UnauthorizedAccessException: Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))     at Microsoft.SharePoint.SPGlobal.HandleUnauthorizedAccessException(UnauthorizedAccessException ex)     at Microsoft.SharePoint.Library.SPRequest.GetSiteFlags(String bstrUrl)     at Microsoft.SharePoint.SPSite.get_ReadLocked()     at Microsoft.ReportingServices.SharePoint.Server.SharePointServiceHelper.<>c__DisplayClass4.<GetAdditionalToken>b__3()            ab231a9e-7b93-e018-3466-ecf670f397df

09/18/2017 07:18:47.02 w3wp.exe (0x1120) 0x2610 SharePoint Foundation Performance ftq1 Verbose SearchFromGC name = contoso.local. start  7e781a9e-5196-5016-c835-9eeac8346420

09/18/2017 07:18:47.02 w3wp.exe (0x1120) 0x2610 SharePoint Foundation General ardb5 Verbose SearchFromGC query = (|(&(objectCategory=person)(|(anr=contoso*)(SamAccountName=contoso*)))(&(objectCategory=group)(groupType:1.2.840.113556.1.4.803:=2147483648)(|(anr=contoso*)(SamAccountName=contoso*)))) 7e781a9e-5196-5016-c835-9eeac8346420

09/18/2017 07:18:58.32 w3wp.exe (0x1120) 0x2610 SharePoint Foundation Performance ftq3 Verbose SearchFromGC name = contoso.local. Error Message: Access is denied 7e781a9e-5196-5016-c835-9eeac8346420

09/18/2017 07:18:58.32 w3wp.exe (0x1120) 0x2610 SharePoint Foundation General 7fbh Verbose Exception when search "contoso" from domain "contoso.local". Exception: "Access is denied", StackTrace: "   at Microsoft.SharePoint.Win32.SPNtDsApi32.SafeDirectoryServiceHandle.Bind(String domainControllerName, String dnsDomainName)     at Microsoft.SharePoint.Win32.SPNtDsApi32.TranslateUserName(String name, DS_NAME_FORMAT formatOffered, DS_NAME_FORMAT formatDesired)     at Microsoft.SharePoint.WebControls.PeopleEditor.SearchFromGC(SPActiveDirectoryDomain domain, String strFilter, String[] rgstrProp, Int32 nTimeout, Int32 nSizeLimit, SPUserCollection spUsers, ArrayList& rgResults)     at Microsoft.SharePoint.Utilities.SPUserUtility.SearchAgainstAD(String input, Boolean useUpnInResolve, SPActiveDirectoryDomain domainController, SPPrincipalType scopes, SPUserCollection usersContainer, Int32 maxCount, String customQuery, String customFilter, TimeSpan searchTimeout, Boolean& reachMaxCount)". 7e781a9e-5196-5016-c835-9eeac8346420

• You are in the process of migrating your user accounts from domain A to domain B. Whereas, you have two way trust between the two domains.
• You migrate the users but keep the same SIDs. Later, you disable the migrated accounts in the destination domain B.
• When a SharePoint user uses people picker to search for any of the migrated users, the people picker will return the user from domain B although the user is disabled.
• In fact this happens because the people picker is using the Windows API LookupAccountSid to resolve the returned users' SIDs to domain\user. However, this API does not consider if the returned users are disabled or not.
• Hence, with this property HideInactiveProfiles, SharePoint is not going to use this API LookupAccountSid, but will use distinguishedName.
• Normally, if you want to enable this property "HideInactiveProfiles", you have to run the below STSADM command.

stsadm.exe -o setproperty -propertyname "HideInactiveProfiles" -propertyvalue "true"

N.B., the above property and value are case sensitive, and the command works for SharePoint 2013, and SharePoint 2016

stsadm.exe -o getproperty -propertyname "HideInactiveProfiles"

stsadm.exe -o setproperty -propertyname "HideInactiveProfiles" -propertyvalue "false"

xx/xx/xxxx xx:xx:xx.xx w3wp.exe 0x1AA0 SharePoint Foundation Topology fg4x High Ending asynchronous provisioning for service application with name 'BDC Service Application' and type 'Microsoft.SharePoint.BusinessData.SharedService.BdcServiceApplication'.

xx/xx/xxxx xx:xx:xx.xx w3wp.exe 0x1AA0 SharePoint Foundation Timer b9u7 High SPTimerJobAsyncResult '01d23aaf-7b11-4131-87e6-519b860597a9': Timer job has not completed.  Blocking until job completes.

xx/xx/xxxx xx:xx:xx.xx w3wp.exe 0x1AA0 SharePoint Foundation Timer b9u9 High SPTimerJobAsyncResult '01d23aaf-7b11-4131-87e6-519b860597a9': An error occurred while waiting for completion: System.Threading.ThreadAbortException: Thread was being aborted.     at System.Threading.Thread.SleepInternal(Int32 millisecondsTimeout)     at System.Threading.Thread.Sleep(Int32 millisecondsTimeout)     at Microsoft.SharePoint.Administration.SPJobDefinition.WaitForCompletion(TimeSpan timeout)     at Microsoft.SharePoint.Administration.SPTimerJobAsyncResult.WaitForCompletion(Object state)

xx/xx/xxxx xx:xx:xx.xx w3wp.exe 0x1AA0 SharePoint Foundation Runtime tkau Unexpected System.Web.HttpException: Request timed out.

xx/xx/xxxx xx:xx:xx.xx w3wp.exe 0x1AA0 SharePoint Foundation General ajlz0 High Getting Error Message for Exception System.Web.HttpException (0x80004005): Request timed out.

1. If you don't have the service application, you won't be able to start its service instance using the Central Administration site (System Settings >> Manage Services on Server), unless this is a Custom MinRole. Hence, we will start the service instances using PowerShell.
2. If the you don't have the service application, and you start the service instance, you will get a message beside the server in the Central Administration site (System Settings >> Manage Servers in this farm)that this server is not compliant! Beside this, you will find fix hyperlink, don't click it, as we will ignore this compliancy issue. Later, we will create the service application, and then the server will become compliant again.

1. Start the service instance
   • Firstly, check the above mentioned article to know which servers in your farm should run the required service instance.
   • Secondly, you will need to know the Type Name of your service instance of your service application; whereas, below is the type names of the SharePoint service instances:
     • Search Host Controller Service
     • Microsoft SharePoint Insights
     • App Management Service
     • Managed Metadata Web Service
     • Access Services
     • Business Data Connectivity Service
     • Request Management
     • Secure Store Service
     • Claims to Windows Token Service
     • Microsoft SharePoint Foundation Workflow Timer Service
     • PerformancePoint Service
     • Microsoft SharePoint Foundation Sandboxed Code Service
     • Visio Graphics Service
     • SharePoint Server Search
     • Document Conversions Launcher Service
     • Document Conversions Load Balancer Service
     • Search Query and Site Settings Service
     • Microsoft SharePoint Foundation Web Application
     • Central Administration
     • Microsoft SharePoint Foundation Incoming E-Mail
     • User Profile Service
     • Access Database Service 2010
     • Project Server Application Service
     • Microsoft SharePoint Foundation Subscription Settings Service
     • Lotus Notes Connector
     • Word Automation Services
     • PowerPoint Conversion Service
     • Machine Translation Service
     • Distributed Cache
   • Once you know the servers to run the service instance on, and the type name of your service instance, run the below command via SharePoint management shell on any of your SharePoint servers.

Get-SPServiceInstance | ?{$_.TypeName -eq "ServiceTypeName" -and $_.Server.Name -eq "ServerName"} | Start-SPServiceInstance

N.B., change the above highlighted values with your service instance type name, and your server name respectively.

1. • After running the above command, run the below command and check the service instance status. It should eventually become Online. If not, you will need to investigate why it's not.

Get-SPServiceInstance | ?{$_.TypeName -eq "ServiceTypeName" -and $_.Server.Name -eq "ServerName"}

1. • After the service gets started (its status become Online), run the same above command to start the service instance on the remaining servers in the farm that are supposed to run the service instance based on their MinRoles.
2. Create the service application normally using the UI from the Central Administration site (Application Management >> Manage Service Applications) or by using PowerShell.