Migrating SP 2010 classic content database to SP 2013 Claims web app. The Test-SPContentdatabase command complains that the database is still in classic mode.

Steps that were performed:

•    Created a new classic web app from PS - http://SPWEB:8080
•    Mounted the 2010 database.
•    Post this we followed steps on the TechNet - Migrate from classic-mode to claims-based authentication in SharePoint 2013 http://technet.microsoft.com/en-us/library/gg251985.aspx

        $WebAppName = "http:// SPWEB:8080"
        $wa = get-SPWebapplication $WebAppName
        $wa.UseClaimsAuthentication = $true
        $wa.Update()

        $account = "yourDomain\yourUser"
        $account = (New-SPClaimsPrincipal -identity $account -identitytype 1).ToEncodedString()
        $wa = get-SPWebApplication $WebAppName
        $zp = $wa.ZonePolicies("Default")
        $p = $zp.Add($account,"PSPolicy")
        $fc=$wa.PolicyRoles.GetSpecialRole("FullControl")
        $p.PolicyRoleBindings.Add($fc)
        $wa.Update()

        $wa.MigrateUsers($true)
        $wa.ProvisionGlobally()
        

•    Post this the test-SPContentDatabase was still showing inconsistency.
•    We then performed this cleanup : We wanted to remove any account that no longer existed in AD.
•    Ran the below PS script to identify inactive users in the site collection [by comparing them to AD].

        $web = get-spweb "http://SPWEB:8080/sites/sitecollection"
        $allusers = $web.siteusers
        $array = @()
        foreach($user in $allusers)
        {
        $login_full = $user.UserLogin
        $elements = $login_full.split("\")
        $login = $elements[1]
        $ds = New-Object System.DirectoryServices.DirectorySearcher
        $filter = "(&(objectCategory=User)(sAMAccountname=" + $login + "))"
        $ds.Filter = $filter
        if ($de = $ds.findone()) {write-host $login_full "User enabled"}
        else
        {
        write-host $login_full "User disabled"
        $array+=$login_full
        }
        }
        $array >path\fiIe.txt

•    From the output file, removed the SharePoint built in accounts and verified that the users are the ones missing from AD. Then fed the modified output file to the next script to remove such users:

        # Remove Pre-DefÌned Users
        $users = get-content -path "path\fiIe.txt"
        $web = get-spweb http://SPWEB:8080/sites/sitecollection
        $allusers = $web.siteusers
        foreach($temp in $ users)
        {
        write-host $temp "needs to be removed"
        remove-spuser -Id $temp -web $web -Confirm:$false
        }

•    Post this, we checked the output of the below SQL query : We wanted to check if there are any admin accounts still listed in the database using Classic Format.

SELECT TOP 100 [tp_SiteID],[tp_Login] FROM [UserInfo] WITH (NOLOCK) WHERE tp_IsActive = 1 AND tp_SiteAdmin = 1 AND tp_Deleted = 0 and tp_Login not LIKE 'i:%'

This returned two entries, which were causing the error :
tp_SiteID tp_Login
SiteID_GUID1 c:0+.w|SID1
SITEID_GUID2 c:0+.w|SID2

•    Using PowerShell, we figured that these SIDs were AD groups:

Domain\GRP1
Domain\GRP2

•    We removed these two groups from SharePoint and then the test-spcontentdatabase no longer complains about claims and classic inconsistency

Unable to create a new farm PSConfig fails with System.Security.Principal.IdentityNotMappedException.

Behind the Scene:

Error in Logs:

[DATE] 13:11:35 9 ERR Task configdb has failed with an unknown exception
[DATE] 13:11:35 9 ERR Exception: System.Security.Principal.IdentityNotMappedException: Some or all identity references could not be translated.
at System.Security.Principal.NTAccount.Translate(IdentityReferenceCollection sourceAccounts, Type targetType, Boolean forceSuccess)
at System.Security.Principal.NTAccount.Translate(Type targetType)
at System.Security.AccessControl.CommonObjectSecurity.ModifyAccess(AccessControlModification modification, AccessRule rule, Boolean& modified)
at System.Security.AccessControl.CommonObjectSecurity.AddAccessRule(AccessRule rule)
at Microsoft.SharePoint.Administration.Claims.SPSecurityTokenServiceCertificate.ProvisionLocal()
at Microsoft.SharePoint.Administration.Claims.SPSecurityTokenServiceApplication.ProvisionLocal(SPServiceInstance serviceInstance)
at Microsoft.SharePoint.Administration.SPIisWebServiceInstance.Provision()
at Microsoft.SharePoint.Administration.SPFarm.Join(Boolean skipRegisterAsDistributedCacheHost)
at Microsoft.SharePoint.PostSetupConfiguration.ConfigurationDatabaseTask.CreateOrConnectConfigDb()
at Microsoft.SharePoint.PostSetupConfiguration.ConfigurationDatabaseTask.Run()
at Microsoft.SharePoint.PostSetupConfiguration.TaskThread.ExecuteTask()

Below is what we are seeing: The problem was rooted outside of SharePoint, in IIS. We proved that with below steps.

+ In IIS, create a new app pool. Use a local admin domain account as the app pool identity. In the below example the App Pool is named "SecurityTokenServiceApplicationPool"
+ Run the below commands in PowerShell :

$NTAccount = new-object System.Security.Principal.NTAccount('IIS APPPOOL\SecurityTokenServiceApplicationPool')
$TranslatedToSidObject = $NTAccount.Translate( [System.Security.Principal.SecurityIdentifier])
$SID = New-Object System.Security.Principal.SecurityIdentifier($TranslatedToSidObject.Value)
$TranslatedToNTAccount = $SID.Translate([System.Security.Principal.NTAccount])
Write-host $TranslatedToSidObject.Value `t`t $TranslatedToNTAccount.Value

+ We will see failure in the commands on affected box, while it works in other environments:

$NTAccount = new-object System.Security.Principal.NTAccount('IIS APPPOOL\SecurityTokenServiceApplicationPool')
$TranslatedToSidObject = $NTAccount.Translate( [System.Security.Principal.SecurityIdentifier])
Exception calling "Translate" with "1" argument(s): "Some or all identity references could not be translated."
At line:1 char:1
+ $TranslatedToSidObject = $NTAccount.Translate( [System.Security.Principal.Securi ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [], MethodInvocationException
+ FullyQualifiedErrorId : IdentityNotMappedException

How this was fixed:

The problem was that the Application Host Helper Service (AppHostSvc) wasn't running. The accounts were getting created just fine when we went to use the app pool (like browse the default web site using the app pool).
WAS makes sure the accounts are created and the rights are assigned when it goes to start a worker process.
The AppHostSvc, though, will create the account when you simply add the app pool to the config, whether you're using it yet or not.
This service was set to manual start. We started it and everything worked as expected. PSConfig completed.

AppFabric installation was failing with "AppFabric installation failed because installer MSI returned with error code : 1603" Error.

Behind the Scenes:

We looked at logs located at -> %temp%

prerequisiteinstaller.<datetime>.log
AppServerSetup1_1_CustomActions<datetime>.log
AppServerSetup1_1<datetime>.log
AppServerSetup<datetime>.log
AppServerSetup<datetime>.err

These were the errors in Pre Requisite Installer Log:

[Date] 21:37:44 - All services have been configured successfully after AppFabric installation
[Date] 21:37:44 - Last return code (0X643=1603)
[Date] 21:37:44 - Reading the following DWORD value/name...
[Date] 21:37:44 – Flags
[Date] 21:37:44 - from the following registry location...
[Date] 21:37:44 - SOFTWARE\Microsoft\Updates\UpdateExeVolatile
[Date] 21:37:44 - Reading the following string value/name...
[Date] 21:37:44 – PendingFileRenameOperations
[Date] 21:37:44 - from the following registry location...
[Date] 21:37:44 - SYSTEM\CurrentControlSet\Control\Session Manager
[Date] 21:37:44 - Reading the following registry location...
[Date] 21:37:44 - SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate\Auto Update\RebootRequired
[Date] 21:37:44 - Error: The tool was unable to install Windows Server AppFabric.
[Date] 21:37:44 - Last return code (0X643=1603)

Below errors were interesting in AppServerSetup1_1_CustomActions<datetime>.log

<datetime> EXEPATH=powershell.exe PARAMS=-command "$content = Get-Content \"c:\Program Files\AppFabric 1.1 for Windows Server\\AppFabricServer.xml\"; $content = ForEach-Object -InputObject $content { $_ -replace \"INSTALLPATH\", \"c:\Program Files\AppFabric 1.1 for Windows Server\\\" }; Set-Content -value $content -path \"c:\Program Files\AppFabric 1.1 for Windows Server\\AppFabricServer.xml\" " LOGFILE=C:\Users\Account\AppData\Local\Temp\AppServerSetup1_1_CustomActions(datetime).log

ExitCode=0

<datetime> EXEPATH=powershell.exe PARAMS=-command "$content = Get-Content \"c:\Program Files\AppFabric 1.1 for Windows Server\\Consolidator.xml\"; $content = ForEach-Object -InputObject $content { $_ -replace \"INSTALLPATH\", \"c:\Program Files\AppFabric 1.1 for Windows Server\\\" }; Set-Content -value $content -path \"c:\Program Files\AppFabric 1.1 for Windows Server\\Consolidator.xml\" " LOGFILE=C:\Users\Account\AppData\Local\Temp\AppServerSetup1_1_CustomActions(datetime).log

ExitCode=0

Executing action : Microsoft.ApplicationServer.InstallHelper.SQMTaskInstallAction

SQMTaskInstall areguments - /create /tn "\Microsoft\Windows\AppFabric\Customer Experience Improvement Program\Consolidator" /xml "c:\Program Files\AppFabric 1.1 for Windows Server\Consolidator.xml"

Error: ERROR: The network address is invalid.
The setup needs to add a scheduled task for customer experience program, but that fails if the task scheduler service is not running.
This could also fail if the task already exists with error : "ERROR: Cannot create a file when that file already exists"

Resolution:

For - "ERROR: The network address is invalid" -> Ensure the Task Scheduler Windows Service is running.

For - "ERROR: Cannot create a file when that file already exists" -> Delete the task "Consolidator" in "%SYSTEM32%\Tasks\Microsoft\Windows\AppFabric\Customer Experience Improvement Program" and re run the installation.

Note -

- Even if the file exists the Task Scheduler UI does not show it
- The file seems to be left there by a previous AppFabric installation
- The setup creates it even if the CU choose to do not join the Customer Experience program
- There are other reasons why we may fail on this installer – refer here.

We are unable to create any site collections. We see error "This page has encountered a critical error. Contact your system administrator if this problem persists".

What we see in the logs:

• Site collection creation was failing because of the below stack as per ULS logs:

[DATE] 09:36:39.25 w3wp.exe (PID) TID Document Management Server Document Sets 55r4 Unexpected

DocumentSet FeatureActivating: throws exception: This page has encountered a critical error. Contact your system administrator if this problem persists.. Stacktrace:
at Microsoft.SharePoint.ApplicationRuntime.SafeControls.IsSafeControl(Boolean isAppWeb, Type type, String& unsafeErrorMessage)
at Microsoft.SharePoint.WebPartPages.TypeCache.CheckIfSafeControl(Type type, Boolean isAppWeb)
at Microsoft.SharePoint.WebPartPages.TypeCache.get_Item(Type type, Boolean isAppWeb, Boolean disableSafeControlsCheck)
at Microsoft.SharePoint.WebPartPages.WebPart.WriteXmlGlobal(Boolean disableSafeControlsCheck)
at Microsoft.SharePoint.WebPartPages.BinaryWebPartSerializer.Serialize(SerializationMode mode, BinaryWebPartSerializerFlag binaryWebPartSerializerFlags, BinaryWebPartSerializerWriter writer)
at Microsoft.SharePoint.WebPartPages.BinaryWebPartSerializer.Serialize(SerializationMode mode, BinaryWebPartSerializerFlag binaryWebPartSerializerFlags)
at Microsoft.SharePoint.WebPartPages.SPWebPartManager.AddWebPartToStore(WebPart webPart, Int32 viewId, String viewGuid)
at Microsoft.SharePoint.WebPartPages.SPWebPartManager.AddWebPartInternal(SPSupersetWebPart superset, Boolean throwIfLocked)
at Microsoft.SharePoint.WebPartPages.SPLimitedWebPartManager.AddWebPart(WebPart webPart, String zoneId, Int32 zoneIndex)
at Microsoft.Office.DocumentManagement.Internal.DocumentSetFeatureReceiver.FeatureActivated(SPFeatureReceiverProperties properties)
at Microsoft.SharePoint.SPFeature.DoActivationCallout(Boolean fActivate, Boolean fForce)
at Microsoft.SharePoint.SPFeature.Activate(SPSite siteParent, SPWeb webParent, SPFeaturePropertyCollection props, SPFeatureActivateFlags activateFlags, Boolean fForce)
at Microsoft.SharePoint.SPFeatureCollection.AddInternal(SPFeatureDefinition featdef, Version version, SPFeaturePropertyCollection properties, SPFeatureActivateFlags activateFlags, Boolean force, Boolean fMarkOnly)
at Microsoft.SharePoint.SPFeatureCollection.AddInternalWithName(Guid featureId, Int32 compatibilityLevel, String featureName, Version version, SPFeaturePropertyCollection properties, SPFeatureActivateFlags activateFlags, Boolean force, Boolean fMarkOnly, SPFeatureDefinitionScope featdefScope)
at Microsoft.SharePoint.SPFeatureCollection.AddInternal(Guid featureId, Version version, SPFeaturePropertyCollection properties, Boolean force, Boolean fMarkOnly, SPFeatureDefinitionScope featdefScope)
at Microsoft.Office.DocumentManagement.VideoAndRichMedia.Internal.VideoAndRichMediaFeatureReceiver.FeatureActivated(SPFeatureReceiverProperties properties)
at Microsoft.SharePoint.SPFeature.DoActivationCallout(Boolean fActivate, Boolean fForce)
at Microsoft.SharePoint.SPFeature.Activate(SPSite siteParent, SPWeb webParent, SPFeaturePropertyCollection props, SPFeatureActivateFlags activateFlags, Boolean fForce)
at Microsoft.SharePoint.SPFeatureCollection.AddInternal(SPFeatureDefinition featdef, Version version, SPFeaturePropertyCollection properties, SPFeatureActivateFlags activateFlags, Boolean force, Boolean fMarkOnly)
at Microsoft.SharePoint.SPTemplateAssociationElement.EnsureTemplateAssociatedSiteFeaturesActivated(SPSite site, String sTemplateName)
at Microsoft.SharePoint.SPFeatureManager.<>c__DisplayClass16.b__15()
at Microsoft.SharePoint.SPSecurity.RunAsUser(SPUserToken userToken, Boolean bResetContext, WaitCallback code, Object param)
at Microsoft.SharePoint.SPFeatureManager.EnsureTemplateAssociatedSiteFeaturesActivated(Byte[]& userToken, Guid& tranLockerId, Int32 nZone, Guid databaseid, Guid siteid, String sTemplateName)
at Microsoft.SharePoint.ApplicationRuntime.SafeControls.IsSafeControl(Boolean isAppWeb, Type type, String& unsafeErrorMessage)
at Microsoft.SharePoint.WebPartPages.TypeCache.CheckIfSafeControl(Type type, Boolean isAppWeb)
at Microsoft.SharePoint.WebPartPages.TypeCache.get_Item(Type type, Boolean isAppWeb, Boolean disableSafeControlsCheck)
at Microsoft.SharePoint.WebPartPages.WebPart.WriteXmlGlobal(Boolean disableSafeControlsCheck)
at Microsoft.SharePoint.WebPartPages.BinaryWebPartSerializer.Serialize(SerializationMode mode, BinaryWebPartSerializerFlag binaryWebPartSerializerFlags, BinaryWebPartSerializerWriter writer)
at Microsoft.SharePoint.WebPartPages.BinaryWebPartSerializer.Serialize(SerializationMode mode, BinaryWebPartSerializerFlag binaryWebPartSerializerFlags)
at Microsoft.SharePoint.WebPartPages.SPWebPartManager.AddWebPartToStore(WebPart webPart, Int32 viewId, String viewGuid)
at Microsoft.SharePoint.WebPartPages.SPWebPartManager.AddWebPartInternal(SPSupersetWebPart superset, Boolean throwIfLocked)
at Microsoft.SharePoint.WebPartPages.SPLimitedWebPartManager.AddWebPart(WebPart webPart, String zoneId, Int32 zoneIndex)
at Microsoft.Office.DocumentManagement.Internal.DocumentSetFeatureReceiver.FeatureActivated(SPFeatureReceiverProperties properties).

• We can see that the failure was while activating the feature called "DocumentSet". In this process, it attempts to check if the feature is a SafeControl – for which it would need to look at the web.config file of the web application in IIS.

• This is when we found that the affected web applications had no files in the IIS virtual directory on the server. They were missing, which was causing the unexpected error at site collection creation.

How did we fix this?

• We provisioned these web applications to bring back these files:

    Add-pssnapin microsoft.sharepoint.powershell
    $contentservice = [Microsoft.sharepoint.Administration.spwebservice]::contentservice
    $webapp = $contentservice.webapplications | where { $_.name -match "WEB_APP_NAME" }
    $webapp.provision()

• After this, we were able to create site collections on the re-provisioned web application. We tested creating a new web application, but that was timing out. So we did this on the central admin application pool in IIS:

IIS Manager > App Pool > Central Admin App Pool > Shut Down Limit > Changed this from 90 seconds to 600 Seconds

• After this we were able to create web applications from central admin site successfully.
• We also noticed that web application creation was taking unusually long to complete. One tip to improve the performance would be to disable Receive Side Scaling and TCP Offloads on the SharePoint and SQL Servers. This is explained in the article below:

Information about the TCP Chimney Offload, Receive Side Scaling - http://support.microsoft.com/kb/951037

• It is likely that a timed out web application create command is what caused this issue in the first place.
• You would need to disable the settings both at the NIC Card level as well as from command prompt on both SharePoint and SQL servers.
• Please be aware that this would cause a temporary network outage on the servers, so it is best to schedule this for a downtime in the environment.

We are not able to open office documents in a new web application. We get the following error: "Sorry we cannot open 'URL.xxx document'"

What we see in the Logs:

• Below is what we saw in IIS for the working scenario:

[DATE] 21:58:09 s-ip OPTIONS /SiteCollection/Shared+Documents/ - 443 - c-ip Microsoft+Office+Word+2013 401 0 0 64
[DATE] 21:58:59 s-ip OPTIONS /SiteCollection/Shared+Documents/ - 443 - c-ip Microsoft+Office+Word+2013 401 1 2148074254 2
[DATE] 21:58:59 s-ip OPTIONS /SiteCollection/Shared+Documents/ - 443 domain\user c-ip Microsoft+Office+Word+2013 200 0 0 80
[DATE] 21:58:59 s-ip POST /SiteCollection/_vti_bin/cellstorage.svc/CellStorageService - 443 - c-ip Microsoft+Office+Word+2013 401 1 2148074254 11
[DATE] 21:58:59 s-ip POST /_vti_bin/cellstorage.https.svc/cellstorageservice - 443 domain\user c-ip Microsoft+Office+Word+2013 200 0 0 607
[DATE] 21:59:01 s-ip POST /_vti_bin/cellstorage.https.svc/cellstorageservice - 443 domain\user c-ip Microsoft+Office+Word+2013 200 0 0 834

• Below is what we saw in IIS for the non-working scenario:

[DATE] 22:07:53 s-ip OPTIONS /sites/SiteCollection/Shared+Documents/ - 443 - c-ip Microsoft+Office+Word+2013+(15.0.4517)+Windows+NT+6.2 500 0 64 2
[DATE] 22:07:53 s-ip OPTIONS /sites/SiteCollection/Shared+Documents/ - 443 - c-ip Microsoft+Office+Word+2013+(15.0.4517)+Windows+NT+6.2 403 16 2148204809 17
[DATE] 22:07:53 s-ip OPTIONS /sites/SiteCollection/Shared+Documents/ - 443 - c-ip Microsoft+Office+Word+2013+(15.0.4517)+Windows+NT+6.2 403 16 2148204809 15
[DATE] 22:07:53 s-ip HEAD /sites/SiteCollection/Shared+Documents/BrokenDocument.docx - 443 - c-ip Microsoft+Office+Word+2013+(15.0.4517)+Windows+NT+6.2 403 16 2148204809 15
[DATE] 22:07:53 s-ip GET /sites/SiteCollection/Shared+Documents/BrokenDocument.docx - 443 - c-ip Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+6.2;+Win64;+x64;+Trident/6.0;+.NET4.0E;+.NET4.0C;+.NET+CLR+3.5.30729;+.NET+CLR+3.0.30729;+.NET+CLR+2.0.50727;+InfoPath.3;+ms-office;+MSOffice+15) 403 16 2148204809 57

• The OPTIONS request never gets authenticated in the non-working scenario.
• We captured failed request tracing logs for 403 and 500 and below is what we saw:

This is for the 500:

20. NOTIFY_MODULE_START ModuleName="IsapiFilterModule", Notification="MAP_PATH", fIsPostNotification="false" 22:25:29.868
21. NOTIFY_MODULE_END ModuleName="IsapiFilterModule", Notification="MAP_PATH", fIsPostNotificationEvent="false", NotificationStatus="NOTIFICATION_CONTINUE" 22:25:29.868
22. MODULE_SET_RESPONSE_ERROR_STATUS
Warning ModuleName="IIS Web Core", Notification="BEGIN_REQUEST", HttpStatus="500", HttpReason="Internal Server Error", HttpSubStatus="0", ErrorCode="The I/O operation has been aborted because of either a thread exit or an application request.
(0x800703e3)", ConfigExceptionInfo="" 22:25:29.868

This is for the 403:

20. NOTIFY_MODULE_START ModuleName="IsapiFilterModule", Notification="MAP_PATH", fIsPostNotification="false" 22:25:29.913
21. NOTIFY_MODULE_END ModuleName="IsapiFilterModule", Notification="MAP_PATH", fIsPostNotificationEvent="false", NotificationStatus="NOTIFICATION_CONTINUE" 22:25:29.913
22. MODULE_SET_RESPONSE_ERROR_STATUS
Warning ModuleName="IIS Web Core", Notification="BEGIN_REQUEST", HttpStatus="403", HttpReason="Forbidden", HttpSubStatus="16", ErrorCode="A certificate chain processed, but terminated in a root certificate which is not trusted by the trust provider.
(0x800b0109)", ConfigExceptionInfo="" 22:25:29.928

• The 403 Errors were for client certificates.

How do we fix this?

• We made this change on the affected site:

IIS > Site > SSL Settings > Client Certificates > Set this to Ignore [this was set to Accept on the affected site].

• As soon as we made this change, Office documents from the affected site opened up in Office Client.

Operation 'crawl hang' is not responding and a recovery action will be attempted.
Search application '-crawl-0' appears to be malfunctioning and will be reinitialized.

What we see in ULS logs:

--ULS is filled with these exceptions:

[DATE] 08:17:17.44 mssearch.exe (PID) TID SharePoint Server Search Crawler:Content Plugin af7zk High CTSSession::Init: m_CSSConnection.Service.CreateSession threw exception (System.ServiceModel.CommunicationException), message = Exception type is System.ServiceModel.FaultException`1[System.String] message = The creator of this fault did not specify a Reason.;, CSSNode = net.tcp://servername/ID/ContentProcessingComponent1/ContentSubmissionServices/content, Flow = Microsoft.CrawlerFlow

[DATE] 08:17:17.44 mssearch.exe (PID) TID SharePoint Server Search Crawler:Content Plugin ahg9y High CSSFeeder::Run: session is not recreated or lost connection - reconnecting

--After every such failure, the noderunner component attempts to provision a flow and fails:

[DATE] 08:17:17.52 NodeRunnerContent1-16d3caaa-174 (PID) TID Search Common Processing ai9on Monitorable [Microsoft.CrawlerFlow-c5e47b65-99e6-4a78-8f64-3eec1cf9afd6] Microsoft.Ceres.Evaluation.Engine.Flows.FlowHandle : Failure during evaluation set-up Flow name: Microsoft.CrawlerFlow Flow identifier: 86fa4f70-16a6-4f12-b7ff-3f9e561e876e System.InvalidOperationException: Sandbox worker pool is closed.
at Microsoft.Ceres.DocParsing.Runtime.Core.Sandbox.SandboxWorkerPool.AcquireProxy(Guid sessionId, ParsingConfiguration configuration)
at Microsoft.Ceres.DocParsing.Runtime.Core.SessionManager.CreateSession(ParsingConfiguration configuration)
at Microsoft.Ceres.ContentEngine.Parsing.Evaluators.DocumentParserEvaluator.DocumentParserProducer.OpenParsingSession()
at Microsoft.Ceres.Evaluation.Processing.Producers.RecordProducerBase`1.SetInputProperties(RecordSetProperties inputProperties)
at Microsoft.Ceres.Evaluation.Processing.Producers.ProducerRecordSetSink.SetProperties(RecordSetProperties properties)
at Microsoft.Ceres.Evaluation.Processing.Producers.MultiOutputProducer`1.SetOutputProperties(Int32 index, RecordSetProperties properties)
at Microsoft.Ceres.Evaluation.Processing.Producers.MultiOutputProducer`1.SetOutputProperties(RecordSetProperties properties)
at Microsoft.Ceres.ContentEngine.Processing.BuiltIn.RouterProducer.Initialize()
at Microsoft.Ceres.Evaluation.Processing.Producers.RecordProducerBase`1.SetInputProperties(RecordSetProperties inputProperties)
at Microsoft.Ceres.Evaluation.Processing.Producers.ProducerRecordSetSink.SetProperties(RecordSetProperties properties)
at Microsoft.Ceres.Evaluation.Engine.PushPull.PushPullThread..ctor(EvaluationContext context, IRecordSetSink output, IRecordSet input)
at Microsoft.Ceres.Evaluation.Engine.PushPull.PullPushEvaluator.AddInput(Edge inputEdge, IRecordSet recordSet)
at Microsoft.Ceres.Evaluation.Engine.Flows.EvaluationContext.SetupOutputs(LinkedList`1 orderedOperators, IDictionary`2 edgeMap)
at Microsoft.Ceres.Evaluation.Engine.Flows.EvaluationContext.DoEvaluate(OperatorFlow flow, EvaluationConfig config, FlowHandle handle)
at Microsoft.Ceres.Evaluation.Engine.Flows.EvaluationContextFsm.Evaluate(EventArgs args)
at Microsoft.Ceres.Common.Utils.StateMachine.FsmActionTask.Execute(EventArgs args)
at Microsoft.Ceres.Common.Utils.StateMachine.FsmAction.Execute(EventArgs args)
at Microsoft.Ceres.Common.Utils.StateMachine.FiniteStateMachine`1.DoHandleEvent(FsmEvent fsmEvent, EventArgs args)
at Microsoft.Ceres.Common.Utils.StateMachine.FiniteStateMachine`1.InternalHandleEvent(FsmEvent fsmEvent, EventArgs args)
at Microsoft.Ceres.Evaluation.Engine.Flows.EvaluationContext.EvaluateGraph(OperatorFlow operatorGraph, FlowHandle handle, EvaluationConfig config)
at Microsoft.Ceres.Evaluation.Engine.EvaluationWorker.InternalEvaluateAndStart(OperatorFlow graph, FlowHandle handle, Guid id, EvaluationConfig config, IFlowDebugHandler debugHandler, BreakpointSpecification[] initialBreakpoints)

[DATE] 08:17:17.52 NodeRunnerContent1-16d3caaa-174 (PID) TID Search Content Processing ai89u Monitorable ContentIntegrationEngine: Execution of flow Microsoft.CrawlerFlow with id 86fa4f70-16a6-4f12-b7ff-3f9e561e876e failed.

[DATE] 08:17:17.52 NodeRunnerContent1-16d3caaa-174 (PID) TID Search Common Processing 25 Warning FlowName=Microsoft.CrawlerFlow. FlowID=86fa4f70-16a6-4f12-b7ff-3f9e561e876e. Flow failed during processing. The original error was Failure during evaluation set-up (Exception type: System.InvalidOperationException. Exception message: Sandbox worker pool is closed.)

[DATE] 08:17:17.52 NodeRunnerContent1-16d3caaa-174 (PID) TID Search Content Processing ai9li Monitorable Microsoft.Ceres.ContentEngine.SubmitterComponent.SessionFlowHandler : Evaluation with parent identifier 1c14e435-cd68-4f29-bc10-b3cc3fbf4f97 is not running. The state of the single evaluation is Failed.

[DATE] 08:17:17.52 NodeRunnerContent1-16d3caaa-174 (PID) TID Search Content Processing ai9mb Monitorable Microsoft.Ceres.ContentEngine.SubmitterComponent.SubmissionService : Create Session failing: Microsoft.Ceres.ContentEngine.Services.SubmissionService.FailedToStartFlowException: Unable to start a flow instance on any of the available nodes. ---> Microsoft.Ceres.ContentEngine.Services.SubmissionService.FailedToStartFlowException: Unable to start a flow instance on any of the available nodes. at Microsoft.Ceres.ContentEngine.SubmitterComponent.SessionWorker.Initialize() at Microsoft.Ceres.ContentEngine.SubmitterComponent.Session.Start() at Microsoft.Ceres.ContentEngine.SubmitterComponent.Session.Open() --- End of inner exception stack trace --- at Microsoft.Ceres.ContentEngine.SubmitterComponent.Session.Open() at Microsoft.Ceres.ContentEngine.SubmitterComponent.SubmissionService.CreateSession(FeedingSessionSettings settings)

[DATE] 08:17:17.52 NodeRunnerContent1-16d3caaa-174 (PID) TID Search Content Processing ai9j8 Monitorable Microsoft.Ceres.ContentEngine.SubmitterComponent.ContentSubmissionDecorator : CreateSession failed: System.ServiceModel.FaultException`1[System.String]: The creator of this fault did not specify a Reason. (Fault Detail is equal to Specified flow: Microsoft.CrawlerFlow could not be started. Unable to start a flow instance on any of the available nodes.).

[DATE] 08:17:17.52 NodeRunnerContent1-16d3caaa-174 (PID) TID Search Content Processing ai9kq Unexpected Microsoft.Ceres.ContentEngine.SubmitterComponent.ContentSubmissionDecorator : CreateSession() failed with unexpected exception System.ServiceModel.FaultException`1[System.String]: The creator of this fault did not specify a Reason. (Fault Detail is equal to Specified flow: Microsoft.CrawlerFlow could not be started. Unable to start a flow instance on any of the available nodes.).

[DATE] 08:17:17.52 NodeRunnerContent1-16d3caaa-174 (PID) TID Search Search Platform Services alm32 Medium Exception in remote service invocation (service=content, method=http://schemas.microsoft.com/ceres/contenttransformation/2009/11/submission/IContentSubmissionService/CreateSession, user=DOMAIN\FarmAccount): System.ServiceModel.FaultException`1[System.String]: The creator of this fault did not specify a Reason. (Fault Detail is equal to Specified flow: Microsoft.CrawlerFlow could not be started. Unable to start a flow instance on any of the available nodes.).

--So essentially, mssearch.exe is unable to feed the content to CPC, because we don't have any flows open/available.

--We see these messages incrementally:

[DATE] 08:54:04.85 mssearch.exe (PID) TID SharePoint Server Search Crawler:Gatherer Service ad9s9 Medium CGthrPrj::PingPlugins no transaction committed in 3706 seconds [cgatherprj.cxx:2581] search\native\gather\gthrsvc\cgatherprj.cxx

[DATE] 08:54:14.93 mssearch.exe (PID) TID SharePoint Server Search Crawler:Gatherer Service ad9s9 Medium CGthrPrj::PingPlugins no transaction committed in 3716 seconds [cgatherprj.cxx:2581] search\native\gather\gthrsvc\cgatherprj.cxx

[DATE] 08:54:24.98 mssearch.exe (PID) TID SharePoint Server Search Crawler:Gatherer Service ad9s9 Medium CGthrPrj::PingPlugins no transaction committed in 3726 seconds [cgatherprj.cxx:2581] search\native\gather\gthrsvc\cgatherprj.cxx

[DATE] 09:04:16.59 mssearch.exe (PID) TID SharePoint Server Search Crawler:Gatherer Service ad9s9 Medium CGthrPrj::PingPlugins no transaction committed in 124 seconds [cgatherprj.cxx:2581] search\native\gather\gthrsvc\cgatherprj.cxx

[DATE] 09:04:26.65 mssearch.exe (PID) TID SharePoint Server Search Crawler:Gatherer Service ad9s9 Medium CGthrPrj::PingPlugins no transaction committed in 134 seconds [cgatherprj.cxx:2581] search\native\gather\gthrsvc\cgatherprj.cxx

--We see that every time the time elapsed hits around 3700 seconds, it resets.

--Below are the entries we see when search application re-initializes itself and this corresponds to the two events we see in the application event logs roughly every one hour:

[DATE] 08:54:30.22 mssearch.exe (PID) TID SharePoint Server Search Crawler:Gatherer Service ds3f High Processing new problem 2 for application GUID-crawl-0 (1, 1, 1) [gthrapp.cxx:5651] search\native\gather\gthrsvc\gthrapp.cxx
[DATE] 08:54:30.22 mssearch.exe (PID) TID SharePoint Server Search Crawler:Common aikdz Assert
location: search\native\gather\gthrsvc\gthrapp.cxx(6686) condition: !"Dismounting application with hanged crawl" StackTrace:
at Microsoft.Office.Server.Native.dll: (sig=c3cf111a-d6fc-4ec5-ba1c-c9b8ebb47e41|2|microsoft.office.server.native.pdb, offset=1F074)
at MSSrch.dll: (sig=bcc79b69-8dc0-4f7b-b42b-469030237946|2|mssrch.pdb, offset=22759)
at MSSrch.dll: (offset=226FD)
at MSSrch.dll: (offset=89F2C)
at MSSrch.dll: (offset=8FAC7)
at MSSrch.dll: (offset=8938F)
at MSSrch.dll: (offset=94802)
at MSSrch.dll: (offset=71855)
at kernel32.dll: (sig=c4312728-ba1f-4691-955e-99b2e026fafc|2|kernel32.pdb, offset=1652D)
at ntdll.dll: (sig=fb62ad7b-aec9-42f9-81e9-c6c84c71c11c|2|ntdll.pdb, offset=2C541

[DATE] 08:54:30.22 mssearch.exe (PID) TID SharePoint Server Search Search service events eq62 Warning Search application 'GUID-crawl-0' appears to be malfunctioning and will be reinitialized.

[DATE] 08:54:30.22 mssearch.exe (PID) TID SharePoint Server Search Crawler:Gatherer Plugin e5aq High locking gatherer for status change NSC_SHUTDOWN (GUID-crawl-0, Portal_Content, crawl -1) [gatherobj.cxx:1664] search\native\gather\server\gatherobj.cxx

[DATE] 08:54:30.22 mssearch.exe (PID) TID SharePoint Server Search Crawler:Gatherer Plugin e5fn Medium CGatherer::CancelCompleteTransactions is called [gatherobj.cxx:7910] search\native\gather\server\gatherobj.cxx

[DATE] 08:54:30.22 mssearch.exe (PID) TID SharePoint Server Search Crawler:Content Plugin dts3 Medium Received signal (PrepareStatusChange): NSC_SHUTDOWN [contentpiobj.cxx:595] search\native\gather\plugins\contentpi\contentpiobj.cxx

[DATE] 08:54:30.22 mssearch.exe (PID) TID SharePoint Server Search Crawler:Content Plugin af7y2 High CSSFeedersManager::Shutdown called

[DATE] 08:54:31.90 mssearch.exe (PID) TID SharePoint Server Search Crawler:Gatherer Plugin e5ga High CGatherer::FlushLinks FlushLinksForCrawl fails project ID 1, crawl ID 1 [gatherobj.cxx:9073] search\native\gather\server\gatherobj.cxx

--The root cause appears to be

[DATE] 08:17:17.52 NodeRunnerContent1-16d3caaa-174 (PID) TID Search Common Processing ai9on Monitorable
[Microsoft.CrawlerFlow-c5e47b65-99e6-4a78-8f64-3eec1cf9afd6] Microsoft.Ceres.Evaluation.Engine.Flows.FlowHandle : Failure during evaluation set-up Flow name: Microsoft.CrawlerFlow Flow identifier: 86fa4f70-16a6-4f12-b7ff-3f9e561e876e System.InvalidOperationException: Sandbox worker pool is closed.
at Microsoft.Ceres.DocParsing.Runtime.Core.Sandbox.SandboxWorkerPool.AcquireProxy(Guid sessionId, ParsingConfiguration configuration)
at Microsoft.Ceres.DocParsing.Runtime.Core.SessionManager.CreateSession(ParsingConfiguration configuration)

How do we fix this?

• Based on prolonged research, this issue appeared to be permissions related.
• Checked account search is running under -> All the NodeRunner processes are running with:

DOMAIN\FarmAccount

• This account is the farm account. The farm account is local admin on the server.
• Checked these group policies:

SeAssignPrimaryTokenPrivilege (Replace a process-level token) -> FARM ACCOUNT IS NOT HERE.
SeImpersonatePrivilege (Impersonate a client after authentication) -> Farm account is present here.
SeServiceLogonRight (Log on as a service) -> FARM ACCOUNT IS NOT HERE.
SeIncreaseQuotaPrivilege (Adjust memory quotas for a process) -> FARM ACCOUNT IS NOT HERE.
Log on as a batch job -> FARM ACCOUNT IS NOT HERE.

• This account requires the below privileges primarily for the crawls to prevent those errors:

SeAssignPrimaryTokenPrivilege (Replace a process-level token)
SeImpersonatePrivilege (Impersonate a client after authentication)
SeServiceLogonRight (Log on as a service)
SeIncreaseQuotaPrivilege (Adjust memory quotas for a process)

• If NodeRunner is not running with the farm account, we need to ensure the specific account it uses has the above set of permissions.

However, this is not exposed in SharePoint Online via CSOM. So in order to touch all the SPWeb objects via CSOM, we need to use recursion – and the below code sample can assist in achieving that.

# replace these details (also consider using Get-Credential to enter password securely as script runs)

$username = "administrator@domain.onmicrosoft.com"

$password = "pwd"

#Provide site collection URL here. Alternatively, you can enumerate all site collections using get-sposite and process each.

#The SPSite will be constructed based on this URL.

$url = "https://domain.sharepoint.com/"

$securePassword = ConvertTo-SecureString $Password -AsPlainText -Force

#Install SharePoint Online Client SDK on the machine you are running this script to load these DLLs.

[System.Reflection.Assembly]::LoadWithPartialName('Microsoft.SharePoint.Client')

[System.Reflection.Assembly]::LoadWithPartialName('Microsoft.SharePoint.Client.Runtime')

# connect/authenticate to SharePoint Online and get ClientContext object

$clientContext = New-Object Microsoft.SharePoint.Client.ClientContext($url)

$credentials = New-Object Microsoft.SharePoint.Client.SharePointOnlineCredentials($username, $securePassword)

$clientContext.Credentials = $credentials

if (!$clientContext.ServerObjectIsNull.Value)

{

Write-Host "Connected to SharePoint Online site: '$Url'" -ForegroundColor Green

}

function processWeb($web)

{

    if($urls -notcontains $web.url)

    {

        $urls += $web.url

    }

    function getchildcount($web)

    {

        $web.Context.Load($web.webs)

$web.Context.ExecuteQuery()

return $web.Webs.Count

    }

    if((getchildcount($web)) -gt 0)

    {

        $ch = $web.Webs

        foreach($c in $ch)

        {

            $WEB.Context.Load($C)

$web.Context.ExecuteQuery()

processWeb($c)

        }

    }

    return $urls

}

$rootweb = $clientContext.Web

$clientContext.Load($rootweb)

$clientContext.ExecuteQuery()

#This command may take some time to complete – can be monitored using a tracing tool like Fiddler.

$subwebURLs = processWeb($rootweb)

#Displaying the list of SPWeb URLs as a result.

$subwebURLs

A while back, we happened to work on an issue with SharePoint Incoming email. We saw that certain types of attachments were causing issues with this set up. In order to check what SharePoint was doing with the eml files in the Drop folder, we first needed to parse them like SharePoint would. Below sample code shows how this is done:

Reference - SPEmailMessage class

[system.reflection.assembly]::LoadWithPartialName('Microsoft.sharepoint')

$fileInfo = new-object System.IO.FileInfo('C:\inetpub\mailroot\DROP\<random_number>.eml')

$fs = $fileInfo.OpenRead()

$fs.Seek(0, [System.IO.SeekOrigin]::Begin)

$message = new-object Microsoft.SharePoint.Utilities.SPEmailMessage( $fs, "account@domain.com")

$message.Attachments | fl 

Using the above parsing technique, we can enumerate and inspect all the attachments as .Net objects, and see if anything stands out for the prospective bad attachments.

#install SharePoint Online Client SDK on the machine you are running this from.

#run this script as local box admin

# Keep tenant admin account details handy

# ensure there is a folder called c:\op on the box that is running this script.

#grab tenant information

$domain = Read-Host "Enter the domain prefix"

$adminURI = 'https://' + $domain + '-admin.sharepoint.com'

$admin_account = Read-Host "Enter the tenant admin account login"

$pwd = read-host -AsSecureString "Enter the admin account password"

#load the assemblies

$loadInfo1 = [System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SharePoint.Client")

$loadInfo2 = [System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SharePoint.Client.Runtime")

$loadInfo3 = [System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SharePoint.Client.UserProfiles")

#SPO Credentials

$creds = New-Object Microsoft.SharePoint.Client.SharePointOnlineCredentials($Admin_Account, $pwd)

# Add the path of the User Profile Service to the SPO admin URL, then create a new webservice proxy to access it

$proxyaddr = "$AdminURI/_vti_bin/UserProfileService.asmx?wsdl"

$UserProfileService= New-WebServiceProxy -Uri $proxyaddr -UseDefaultCredential False

$UserProfileService.Credentials = $creds

# Take care of auth cookies

$strAuthCookie = $creds.GetAuthenticationCookie($AdminURI)

$uri = New-Object System.Uri($AdminURI)

$container = New-Object System.Net.CookieContainer

$container.SetCookies($uri, $strAuthCookie)

$UserProfileService.CookieContainer = $container

# Grab the first User profile, at index -1

$odbsites = @()

$UserProfileResult = $UserProfileService.GetUserProfileByIndex(-1)

Write-Host "Enumeration of PersonalSites on the tenant"

Write-Host "Starting- This could take a while."

$NumProfiles = $UserProfileService.GetUserProfileCount()

$i = 1

# As long as the next User profile is NOT the one we started with (at -1)...

While ($UserProfileResult.NextValue -ne -1)

{

Write-Host "Examining profile $i of $NumProfiles"

# Look for the Personal Space object in the User Profile and pull it out

# (PersonalSpace is the name of the path to a user's mysite)

$Prop = $UserProfileResult.UserProfile | Where-Object { $_.Name -eq "PersonalSpace" }

$Url= $Prop.Values[0].Value

# If "PersonalSpace" (which we've copied to $Url) exists, log it to our array...

if ($Url)

{

#$Url | Out-File $LogFile -Append -Force

$odbsites+=('https://'+$domain+'-my.sharepoint.com'+$url)

}

# And now we check the next profile the same way...

$UserProfileResult = $UserProfileService.GetUserProfileByIndex($UserProfileResult.NextValue)

$i++

}

#Dump out the ODB Sites URL to the file system

$odbsites | out-file c:\op\odbsites.txt

#The oob page will have a row limit of 30 that is hard coded - but in this script, we can alter it.

#The sample gives you Raw search results. Further development will be required to utilize these search results in a useful manner.

#Note - the results are heavily dependent on Search Freshness.

# replace these details (also consider using Get-Credential to enter password securely as script runs)

$username = "administrator@domain.com"

$password = "pwd"

$url = "https://domain-my.sharepoint.com"

$securePassword = ConvertTo-SecureString $Password -AsPlainText -Force

#Install SharePoint Online Client SDK

[system.reflection.assembly]::LoadWithPartialName('Microsoft.SharePoint.Client')

[system.reflection.assembly]::LoadWithPartialName('Microsoft.SharePoint.Client.Search')

[system.reflection.assembly]::LoadWithPartialName('Microsoft.SharePoint.Client.Runtime')

# connect/authenticate to SharePoint Online and get ClientContext object..

$clientContext = New-Object Microsoft.SharePoint.Client.ClientContext($url)

$credentials = New-Object Microsoft.SharePoint.Client.SharePointOnlineCredentials($username, $securePassword)

$clientContext.Credentials = $credentials

if (!$clientContext.ServerObjectIsNull.Value)

{

Write-Host "Connected to SharePoint Online site: '$Url'" -ForegroundColor Green

}

$web = $clientContext.Web

$clientContext.Load($web)

$clientContext.ExecuteQuery()

#enter the UPN of the user for whom you need to get SharedWithMe details.

$username = 'testuser@domain.onmicrosoft.com'

#set RowLimit below to the number of results you want to see.

$keywordQuery = New-Object Microsoft.SharePoint.Client.Search.Query.KeywordQuery($clientContext)

$keywordQuery.QueryText = 'SharedWithUsersOWSUSER:\' + $userName.ToString() + '\'

$keywordQuery.ClientType = "DocsSharedWithMe"

$keywordQuery.StartRow = 0

$keywordQuery.RowLimit = 100

$keywordQuery.ProcessBestBets = $false

$keywordQuery.BypassResultTypes = $true

$keywordQuery.EnableInterleaving = $false

$keywordQuery.EnableQueryRules = $false

$keywordQuery.EnableStemming = $true

#modify below property set as per requirement - pull down only the ones that are required and will be used down the pipeline in your code.

$keywordQuery.SelectProperties.Add("Author");

$keywordQuery.SelectProperties.Add("ContentTypeId");

$keywordQuery.SelectProperties.Add("FileExtension");

$keywordQuery.SelectProperties.Add("Filename");

$keywordQuery.SelectProperties.Add("IsMyDocuments");

$keywordQuery.SelectProperties.Add("LastModifiedTime");

$keywordQuery.SelectProperties.Add("ListID");

$keywordQuery.SelectProperties.Add("ListItemID");

$keywordQuery.SelectProperties.Add("EditorOWSUSER");

$keywordQuery.SelectProperties.Add("ServerRedirectedURL");

$keywordQuery.SelectProperties.Add("ServerRedirectedEmbedURL");

$keywordQuery.SelectProperties.Add("ServerRedirectedPreviewURL");

$keywordQuery.SelectProperties.Add("SiteID");

$keywordQuery.SelectProperties.Add("SPSiteURL");

$keywordQuery.SelectProperties.Add("Path");

$keywordQuery.SelectProperties.Add("Title");

$keywordQuery.SelectProperties.Add("ParentLink");

$keywordQuery.SelectProperties.Add("IsContainer");

$searchExecutor = new-object Microsoft.SharePoint.Client.Search.Query.SearchExecutor($clientContext)

$results = $searchExecutor.ExecuteQuery($keywordQuery)

$clientContext.ExecuteQuery()

#This gives you all the documents that are shared with the user -> 'testuser@domain.onmicrosoft.com'

$results.Value

$results.Value[0].ResultRows