This blog is about how to find the threshold number “N” from a baseline in your environment.

From MS Ignite 2015:

First we have to explain what that value is and why it matters. The value is a very important feeding data point when troubleshooting or during content feeding tuning. The data point is shown in graph form in Crawl Load report under the Crawl Health reports off of the SSA.

What you need to know is that having a steady or plateaued high number here can be “normal”, again it’s about finding a baseline. This is not a latency report but a count of items.

Take for example the report below where we see 4 large plateaus all just under 100K items over a 19 hour time frame during full crawls of an environment.

Yes, this is completely normal for this environment. Here we can find a solid threshold of just under 100K (as shown by the red line) for 4 hours at a time and which you can then set your threshold around that to find any anomalies.

Using this great car on a bridge analogy from my colleague Brian Pendergrass we see that it is common for this to be “backed up” during a full crawl or “rush hour”, think more in terms of how many cars are there on the bridge not how long have they been there, the cars could all be moving at 100MPH and that is ok. 

Now if your numbers are not normal and continuing to climb or plateau for long periods of time then you want to look at the “Crawler Latency” report next to see why and what it is the crawler waiting on.

For some more troubleshooting and tuning on feeding and crawling please check out Peter Dempsey’s great 3 piece blog on the subject.

Effective Search Deployment and Operations in SharePoint Server 2013

 [View:https://www.youtube.com/watch?v=D24saOZH7cE]

We will cover some architecture best practices, detailed look at query flows and the indexer. Most of the focus will be OnPrem SP2013 but there will be some info on hybrid and Azure IaaS.

 -Kris

With the introduction of the Nov 2014 CU a new search setting on the SSA level was added as a feature, AllowPartialResults is a switch to turn on partial results at a farm level. After enabling this feature, if one (or more) of the search replicas even partitions is degraded, down for maintenance or has gone down unexpectedly, end users will still receive search results from available indexer components. This is great for sites that have an SLA in place to have Search available “5 9s” and also gives better flexibility around topology architecture.

What is also worth noting is that all relevancy and refinement results are returned from any available and responding index partitions. Each and every returned result will have complete summary info for a consistent UI rendering. When this feature is enabled, a new metadata property is added in ResultTableCollection (PartialResults = true) only when partial results are returned in the backend. CSOM / REST API also pick up this new property automatically after Server OM has been changed.

Process to enable Partial Results:

•     Install and/or confirm Nov 2014 CU:       
• https://support.microsoft.com/kb/2889944

• Open a SharePoint 2013 management shell with farm access and complete the following:

$ssa=Get-SPEnterpriseSearchServiceApplication

$ssa.AllowPartialResults = $true

$ssa.Update()

 **Please note once turned on in SSA, this feature will affect all search sites.

Quick 2 minute video Demo:

[View:https://www.youtube.com/watch?v=lgronaLfp2I]

By default, SP2013 skips processing embedded objects in Office documents, this is seemingly set to off by design to save index space size. Until the Nov 2014 CU, there was no easy way to index and search embedded content of your Office documents until the iFilter file Offiltx.dll was updated to build 15 of this new CU.

Below I have created a simple process to enable embedded content within PowerPoint, but this will work for any Office document, docx, xlsx, etc.

 **Please note: enabling this will grow your index size as more content per document that contains embedded objects is indexed, also there will be a slight overhead at processing time on your content processor components, see below for more info on that.

 **Update: PDF objects embedded to office documents are not parsed and indexed in this version.

Process to enable embedded docs for .pptx files:

• Install and/or confirm Nov 2014 CU:
• https://support.microsoft.com/kb/2889944

• SP management console, run this set cmdlet after setting your SSA:
• Set-SPEnterpriseSearchFileFormatState -SearchApplication $ssa -Identity pptx -Enable $true -UseIFIlter $true

• Confirm with PowerShell command:
• Get-SPEnterpriseSearchFileFormat -SearchApplication $ssa -Identity pptx

• Stop/Start service “SharePoint Search Host Controller”
• net stop SPSearchHostController
• net start SPSearchHostController

• Crawl docs

Ex: .pptx slide with embedded Word .docx

• Search

*Bacon is indeed good

Some more technical info:

The SP2013 reg key and value for .pptx:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Office Server\15.0\Search\Setup\ContentIndexCommon\Filters\Extension
DDFE337F-4987-4ec8-BDE3-133FA63D5D85

Searching the registry for this value you will find:

HKEY_CLASSES_ROOT\CLSID\{DDFE337F-4987-4EC8-BDE3-133FA63D5D85}\InprocServer32

At path:
C:\Program Files\Common Files\Microsoft Shared\Filters\offfiltx.dll

New parser build:

With Procmon we now see ParserServer.exe process load that image into its process.

This could potentially lead to some resource overhead on your CPCs.

Procmon with filter Path contains filtx.dll:

#1 AntiVirus exceptions:

AV does a number on big blobs of data like indexes and lots of small little files being written and deleted quickly such as crawler temp directories.

Confirm that AntiVirus exceptions for the entire Search farm as well as the crawl component temp directories.

This is the list of directories and processes that should be excluded:

• C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\15\Data (and sub directories)
• C:\Program Files\Microsoft Office Servers\15.0\Data
• C:\Program Files\Microsoft Office Servers\15.0\Bin
• C:\Program Files\Microsoft Office Servers\15.0\Search\
• C:\Program Files\Microsoft Office Servers\15.0\Search\Runtime\1.0\noderunner.exe (NodeRunner process exclusion)
• C:\Users\[Search Service Account]\AppData\Local\Temp
• If you Index data -RootDirectory is customized separately from SharePoint data directory, also exclude (this is where the Index lives)

 You could also just exclude the entire Office 15 hive if that is easier for your AV software.

Some more info is on TechNet but it isn't aimed at Search

https://support.microsoft.com/kb/952167/en-us

#2 IP Offloading:

TCP/IP (Or TOE or Chimney) was a big problem for FS4SP customers where it would cause IP Sec breakage resulting in entire communications outages between nodes. With SP2013 Search there have been some recorded communications performance issues and it is recommended to turn this off for IPV4 and IPV6 via:

netsh int ipv4 show global | findstr "Task Offload"

netsh int ipv6 show global | findstr "Task Offload"

There are also some more in depth instructions for nic cards with more options to ensure that IP Offloading is completely off. The instructions below are for the FAST Search product but hold true for SP2013 as well.

 https://support.microsoft.com/kb/2570111

For some search bait here is a single ULS log that has shown up multiple times for customers with symptoms above:

failed to get default collection for application SSA System.ServiceModel.FaultException`1[System.ServiceModel.ExceptionDetail]: Object reference not set to an instance of an object. (Fault Detail is equal to An ExceptionDetail, likely created by IncludeExceptionDetailInFaults=true, whose value is: System.NullReferenceException: Object reference not set to an instance of an object.  

 at Microsoft.Office.Server.Search.Administration.SearchApi.RunOnServer[T](CodeToRun`1 remoteCode, CodeToRun`1 localCode, Boolean useCurrentSecurityContext, Int32 versionIn)   

 at Microsoft.Office.Server.Search.Administration.SearchApi.RunOnServer[T](CodeToRun`1 remoteCode, CodeToRun`1 localCode)   

 at SyncInvokeGetVersion(Object , Object[] , Object[] )   

 at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs)   

 at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc)   

 at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc)   

 at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage31(MessageRpc& rpc)   

 at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)). StackTrace:

 at Microsoft.Office.Server.Native.dll: (sig=678c0f87-966f-4d99-9c94-b49e788d2672|2|microsoft.office.server.native.pdb, offset=131CE)

 at Microsoft.Office.Server.Native.dll: (offset=21BE5)

netsh int ipv4 show global | findstr "Task Offload"

netsh int ipv6 show global | findstr "Task Offload"

*Reset index is the last resort as you must complete a full crawl of your data again and users will not be able to retrieve search results until you create a new search index.

In my cases the index is out of sync and never catches up, the index components are yellow in the topology view.

In the SP management console run:

get-spenterprisesearchstatus -SearchApplication $ssa -Detailed -Text

Name      : IndexComponent6

State     : Degraded

State     : List of degraded cells: Cell:IndexComponent6-SP34fdebde8866I.1.3(Index out of sync - catching up);

catch_up  : True

Partition : 3

Host      : SOMESERVER1

Name        : Cell:IndexComponent6-SP34fdebde8866I.1.3

State       : Degraded

State       : Components OK. Index out of sync - catching up

Primary     : False

Partition   : 3

catch_up    : True

in_sync     : False

left_behind : True

What I have found is that sometimes the reset index process hangs and there is no documentation as to what to do about it.

On the SSA you will see something like this:

Administrative status     Paused for:Index reset

Or:

Administrative status     Paused for:External request, Index reset

Or:

Administrative status     Paused for:External request

You can run this command in the SSA management console to verify if the SSA is paused. If this command returns False, the SSA is running. If this command returns True, the SSA is paused.

$ssa.IsPaused() -ne 0

 This is the most common I have been seeing with indexers out of sync and reset index stuck:

 ($ssa.IsPaused() -band 0x100) -ne 0

If True returns the search index is being deleted and you must “Wait until the search index is deleted.”

Well if your indexers are out of sync this can take many many hours or never return!

(If False you may want to look at the link at the bottom for more commands on this to see why the SSA is still paused, could be a backup or topo change, etc)

Here is the process I have been able to use to fix this issue:

#1 First try to force a resume, if this doesn’t work that’s fine and go to #2 anyways, just more of a try a last ditch force.

$ssa.ForceResume($ssa.IsPaused())

#2 On all of the nodes that contain an indexer component, stop the timer services and delete the indexes:

Stop “SharePoint Timer Service”

Stop “SharePoint Search Host Controller”

Delete data directory, by default install it will look like this:

C:\Program Files\Microsoft Office Servers\15.0\Data\Office Server\Applications\Search\Nodes\[Hex ID\IndexComponentN\storage\data\*

Start “SharePoint Timer Service”

Start “SharePoint Search Host Controller”

#3 Once they directories are deleted, run reset index again (which should complete very quickly)

For more information on reset index:

https://technet.microsoft.com/en-us/library/jj219652(v=office.15).aspx

The PowerShell to complete this is here as the UI sometimes times out:

$ssa = Get-SPEnterpriseSearchServiceApplication

$disableAlerts = $true

$ignoreUnreachableServer = $true

$ssa.reset($disableAlerts, $ignoreUnreachableServer)

if (-not $?) {

 Write-Error "Reset failed"

}

To find why the SSA is paused for other reasons check here:

Manage a paused Search service application in SharePoint Server 2013

https://technet.microsoft.com/en-us/library/dn745901(v=office.15).aspx

Question:

How do I find out which Search component under noderunner.exe is taking up all my (CPU|Memory)?

Usually after seeing this slide or something similar:

Solution #1 (aka the easy way):

Download Microsoft's SysInternals:

https://technet.microsoft.com/en-us/sysinternals/bb545021.aspx

Specifically Process Explorer or "procexp.exe":

https://technet.microsoft.com/en-us/sysinternals/bb896653

Once this is installed and running you will see the hostcontroller.exe process with the multiple noderunner.exe sub processes.

Mouse over the process that you are seeing CPU or Mem contention with and you will get your answer of which process is causing the issue.

Alternatively you can right click on the titles -> Select Columns... -> Add Command Line and you will see the same information 

*Note below the "AnalyticsProcessingComponent1" is taking up the most memory and the PID is there for reference.

Solution #2 (aka the hard way or no install way):

Using WMIC calls you can gather the info needed:

C:\>WMIC /OUTPUT:C:\ProcessListWMIC.txt PROCESS get Caption,ProcessId,KernelModeTime,UserModeTime,PrivatePageCount,WorkingSetSize,Commandline

output:

noderunner.exe               "C:\Program Files\Microsoft Office Servers\15.0\Search\Runtime\1.0\NodeRunner.exe" --noderoot "C:\Program Files\Microsoft Office Servers\15.0\Data\Office Server\Applications\Search\Nodes\991EE7\AnalyticsProcessingComponent1" --addfrom "C:\Program Files\Microsoft Office Servers\15.0\Data\Office Server\Applications\Search\Nodes\991EE7\AnalyticsProcessingComponent1\Configuration\Local\Node.ini" --tracelog "C:\Program Files\Microsoft Office Servers\15.0\Data\Office Server\Applications\Search\Nodes\991EE7\AnalyticsProcessingComponent1\Logs\NodeRunner.log"  2696875000      954486784         3212       50110312500   862552064       

Where these numbers are the performance measurements:

Memory should be rather simple, this is in bytes.

PrivatePageCount  - 954486784 (or 910 megabytes)
WorkingSetSize (Mem) - 862552064 (or 822 megabytes)

CPU, WMI tracks processor time using two different properties: KernelModeTime and UserModeTime. To track total processor time we need to add the values of these two properties. In addition, WMI tracks processor time in 100-nanosecond units. We also divide the sum of these two properties by 10,000,000, something that will convert 100-nanosecond units to seconds

KernelModeTime (CPU) - 2696875000 
UserModeTime (CPU) - 50110312500 

For a quick PowerShell example on how to convert this, visit the always great scripting guy:

https://blogs.technet.com/b/heyscriptingguy/archive/2005/09/22/how-can-i-monitor-the-activity-level-of-a-process.aspx

This is mostly information aggregator type blog but I wanted to let the SP 2013 search community know of the Office / SharePoint 2013 SP1 details. Mostly bug fixes, a good fix for FQL/KQL highlighting and one interesting bit "Search schema compression is now enabled by default to allow larger search schemas".

A detailed list of SP1 fixes is available for download.

SP1 includes all the Public Updates (PU) and Cumulative Updates (CU) that have been released since the 2013 family of products first became available, up to the December 2013 CU and January 2014 PU.

Download SP2013 SP1 and change list today!

https://support.microsoft.com/kb/2817429

This blog will help you understand where those files are and how to change them.

TechNet reports on hardware recommendations for Indexing and Analytics but nothing for Crawler temp files or how to make that change HERE:

Application servers hosting search components

Index component

                       
             80 GB regardless of the number of search components hosted on the server.

            500 GB additional disk space, preferably a separate disk volume/partition.

Analytics processing component

           
            80 GB regardless of the number of search components hosted on the server.

            300 GB additional disk space, preferably a separate disk volume/partition. This disk space is necessary for local processing of analytics data before it is written to the analytics reporting database.

Index file location, default:

C:\Program Files\Microsoft Office Servers\15.0\Data\Office Server\Applications\Search\Nodes\34357C\IndexComponentN

Analytic file location, default:

C:\Program Files\Microsoft Office Servers\15.0\Data\Office Server\Analytics_f04b3beb-ee00-4f2b-b7db-ed368ccca47f -This GUID will be different per install

Gatherer temp files were being place in two places:

C:\Program Files\Microsoft
Office Servers\15.0\Data\Office Server\Applications\gthrsvc

C:\Program Files\Microsoft
Office Servers\15.0\Data\Office Server\Applications\f04b3beb-ee00-4f2b-b7db-ed368ccca47f-crawl-0\gthrsvc  -This GUID will be different per install

This can be found in these two registry keys respectfully:

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Office Server\15.0\Search\Global\Gathering Manager] TempPath

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Office Server\15.0\Search\Components\c96ac2e3-b2c0-4abc-aabe-e61c6546d90b-crawl-0] GathererDataPath -This GUID will be different per install

Currently (As of Oct and Dec 2013 CU) the only supported way to make this change is by reinstalling SharePoint 2013 server, at install time change the location of your new drive and bring the node back into your farm. There are a few scripts showing up online that make these changes on the fly but this may create issues when updating CUs. Screenshot of what this look like at install time:

We need to break your issue down into two sides, narrow down the problem by eliminating extraneous information that likely confuse what's causing the real problem or divide and conquer as the art
of war suggests, SharePoint 2010 and FAST.

SharePoint 2010 and outside influences:

From my experiences 90+% of the root causes for content ingestion issues with SharePoint 2010 and FS4SP are related to crawler, misconfiguration/architecture, content sources / WFE, or SQL
related. Here is what I would recommend to check in order:

1. Disable TCP offloading, or ToE, on the network interface cards for all SharePoint Search Component Servers, and for all FAST Servers per:
https://support.microsoft.com/kb/2570111

Not only does this cause slow crawling but it also creates query timeouts if not disabled on the FAST search/QR nodes.

2. SQL, have your SQL admins check for any locked tables, specifically the FAST content DB and Crawler database(s), this will cause all
sorts of problems down the line, specifically huge spikes in the performance counters discussed below. Multiple instances of the crawl database are a good
idea to support scaling. It is not recommended to have more than 25 million items in a single crawler database and no more than 10 crawl databases per
Search Service Application.

https://technet.microsoft.com/en-us/library/ff599536(v=office.14).aspx –Add (or remove) a crawl database

https://technet.microsoft.com/en-us/library/hh292622(v=office.14).aspx –Best practices for SQL Server 2008 in a SharePoint Server 2010 farm

https://technet.microsoft.com/en-us/library/cc298801(v=office.14).aspx –Storage and SQL Server capacity planning

3. Exclude AntiVirus on FAST data directory:

%FASTSEARCH%\data\*

And the SharePoint Crawler gatherer temp directory:

C:\Users\WSS_Search_svc\AppData\Local\Temp\gthrsvc_SPSearch4

*This will very in terms of name obviously but is essentially the search service accounts Temp directory in its user profile

More info: https://support.microsoft.com/kb/952167

4. Is your content source fast enough to crawl? Your crawler will be starved if your content source is throttling. This is a case for redirecting crawler traffic to a dedicated WFE:

https://technet.microsoft.com/en-us/library/dd335962(v=office.14).aspx

5. Check performance on crawlers

1. Check CPU, SP crawler components will spike out a system at 80%, if this is the case, add more CPU/resources, more crawler components or more
   crawler DBs.
2. Check these perfmon variables multiple times to see what the feeding chain is doing.

Batches Total - The total number of batches that were submitted for processing.

Good for a baseline, we want to see this number progress along with Batches Success.

Batches Success - The number of batches that have been successfully indexed.

Over time we want to see this increment, if it is flat lining there is a problem, but you knew that.

Batches Failed - The number of batches that failed prior to indexing.

Failed = Bad, there must be a reason, SQL issues? It's beyond the content at this point and waiting for resources from the crawler/SQL down.

Batches Open - The number of batches that are currently being processed and that were neither successful nor have they failed.

The gatherer has collected what it needs, now we are waiting on SQL for some checks, high numbers here could potentially reflect SQL issues.

Batches Ready - The number of batches that have been queued up and are not yet submitted to FAST Search Server 2010 for SharePoint.

This number should always be low unless there is some traffic from FAST.

Batches Submitted - The number of batches that have been submitted to FAST Search Server 2010 for SharePoint but that have not yet completed processing.

High numbers here usually point to a FAST problem, see below.

FAST:

For the FAST side this is pretty similar to the legacy products ESP, many of these discussion are well documented in blogs and forums.

1. Infotool is your friend, if you need to open a support case this is the first item CTS will ask for https://technet.microsoft.com/en-us/library/ee943528(v=office.14).aspx
2. Most common issue here is IOPs, the indexers use an amazing amount of disk IO, so much so that it is recommended to place the indexer
   data directories on direct attached storage. Check Perfmon for IO problems https://technet.microsoft.com/en-us/library/cc300400.aspx
3. Document Processor count is surprising up there as well, by default this daemon is way too small. Most of my medium sized enterprise customers have 40-62 DPs, depending on their installation. https://technet.microsoft.com/en-us/library/ff381247(v=office.14).aspx
4. Check for indexing issues, Check %FASTSEARCH%\var\log\syslog from the admin node, Run “indexerinfo status –a” to make sure your indexers are healthy and turning over https://technet.microsoft.com/en-us/library/ee943511(v=office.14).aspx

Healthy example, note time_since_last_index and status, during feeding these should appear
low and doing something other than idle respectively:

<?xml version="1.0"?>

<indexer hostname="host1.contoso.com" port="13050" cluster="webcluster" column="0" row="0" factory_type="filehash" preferred time_since_last_index="3034">

<documents size="7901944935537.000000" total="10318975" indexed="10318975" not_indexed="0"/>

<column_role state="Master" backups="1"/> <index_frequence min="0.250000" max="2745.937500"/>

<partition id="0" index_id="1366738633614254000" status="indexing" type="dynamic" timestamp_indexed="1366738645" indexed_per_seconds>

<documents active="5792" total="5792"/>

....

     5. Review %FASTSEARCH\var\log\syslog indexer and contentdistributor log files for full queues

     6. Also I have seen HUGE FiXML files trying to be written, so check that as well, the indexer will choke if this is the case, you
can view this in %FASTSEARCH%\data\data_fixml\, look for the newest files being written and verify if they are <50meg.  If so you will need to make the default file
size crawled smaller and/or create an exclude for the offending site. https://technet.microsoft.com/en-us/library/ff473168(v=office.14).aspx –Create exclude crawl rules

     7. Utilize our “Fast Search Performance and capacity tuning Guide”, to reduce the “MaxSubmittedBatches” and “MaxSubmittedPUDocs”, based on your Farm configuration according to the formula detailed in the article below.  

MaxSubmittedBatches = The total number of batches from each crawl component.

MaxSubmittedPUDocs = The total number of Partial Updates (Security Changes) from each crawl
component.

Performance and capacity tuning (FAST Search Server 2010 for SharePoint)

https://technet.microsoft.com/en-us/library/gg604781.aspx

If this does not help your issue contact your local TAM, as my team has multiple services to resolve FS4SP issues as well as proactive services for your environment.