$mps = Get-SPEnterpriseSearchMetadataManagedProperty -SearchApplication "<SearchServiceApplicationName>" $mps.count

1. From the Search Service Administration page, click Search Schema.
2. On the Search Schema Page, Click Categories in the “breadcrumb” navigation at the top of the page.
3. Locate the SharePoint category and use the dropdown menu to select Edit category. Only 50 categories are listed per page so you may need to navigate to additional pages to location the SharePoint category.
4. Select the Searchable, Queryable, Retrievable checkboxes and click OK.
5. Locate the SharePoint category and use the dropdown menu to select Edit category.
6. Deselect the Searchable, Queryable, Retrievable checkboxes and click OK.

$mcat = Get-SPEnterpriseSearchMetadataCategory -SearchApplication "<SearchServiceApplicationName>" -Identity SharePoint $mcat.AutoCreateNewManagedProperties = $true $mcat.update() $mcat.AutoCreateNewManagedProperties = $false $mcat.update()

1. User Get-SPEnterpriseSearchMetadataManagedProperty to get the managed property you wish to remove.
2. Retrieve the mappings for the Managed Property you wish to delete using Get-SPEnterpriseSearchMetadataMapping.
3. Iterate through the mappings, removing each one using Remove-SPEnterpriseSearchMetadataMapping
4. Once all mappings have been removed, remove the Managed Property with Remove-SPEnterpriseSearchMetadataManagedProperty

Certain folders may have to be excluded from antivirus scanning when you use file-level antivirus software in SharePoint
https://support.microsoft.com/en-us/kb/952167

Which states:
These folders may have to be excluded from antivirus scanning when you use file-level antivirus software in SharePoint. If these folders are not excluded, you may see unexpected behavior. For example, you may receive “access denied” error messages when files are uploaded.

Admittedly, that statement seems a bit ambiguous, but I  whole-heartedly recommend the exceptions and hope to convince you of their need with the all-too-common scenario described below.

The [representative] scenario…

The consequence of Anti-Virus locking out files can manifest in many different ways. In some cases, the signs are obvious (e.g. the IO Exception “The process cannot access the file because it is being used by another process“), whereas others throw no particular exception, but instead exhibit terrible performance (e.g. very low documents per second processed). Further, I’ve also seen several cases where AV ultimately led to Index replicas regularly cycling through degraded states.

The point… there is no single path to failure caused by AV, but I’ll try to walk you through some representative scenarios. Conversely, other factors can cause similar performance impacts, too (e.g. saturated network, poor disk, etc)

…but for now, we’ll focus on AV (and for anyone asking… that is a great example of AV totally wrecking the performance in my lab).

Follow the money crawl flow…

As we dive in, I will refer to the following high level flow for a crawl when describing a particular point of a crawl:

The Crawl Load report (found in Central Admin -> Crawl Health Reports -> in the Crawl Latency tab) illustrates the number of items being processed by the system at any a given moment in time, whereas the Crawl Latency chart (in the same tab, but presented under the Crawl Load chart) shows you the latency for various stages. At Ignite (around the 47m 30s mark), I talked about the Crawl Load report and noted that I use it as my starting point when troubleshooting any crawl performance issue.

For the Crawl Load, I use the analogy of cars on the road. If you take an aerial photo of rush hour traffic, you would see a lot of cars on the road, but have no real measure of how fast each car is actually moving. Being said, if in that photo you could see that lots of cars where packing in between mile marker 10 and 11, then there was likely something between mile marker 11 and 12 causing a slow-down (e.g. all the cars on the road are backed up by some bottleneck further down the road).

This Crawl Load chart is similar in that we can see where the items are in the system as they are being processed. Although overly simplified, it may also help to think about this Crawl Load chart as an indicator of where the Crawl Component’s [gatherer, aka “robot”] threads are currently working (e.g. each thread is working on some batch of items at any given time, which is why the chart shows number of items in the Y axis instead of threads). To take out the previous analogy further, each car [thread] is moving a batch of people [items] down the road [crawl flow].

When a crawl is flowing well, you’d typically expect to see a roughly even split of items in the “In Crawler Queue” and “Submitted to CTS”. In other words, about half of the threads are busy gathering new items to process and the other half of threads are waiting for callbacks on items being processed.

However…

…here we see the threads are mostly pooling in “Waiting in Content Plugin” (e.g. between steps 2b and 3 from the flow illustration above) and in “Submitted to CTS” (which means the threads are waiting on a callback until the processing in steps 3 to 6b completed). Going back to the traffic analogy again, we would expect to see corresponding bottlenecks somewhere down the road, which we can see using the “Content Processing Activity” report.

The very first chart at the top of this post (e.g. the “Content Processing Activity” report dominated by purple [DocumentRetrieval] and gold [Indexing] values) best illustrates the negative impact that AV can have on crawling…

Scenario 1: DocumentRetrieval

In my post here describing the various paths used during a crawl, we can see that the Content Processing Component reaches back to the Crawl Component (via the GathererDataShare) during the DocumentParsing stage in the processing flow (during step 4 above) to retrieve an item’s blob. For example:

When AV interferes with this retrieval over the GathererDataShare file share… well, you see lots of purple [DocumentRetrieval] in the “Content Processing Activity” report.

Scenario 2: Indexing

For this particular customer, the DocumentRetrieval stage was playing nice. Instead, we found GIGANTIC delays in “Indexing” (steps 5 and 6a in the flow above)

   *(apologies for the grainy images, but that is ~170 seconds for each item)

After disabling AV, the very same system processed most items in ~200-300 milliseconds as seen below:

Into the weeds…

As I described further in this post (in the section labeled Glimpse into the Mechanics for Managing the Index about 25% into that post), the Index regularly creates, merges, and deletes many small files as part of the normal processing.

“Index files are never directly modified. Instead, when the Index parts on one level become >10% of the level above, they are “merged” into a new part file at that next level above. After the merging has completed, the parts on the lower levels (now merged into the higher level) can then be deleted from disk. During normal operation, the master part typically accounts for >90% of the active documents in the given partition (100% when everything is fully merged into the master part) and the levels below for the other ~10%.”

When AV gets involved, these low level Index files get locked – at times temporarily and sometimes indefinitely (until the noderunner.exe gets recycled) leading to longer than expected processing times (read: poor performance) and/or degraded components.

For example, exceptions such as the following commonly (but not always) occurs when AV locks out a file needed by Search:

NodeRunnerIndex10-bc8226eb-4f3e      Search      Search Component      ajkma      Unexpected
JournalReceiverWorker[SPae590e863ac2]: Generation redo failed with exception.:
System.IO.IOException: The process cannot access the file because it is being used by another process.
        at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
        at System.IO.File.InternalMove(String sourceFileName, String destFileName, Boolean checkHost)
        at Microsoft.Ceres.CoreServices.Storage.LocalDisk.LocalDiskStorage.MoveFile(StoragePath oldPath, StoragePath newPath)
        …etc…

In this particular scenario, one of the Journal files were locked out preventing the secondary replica from fully synchronizing with the primary replica. Eventually, this caused the secondary replica to get “left behind” by the primary replica where the problem manifested itself as a degraded Index component.

Keep in mind a few points here:

• The index has lots of different types of files that can be impacted (e.g. Journal files, mfs files, parts, merges, generation state files, and so on) and the exception can happen with any of them. Being said, when hunting in ULS, search for the exception (i.e. “it is being used by another process”) rather than this specific event ID.
• This exception does not always occur (so “no message” does not mean “no interference from AV”). But if it does appear in your ULS, AV is most certainly at play (and Process Monitor can help verify).
• With the redundancy built into Search, a single degraded component does not result in any sort of query outage meaning the failure may “silently” occur. In this specific scenario, the customer noticed the degraded components several hours after this event occurring in ULS.
• When this specific scenario occurred, the thread in this Index component was blocked from moving forward and thus stuck on a particular Generation. On a recycle of the noderunner.exe process (by restarting the HostController service), this secondary replica could then be re-seeded and caught up.

As noted above, the “used by another process” exception does not always occur. In other times, we may also see excessively long operations that simply time out. For example:

NodeRunnerIndex10-bc8226eb-4f3e      Search      General      ainw9      Medium
Remote service invocation: method=GetRecoveryAction(lastPreparedGeneration) Service = { 
      Implementation type=Microsoft.Ceres.SearchCore.JournalShipper.JournalShipper 
Component: SPd9a8b223b429I.2.0.JournalShipper
Exposer Name: JournalStreamControl
} terminated with exception:
      System.Threading.ThreadInterruptedException: Thread was interrupted from a waiting state.
            at System.Threading.Monitor.Enter(Object obj)
            at Microsoft.Ceres.SearchCore.Journal.Journal.get_LastPreparedGeneration()
            at Microsoft.Ceres.SearchCore.JournalShipper.JournalShipper.GetRecoveryAction(UniqueGeneration generationId)

Followed by:
NodeRunnerIndex10-bc8226eb-4f3e      Search      Search Platform Services      ajhl2      Monitorable
Microsoft.Ceres.CoreServices.Remoting.InvokerService : Service invocation timed out after 179999 ms,
method=GetRecoveryAction(lastPreparedGeneration) Service = { 
            Implementation type=Microsoft.Ceres.SearchCore.JournalShipper.JournalShipper 
            Component: SPd9a8b223b429I.2.0.JournalShipper 
            Exposer Name: JournalStreamControl
}

If you’re actually still reading at this point, the 179 seconds reported in the timeout should trigger memories of a “Content Processing Activity” chart above where the average time was ~170 seconds (aka: the wall of yellow).

The takeaway…

If you see anything like this:

Be thinking…

• Verify AV exclusions  /* this is by far the most typical root cause that I’ve seen */
• Check for failing Master Merge
• Disable TCP Chimney Offload and IP Task Offload  /* rare, but it can make an impact */
• And then dive into the other typical resources constraints (e.g. Disk, Network, CPU)

I hope this helps…

Move Admin Components

In some environments it is necessary to separate admin components and crawl components. The PowerShell commands below may be used to move the admin components to a new server. This process assumes that search components are already running on the new admin servers and also assumes some search components will remain on the old admin servers. These steps describe the general process:

1. Verify all search components are healthy
2. Add new admin components
3. Verify all search components are running
4. Remove old admin components (NOTE: Search will be offline for a period of time during this step)

Use the following PowerShell to add the new admin components. Make sure you edit the server names.

# Add new admin components

$AddAdminServer1 = "Server2"
$AddAdminServer2 = "Server1"

$ssa = Get-SPEnterpriseSearchServiceApplication

$active = Get-SPEnterpriseSearchTopology -SearchApplication $ssa -Active
$clone = New-SPEnterpriseSearchTopology -SearchApplication $ssa -Clone -SearchTopology $active

$NewAdminServer1 = Get-SPEnterpriseSearchServiceInstance -Identity $AddAdminServer1
$NewAdminServer2 = Get-SPEnterpriseSearchServiceInstance -Identity $AddAdminServer2

# start a new admin component on the new server

New-SPEnterpriseSearchAdminComponent -SearchTopology $clone -SearchServiceInstance $NewAdminServer1
New-SPEnterpriseSearchAdminComponent -SearchTopology $clone -SearchServiceInstance $NewAdminServer2

Set-SPEnterpriseSearchTopology -Identity $clone

It will take approximately 10 minutes for the add command to complete. Once it has finished, refresh the Central Admin Search Application Topology and verify that all components are running.

Use the following PowerShell to remove the original admin components. Make sure you edit the server names. (NOTE: Search will be offline for a period of time during this process)

# Delete old admin components

$RemoveAdminServer1 = "OldAdminServer1"
$RemoveAdminServer2 = "OldAdminServer2"

$ssa = Get-SPEnterpriseSearchServiceApplication

$active = Get-SPEnterpriseSearchTopology -SearchApplication $ssa -Active
$clone = New-SPEnterpriseSearchTopology -SearchApplication $ssa -Clone -SearchTopology $active

$adminIdentiy1 = (Get-SPEnterpriseSearchComponent -SearchTopology $clone | ? {($_.ServerName -eq $RemoveAdminServer1) -and ($_.Name -Like "Admin*")}).ComponentId.GUID
$adminIdentiy2 = (Get-SPEnterpriseSearchComponent -SearchTopology $clone | ? {($_.ServerName -eq $RemoveAdminServer2) -and ($_.Name -Like "Admin*")}).ComponentId.GUID

Remove-SPEnterpriseSearchComponent -Identity $adminIdentiy1 -SearchTopology $clone -Confirm:$False
Remove-SPEnterpriseSearchComponent -Identity $adminIdentiy2 -SearchTopology $clone -Confirm:$False

Set-SPEnterpriseSearchTopology -Identity $clone

It will take between 15-30 minutes for the remove command to complete. Once it has finished, refresh the Central Admin Search Application Topology and verify that all components are running.

Please leave comments and share your experience. How long did the commands take to finish? How long was the search query experience offline?

CEWS Pipeline Toolkit

CEWS Pipeline Toolkit is a content processing framework for search that supports both Content Enrichment Web Service (CEWS) in SharePoint 2013/2016 and the Pipeline Extensibility feature in FAST Search for SharePoint 2010. The tool comes with an installer and graphical user interface. Developers may configure and test pipelines using the built-in modular stages, or develop totally custom processing using Visual Studio and the included sample projects. Pipelines can be executed offline without SharePoint installed, or integrated as part of SharePoint's processing. The CEWS Pipeline Toolkit has a low overhead, is modular, and extensible. (TechNet documentation)

Steps to install

1. Install CEWS Pipeline Toolkit
2. Open a PowerShell Script as admin and cd to the “$($Env:CEWS)\scripts” directory
3. Run the script Install-CEWSWindowsService.ps1
4. Verify the windows service is running (Sp2013ContentEnrichmentWebService)
5. Edit or copy custom configuration files (CEWS.PipelineConfig.xml, Register-CEWS.config) to the CEWS\etc directory
6. Copy the custom .dll from the Visual Studio output directory to CEWS\bin
7. Run Register-CEWS.ps1 (all crawls must be stopped or paused)
8. Verify the managed properties were created as expected
9. Restart the windows service (Sp2013ContentEnrichmentWebService)
10. Start a crawl and verify the managed properties get populated 

Steps to update the configuration

Register-CEWS.config

1. Edit Register-CEWS.config
2. Run Register-CEWS.ps1 (all crawls must be stopped or paused)
3. Verify the managed properties were created as expected
4. Start a crawl and verify the managed properties get populated 

CEWS.Pipeline.config

1. Edit CEWS.PipelineConfig.xml
2. Restart the windows service (Sp2013ContentEnrichmentWebService)
3. Start a crawl and verify the managed properties get populated 
   

Steps to update DLL

1. Stop the windows service (Sp2013ContentEnrichmentWebService)
2. Copy .dll from Visual Studio output directory to CEWS\bin
3. Start the windows service (Sp2013ContentEnrichmentWebService) 

Steps to Unregister the SharePoint CEWS callout

The only reason to run the unregister command would be to stop all custom processing. Note that this command requires all crawls to be stopped.

1. Run UnRegister-CEWS.bat (all crawls must be stopped)

Helper scripts

Restart CEWS Pipeline Toolkit windows service on all servers

# restart Windows Services
#$servers = Get-Content .\serverList.txt
$servers = @("server1","server2")
$services = @("SharePoint2013ContentEnrichmentWebService")

foreach($service in $services)
{
    foreach($server in $servers)
    {
     $sc = sc.exe "\\$server" query $service
      if($sc[3].Contains("RUNNING"))
        {
            "Stopping service $service on server $server"
         $t = sc.exe "\\$server" stop $service
            sleep 1
        }
        else
        {
            "Service $service is already stopped on server $server"    
        }
    }

    foreach($s in $servers)
    {
     $sc = sc.exe "\\$server" query $service
      if($sc[3].Contains("STOPPED"))
        {
            "Starting service $service on server $server"
         $t = sc.exe "\\$server" start $service
            sleep 1
        }
        else
        {
            "Service $service is already started or starting on server $server"
        }
    }
}

It has some dust and time to blow it off.  My name is Russ Maxwell and I’m blogging again and excited to share something I’ve been working on for a few months. As you may or may not know, my colleague Brian Pendergrass developed the core of Search Dashboard. Eric Dixon and Brent Groom were also part of the team and they developed the UI portion. It was a tremendous team effort and I was glad to be a part of it. If you’re not aware of Search Dashboard and its capabilities, please review Brian’s post here.

I wanted to contribute and put my stamp on Search Dashboard. I’m excited to produce the first major enhancement to Search Dashboard with a Topology Report. This was more work than I expected and I really appreciate the contributions made by Brian Pendergrass that helped make this a really rock solid report. Internally working with our customers, We’ve often encountered Search performance issues related with Crawl, Content Processing, Indexing, or Query processing. Many times during the discovery phase of these Search performance issues, we found Search Topologies in a questionable state. At times, we made suggestions to update the Search Topology and this resulted in improved performance while optimizing for future growth.

An optimal Search Topology is one that will provide the best Crawl freshness and Query timings. The Search Topology Report is a module I developed which is an extended feature of Search Dashboard. It’s a PowerShell command-let that outputs a report based on a variety of categories. It’s all PowerShell driven and initially designed for SharePoint 2013 On-Premises environments. I’ll discuss the different categories outputted in more details below.

Topology Type Detection

Topology Type Detection determines the size of a Search Topology. This is based on how many index partitions are provisioned for the active search topology.

• Small Search Topology is detected with 1 Index Partition
• Medium Search Topology detected with (2-4) Index Partitions
• Large Search Topology detected with 5 or more Index Partitions

For Example:

Depending on what type of Topology is detected determines how the Search Topology is evaluated.

Redundancy Checks

This check ensures that all components are redundant. For Index Components, it ensures that at least two index replicas/components exist for a given index partition.

If a component fails a redundancy check, it will look like the following:

Duplicate Component Check

The duplicate component will report if more than one component of the same type is running on a given server. For Example, two Analytic Processing Components are running on Server A. In this case, Search Topology Report will flag this. In my case, I had no duplicate components running so it came back clean.

Search Component Compatibility Check

We’ve reviewed Search Topologies over the years and know what mix of components run well together and what mix of components may not run as well together. The Search Component Compatibility check evaluate components running on each server and will provide feedback if components are running together that traditionally shouldn’t be.

Search Topology and Scale Report

The Search Topology and Scale Report contains a ton of checks to evaluate a Search Topology based on the size of the Search Topology. So again, we determine the Search Topology based on how many Index partitions exists. The outline for the specific checks based on Search Topology are the following:

Small Search Topology

• Flags if running more than 2 Admin components
• Flags if running more than 2 Analytic Processing Components
• Flags if running more Crawl Components than Content Processing Components provisioned

Medium and Large Search Topology

• Flags if running more than 4 Admin Components
• Running more than 4 Analytic Processing Components
• Flags if running more Crawl Components than Content Processing Components provisioned

Question: What about Crawl and Index Component Scale?
Answer: These checks require more details which I outlined in their own sections below.

Crawl Component and Scale

First, we ensure that the number of Crawl Components doesn’t exceed the supported limit as well as the number of Crawl Databases. If either of these are exceeded, we return back without checking anything further. Assuming a Search Topology is under the supported limits with crawl components we proceed further. With Crawl Component check, more goes into checking how many components are recommended based on a Search Topology size. We must take into account the index performance level as it is the control mechanism that tells the crawler how many gathering threads are available for a given crawl. Some math is required to determine if you’re scaled properly based on the following setup:

• Index Performance Level
• # of Unique Hosts (total amount of hosts defined in all Content Sources)
• Crawler Impact Rules
• Total # of Crawl Components

Based on a Crawl setup, we established baselines using calculations. If baselines are beneath the baseline or exceed the baseline, the report will suggest the following:

1. Over Sized – Recommend a lower number of crawl components
2. Under Sized – Recommend a higher number of crawl components

Before this functionality existed this report, it was very difficult for customers to determine if a crawl setup was scaled properly. Also, many assume that the # of crawl components is based on the size of a Search Topology which is established as the number of index replicas. That is not an accurate statement.

Index Component and Scale

Quite a few checks take place within the index and scale portion. We’ve encountered several index performance issues in the past that can index both Crawl freshness and Query timings. Some of these issues were directly correlated to an Index configuration within the active Search Topology. The index component and scale portion will go through a few checks and report if irregularities are discovered. First, Microsoft does leverage a supported boundary when it comes to number of replicas based on a given Index Partition. If less than four million exists per partition, we recommend a max of six replicas per index partition. If more than 4 million items exist per index partition, we flag if more than 3 replicas per index partition is detected.

Another check evaluates if more than ten million items exist in a given index partition. If true, we flag it and provide feedback. As part of the Index Component and Scale evaluation, this command-let also test if an uneven number of index replicas exists across index partitions. That is, it scans the number of index replicas per index partition across the Search Topology and flags if an uneven number of partitions exists. For Example, if Index Partition 1 hosts two replicas while Index Partition 2 hosts three replicas it will be flagged. Another example of a check performed is if more than 25 index partitions exists in the active Search Topology, we’ll flag that the supported limit has been reached. Finally, the index component and scale portion checks if a large number of partitions with a low number, less than 50%, of active docs in each partition. This could potentially suggest an environment is over scaled and provides a heads up. However, it’s quite possible the index corpus will grow into this Index setup which is perfectly okay.

Setup Get-SRxTopologyReports

1. SRx Dashboard is required to be installed on the machine. Follow guidance here if needed.
2. Download the Enable-SRxTopologyDiag.zip file near the bottom of this blog and unzip into the SRx\Modules\Search Folder:

Note: The above screenshot shows the new module Enable-SRxTopologyDiag.psm1 is extracted.

Running SRx Topology Reports

1. Launch SharePoint 2013 Management shell and navigate to the SRx directory and run the following:

.\initSRx.ps1

2. Next, run the command-let: Get-SRxTopologyReports

Here is an example of the output:

Disclaimer:  We released the Search Health Reports (SRx) to help others get more out of Search, but the SRx should not be considered an official product . ALL information in this blog is provided "AS IS" with no warranties and confers no rights. This blog does not represent the thoughts, intentions, plans or strategies of my employer. All content is solely my opinion and provided with a best effort to be based in reality.

All examples, code samples, demonstrations, or anything resembling a “how-to” are provided "AS IS" without warranty of any kind, either express or implied, including but not limited to the implied warranties of merchantability and/or fitness for a particular purpose. Inappropriate comments will be deleted at the authors discretion.

To investigate this functionality we'll consider a hypothetical requirement to mark each document stored in a site's document library with a Color label, choosing from one of a small set of values.  We begin by creating a site column in our Office 365 SharePoint site[3] from the Site Settings page of the site collection, and add it to the group Custom Columns.  By selecting data type Choice, we can list the explicitly allowed values of either Red, Yellow, Green or Blue.

The Description section provides an opportunity to provide an explanation of what this field is to be used for, and whatever text is added here will be shown in the document upload form to help the users make an appropriate choice of value.

It is then added to the site's shared document library as a default column, via the Library Settings interface on the Library tab of the document library, by clicking the "Add from existing site columns" option.

If we then upload an existing word document to this library, the form will prompt for this new property, and supply the default value that was specified when the site column was created.

The next interesting case is to create a Word 2016 document with this metadata setting as a built-in custom property, to verify that it would be recognized during the upload to SharePoint Online and mapped to the existing site column.  Adding a custom property to a document is described in the Office.com support pages [4], and won't be spelled out here.  For our testing we created a custom property named Color of type Text in a Word document (via the Advanced Properties, Custom tab), and gave it a non-default value matching a one of the choices created in the SPO site column.  The SharePoint Online upload process did successfully read this value from the Word 2016 document, and displayed it as the default value in the upload form.

A SharePoint Online site crawl generally picks up new content within a short period of time, so after some interval the document should be returned in search results for an appropriate query.  Site column fields will be picked up in the crawl as a Crawled Property and mapped to a Managed Property[5], with both of them created automatically as necessary.  More details on how the automatically created crawled and managed property names are determined have been documented in a blog post[6], and in this specific case the property names, incorporating the field name and data type of the site column, would be:

On the SharePoint Online site a search for ColorOWSCHCS:Restricted returned the test documents uploaded with that setting, but not others with value “Confidential”.

While having this automatically created Managed Property available is helpful, it is not particularly user-friendly and  lacks the full support of a custom created Managed Property, in particular the ability to use it as a refiner.  In SharePoint Online this can be configured via the tenant's Search Schema interface by modifying one of the numerous unused predefined Managed Properties[7], giving it a user-friendly Alias matching the document property name.  In this case, we'll utilize the RefinableString00 property, giving it the Alias value Color, and map the automatically created ows_q_CHCS_Color crawled property to it.

This customization improves the metadata search experience, as shown below, and will also enable its use as a refiner in the query results.

Customization of the search results page is done by first choosing "Edit Page" from the Settings cog , then selecting Edit Web Part from the pulldown in the Refinement web part on the left side of the page in the Navigation Zone.

By clicking the "Choose Refiners" button on the resulting Properties menu, the RefinableString00 refiner is added to the "Selected Refiners" list, and given the display name Color.

After using the "Move Up" button to put this new navigator at the top of the "Selected Refiners" list in the Refinement web part, the Color refiners are displayed for search results and available to filter the results to the appropriate selection.

While the focus of this article has been on customization of SharePoint Online, the techniques described here are also  available in on-premises SharePoint installations.  For further background on aligning document properties and SharePoint columns, see the blog "Document Property Promotion and Demotion Overview and Considerations" [8].

References:

[1] "Building Block: Columns and Field Types" - https://msdn.microsoft.com/en-us/library/ee535893.aspx

[2] "Introduction to Columns" general SharePoint overview - https://msdn.microsoft.com/en-us/library/ms450825.aspx

[3] SharePoint Online "Create a site column" -  https://support.office.com/en-us/article/Create-a-site-column-844EBA84-C32E-490E-9709-FD7DC84C6914

[4] "View or change the properties for an Office 2016 file" - https://support.office.com/en-US/article/View-or-change-the-properties-for-an-Office-2016-file-672540d1-019c-46bb-8a48-2e05770c4b9b

[5] "Manage the search schema in SharePoint Online" - https://support.office.com/en-us/article/Manage-the-search-schema-in-SharePoint-Online-d4fab46d-ba41-4c03-9d4c-32b5b33198b6

[6] "From site column to managed property – What's up with that?" - https://blogs.technet.com/b/tothesharepoint/archive/2013/03/06/from-site-column-to-managed-property-what-s-up-with-that.aspx

[7] "Create a managed property by renaming an existing one" - https://support.office.com/en-US/article/Manage-the-search-schema-in-SharePoint-Online-D4FAB46D-BA41-4C03-9D4C-32B5B33198B6#__ref341260321

[8] "Document Property Promotion and Demotion Overview and Considerations" - https://blogs.technet.com/b/wbaer/archive/2014/08/29/document-property-promotion-and-demotion-overview-and-considerations.aspx

For the uninitiated, the Popular Searches web part simply displayed a number of popular search terms and provided the ability for a user to click on the term and immediately be taken to the search results page for that query. This could be placed anywhere but typically was found on the primary search entry page somewhere near the search box. In order to recreate this we have 2 challenges:

1. GET THE DATA
2. DISPLAY THE DATA

STEP 1: GET THE DATA

While Web Analytics no longer exists in SharePoint 2013, the SharePoint UI does track almost all search interactions (queries, clicks, etc). We won't get into much detail about it here (see this for more detail: https://blogs.msdn.com/b/nathan_brackett/archive/2014/07/03/search-reporting-and-sharepoint-2013-introduction.aspx), but we will be leveraging some custom PowerShell scripts found here: https://www.microsoft.com/en-us/download/details.aspx?id=43394

Once downloaded and imported, let's grab a bit of data just to make sure there are no connectivity problems. On our lab environment we can pull down 30 days of data relatively quickly. This will vary based on how much traffic your search center receives:

So far so good. Now, since we're looking to get popular searches, let's focus only on searches that are coming directly to our Search Center Site. This will immediately prune out queries that are coming in from other sites and will reduce the complexity of queries that we're running a popular analysis on:

At this point, you can either use the popular aggregation functions that exist in the script or roll your own. For the purposes of simplicity and example, we rolled our own. You can of course add any additional logic you'd like to whether it's filtering, parsing the query a bit, etc.:

A little utility function like this will group up your raw search entries on QueryHash, which is a numeric representation of the actual query term. It will then sort them out by occurrence and choose the top X searches by term. PopularSearchEntry in this case is a convenience class to store our data. The Query and Count members are relatively obvious. We'll get into why we need the Url member very soon. Let's take a look at some popular searches based on what we have so far:

Great, this is looking close to the data structure we need to put this together. Now for the Url. The Url in this case is going to be a link that lets the user go directly to a search results page. In our lab, we want that link structure to look something like this: https://nmsp01/sites/ootbsearch/Pages/results.aspx?k=sharepoint

Now that we have the data prepared, let's move on to the next step.

STEP 2: DISPLAY THE DATA

The data we are looking to display is relatively straightforward. We have a Query Term, optionally a Count, and we want to use our Url field as the link for each item. In the interest of keeping things very simple, we can use an out of the box Content by Query web part. That requires that we push our popular search data into a SharePoint list. In our lab, we created a list with a few custom fields directly on the Search Center site. Also, don't forget to disable crawling of this list as we don't really need this data being surfaced in search results.

Populating this data can be automated using whatever mechanism you prefer. PowerShell can be used either on server side or using CSOM. We just need to get the data into a list in order to display it to end users. Eventually, you'll want a list that looks a bit like this:

And finally for the end user display. Add in a Content Query web part from the Content Rollups section. You'll want to reference the list you created (in our case, "Popular Queries") and use the Query field as the title and the Link To Search field as the link. You can add this web part to any page you like. In our lab, we added it to the front page of the search center. Clicking on any of the links will take a user directly to the results page using each of the query terms presented: