Share via

Customize the SharePoint 2013 search experience with a Content Enrichment web service

  • Article

Did you ever wish you had more control over how your content is indexed and presented as search results? In SharePoint 2013, you have the option to add a callout from the content processing component to your own web service so you can manipulate the content before it gets indexed. This makes it possible to create new ways to pivot over the search results. Here are some examples of what you could do:

  • Normalize the case for refinable managed properties to make the refiners look cleaner.

  • Create new refiners by extracting data from unstructured text.

  • Calculate new refiners based on managed property values.

This feature is similar to the pipeline extensibility stage in FAST Search for SharePoint 2010, but there are some differences:

  • In FAST Search for SharePoint 2010, the code was executed in a sandbox, whereas SharePoint 2013 lets you put the code in a web service. This reduces the overhead and lets you run the code anywhere.
  • The web service processes and returns managed properties, not crawled properties. Managed properties correspond to what actually gets indexed and are therefore easier to manage.
  • You can define a trigger to limit the set of items that are processed by the web service. This will optimize the overall performance if you only want to process a subset of the content.

In this blog post, we’ll go through a basic scenario where we have a list of popular movies. The metadata is not consistent with regards to casing, and there is limited metadata to use for pivoting over the search results.

We will walk you through how you can go from this search experience …

… to the one below, without modifying the original content. Notice that the list Title and the Director refiner have been normalized to title case, and that there is now a new refiner called “YearsSinceRelease”.

 

To understand what’s possible to achieve with the content enrichment web service, it is important to have a good conceptual understanding of what goes on during content processing, and where the web service callout takes place in relation to everything else.

This blog post goes into detail on what happens during content processing and how the web service callout fits into the overall picture. We’ll then show you how to create a sample list and create a web service that manipulates the data in the list to create new search refiners.

What happens during content processing

The content processing component receives crawled Properties from the crawler component and outputs managed properties to the index component, but what goes on in-between?

Inside the content processing component, there are “flows” that process one crawled item at a time. When an item has been indexed, a callback is sent back to the crawler to acknowledge whether the item is searchable or not. The success or failure of an item can then be inspected by the administrator in the Search Administration (navigate to Crawl Log and then to Error Breakdown).

The flow is the specification for how the crawled item should be processed to prepare it for indexing. The flow has branches that handle different operations, like inserts, deletes, and partial updates. The branches that handle deletions and partial updates do not have a web service callout.

 

The main branch of the flow handles insertion of new and updated documents and contains different stages that extract information from the crawled item and create managed properties.

At the start of the flow, new crawled properties are registered in the Search administration database. This is followed by a stage that parses binary document formats such as Office or PDF documents. During document parsing, there is a callout to IFilters for certain document types. Document parsing adds new crawled properties from the parsers. A crawled property cannot be mapped to a managed property until it has been registered in the admin database. After all crawled properties have been collected, another stage maps crawled properties to managed properties according to the Search schema. All stages after this one work on managed properties only.

The next stage processes security descriptors and converts them to the internal format used by the index component. Automatic Language detection takes place before the web service callout, and sets the value of the languages managed property. After the web service callout, there is processing related to people search that creates phonetic name variations.

Next, word breaking is done on all managed properties that are marked for word breaking in the Search schema. This is followed by entity extraction and other stages, like metadata extraction and document summarization. Links that are discovered within the document are written to the Analytics reporting database for later processing. Finally, the managed properties are indexed.

To summarize, it is important to note that the web service callout can only read managed properties. Any crawled property value that the web service needs as input must first be mapped to a managed property. The web service callout can only Access managed properties that exist before the web service callout, and not managed properties that are set further down in the flow. The web service callout can pass managed properties back to the flow, but only if they are a part of the Search schema.

The “Popular Movie” web service

Let’s go through a full scenario that covers everything from creating the content and the web service, to customizing the search experience with new refiners. The first thing we’ll do is to create a list of some all-time popular movies with information about the release year, director, and whether the film received an Oscar. We’ll then create a web service that does a couple of things to improve the search experience:

  • Calculates the years since the movie was released by looking at the ReleaseYear managed property

  • Normalizes the Director managed property to title case

  • Creates a time stamp for when the web service processed the item

These are the steps that we’ll walk you through:

  • Create a sample list

  • Create the web service

  • Configure crawled and managed properties

  • Crawl and search the content

Create a sample list

Open the SharePoint 2013 Management Shell as administrator. Make sure that the user you are logged on as is a member of the WSS_ADMIN_WG
group, because we need write access to SharePoint resources.

Type the following commands to create a new list:

 $SPWeb = Get-SPWeb https://myserver

$CustomListTemplate = $SPWeb.ListTemplates | where {$_.Name -eq "Custom List"}

$listUrl = "PopularMovies"

$description = "A list of popular movies"

$lists = $SPWeb.Lists

$lists.Add($listUrl, $description, $CustomListTemplate)

You now can browse to your list by going to https://myserver/lists/PopularMovies

 $list = $lists | where {$_.Title -eq "PopularMovies"}

Below are the commands to create the different fields (columns) in the list.

 $spFieldType = [Microsoft.SharePoint.SPFieldType]::Text
$list.Fields.Add("Director", $spFieldType, $false)
$spFieldType = [Microsoft.SharePoint.SPFieldType]::Integer
$list.Fields.Add("ReleaseYear", $spFieldType, $false)
$spFieldType = [Microsoft.SharePoint.SPFieldType]::Boolean
$list.Fields.Add("WonOscar", $spFieldType, $false)
$spFieldType = [Microsoft.SharePoint.SPFieldType]::DateTime
$list.Fields.Add("ReleaseDate",$spFieldType, $false)
$list.Update()

If you want the newly created fields to be part of the default list view, you can add them to the default view. This is an example of how we did it for the Director field.

 $spView = $spWeb.GetViewFromUrl("/Lists/PopularMovies/Allitems.aspx”)
$spfield = $list.Fields["Director"]

$spview.ViewFields.Add($spfield)

$spview.Update()

Now we are ready to start populating the list. Each list item represents a movie, and we start off with “Pulp Fiction”.

 $spListItem1 = $list.AddItem()
$spListItem1["Title"] = "Pulp Fiction" 

  

 $spListItem1["ReleaseYear"]=1994 

  

 $spListItem1["Director"]="Quentin Tarantino" 

  

 $date = Get-Date "10/21/1994" 

 $spListItem1["ReleaseDate"]=$date 

  

 $spListItem1.Update()

 

 

 

The screen shot below shows the list after we have added a few more movies:

 

The Title and the Director have been added with inconsistent casing to show how we can fix this automatically in the web service.

Create the web service

Now let’s create a web service that can read the list we created, and create some new managed properties that we can use as refiners.

 using System; 
using System.Collections.Generic; 
using System.IO; 
using Microsoft.Office.Server.Search.ContentProcessingEnrichment; 
using Microsoft.Office.Server.Search.ContentProcessingEnrichment.PropertyTypes; 
using System.Globalization; 
using System.Threading; 

namespace PopularMovieService { 
public class PopularMovieService : IContentProcessingEnrichmentService { 
// Define variables to hold the managed properties that the 
// web service will populate. 
private Property<Int64> NewIntegerMP = new Property<Int64>(); 
private Property<DateTime> NewDateTimeMP = new Property<DateTime>(); 
private readonly ProcessedItem processedItemHolder = 

 new ProcessedItem { ItemProperties = new List<AbstractProperty>() }; 
public ProcessedItem ProcessItem(Item item) { 
// Iterate over all managed properties passed to the web service. 
foreach (var property in item.ItemProperties) { 
var s = property as Property; if (s != null) { 
// The value of the new text managed property is the 
// string in title case. 
CultureInfo cultureInfo = Thread.CurrentThread.CurrentCulture; 
TextInfo textInfo = cultureInfo.TextInfo; 
string normalizedString = textInfo.ToTitleCase(s.Value.ToLower()); 
s.Value = normalizedString; 
processedItemHolder.ItemProperties.Add(s); 
} 
var l = property as Property; if (l != null) { 
// The value of the new integer managed property the 
// number of years since the release date. 
int CurrentYear = DateTime.Now.Year; NewIntegerMP.Name = "YearsSinceRelease"; 
NewIntegerMP.Value = CurrentYear - l.Value; 
processedItemHolder.ItemProperties.Add(NewIntegerMP); } 
// Set the time for when the properties where added by the 
// web service. NewDateTimeMP.Name = "ModifiedByWebService"; 
NewDateTimeMP.Value = DateTime.Now; 
processedItemHolder.ItemProperties.Add(NewDateTimeMP); } 
return processedItemHolder; } 
}}

Configure crawled and managed properties

When you create a list in SharePoint, the column names are picked up by the crawler as crawled properties. The name of the crawled property is the same as the list column name, but with “ows_” in front, so “ReleaseYear” becomes “ows_ReleaseYear”. There are several ways a new crawled property can be registered as part of the Search schema: by crawling or by adding it programmatically, for example, through a Windows PowerShell cmdlet. In this blog post, we stick to Windows PowerShell whenever possible.

$cp = New-SPEnterpriseSearchMetadataCrawledProperty -SearchApplication $ssa -Category SharePoint -Name "ows_ReleaseYear" -IsNameEnum $false -PropSet "00130329-0000-0130-c000-000000131346" –VariantType 0

$mp = New-SPEnterpriseSearchMetadataManagedProperty -SearchApplication $ssa -Name "ReleaseYear" -Type 2 -Queryable $True

$mp.Refinable = $True

$mp.Update()

New-SPEnterpriseSearchMetadataMapping -SearchApplication $ssa -ManagedProperty $mp -CrawledProperty $cp

We need to do the same for the other managed properties (Director, WonOscar, and ReleaseDate). For easy reference, these are the types to use when creating the managed property: Text = 1, Integer = 2, DateTime = 4, and Boolean (YesNo) = 5.

We also need to create the new managed properties that the web service populates:

$mp = New-SPEnterpriseSearchMetadataManagedProperty -SearchApplication $ssa –Name "ModifiedByWebService" –Type 4 –Queryable $True

$mp.Refinable = $True

$mp.Update()

$mp = New-SPEnterpriseSearchMetadataManagedProperty -SearchApplication $ssa –Name "YearsSinceRelease" –Type 2 –Queryable $True

$mp.Refinable = $True

$mp.Update

Crawl and search the content

Before starting the crawl, we must first enable and configure the web service callout. This is done in Windows PowerShell.

$config = New-SPEnterpriseSearchContentEnrichmentConfiguration

$config.Endpoint = "https://localhost:817/PopularMovieService.svc"

$config.InputProperties = "Director", "Title", "ReleaseYear"

$config.OutputProperties = "Director", "Title", "YearsSinceRelease", "ModifiedByWebService"

Set-SPEnterpriseSearchContentEnrichmentConfiguration -SearchApplication $ssa -ContentEnrichmentConfiguration $config

We are now ready to kick off a full crawl of the list we created, and observe what gets sent to the web service.

If you don’t already have an Enterprise Search Center, you can create one using the site collection template called “Enterprise Search Center” under the Enterprise tab.

To make it easier to see the effect of the web service, we’ve customized the search center to show refiners for the managed Properties that the web service modifies.

To show the new refiners, we will edit the result template in the search site collection. First we do a query to see the result page with refiners. If we search for “Pulp Fiction”, this will show a refiner for Modified Date, which is one of the out-of-the box refiners. Click the tools icon at the upper-right, and then click Edit Page.

Click Edit webpart for the Refinement Web Part. This brings up a configuration panel, which lets you choose refiners for the Web Part. Add all the new managed properties as refiners.

 

 

We now need to check in and publish our changes.

If we now try out a query for “ReleaseYear>0”, we get back all three results with refiners, just like the screen shot at the beginning of the blog post.

 

Looking for more information?

Check out the MSDN documentation for content enrichment . There is also a How to article that gives you an example of how to create a content enrichment web service in Visual Studio.

Finally, look out for our upcoming blog posts on that will provide best practices for debugging the content enrichment service and show you how to use WCF Routing to do content-based routing and load balancing.

Comments

  • Anonymous
    February 20, 2013
    Hi, I tried to do the same following the article. But it is not working for me. The Managed Properties are not passed to the content enrichment wcf service. Am I missing something? Thanks Sagar

  • Anonymous
    February 26, 2013
    The comment has been removed

  • Anonymous
    March 11, 2013
    The comment has been removed

  • Anonymous
    March 21, 2013
    Hi Kathrine, Thanks for the reply. Even I also tried with the method u suggested. But that did not help either. May be I am doing something wrong. I will look into it. Thanks again.

  • Anonymous
    May 09, 2013
    Great Article! Can we update a column of the item passed in the ProcessItem with a value so that it can be used for other purposes?

  • Anonymous
    September 05, 2013
    The best overall walk-through I've seen yet. Thank you for this offering.

  • Anonymous
    September 17, 2013
    Excellent article! One question, how do you handle more than one search applications?

  • Anonymous
    December 11, 2013
    Sorry I don't check in here often enough. I am still working on content processing in SharePoint. @Harish: when you say "update a column of the item passed in the ProcessItem with a value", do you mean a) updating eg a list item in SharePoint, or b) updating the item values that get indexed? If you mean a), then you could use the List OM to update the original list item in SharePoint given that you write the necessary identifiers to the web service to enable you to locate the list item in SharePoint, if you mean b) then yes you can modify the value and pass it back to the processing flow. The value that you created in the web service is then the value that that will be indexed.

  • Anonymous
    December 11, 2013
    @Miran: not sure exactly what you mean by handle more than one search application. Do you mean several SSAs talking to the same web service? It is possible to configure several SSAs to talk to the same web service endpoint, you just need to make sure the endpoint can handle the load. See blogs.msdn.com/.../content-enrichment-service-scaling-and-aggregation.aspx for how to implement load balancing on the endpoint.