Query suggestions are suggested terms that appear in a list view below the Search box while the user is typing a query. Query suggestions are automatically generated for a query when users click a result for that same query at least 6 times. The clicked result does not need to be the same one, but can be any result that was returned for the query. Users’ clicks on results are logged and entry pairs of (query, result) are added to the Analytics Link Store DB, specifically to the MSSQLogPageClick table.

Then on a daily basis, the SharePoint timer job titled “Prepare Query Suggestions” fires off to analyze and process the logged queries and clicks to prepare query suggestions.

The default OOTB behavior for generating query suggestions depends on users' actually performing searches in the Search Center and clicking on documents. For new deployments of SharePoint, this would mean that we are going to start clean and it might take a while until query suggestions start showing up for us. Enterprises have a wide set of domain specific terms that would be very useful to be used as a seeding collection for query suggestions, or simply query phrases that are considered so important that should always be suggested to users regardless of being searched for previously by other users. Also, in upgrades, we already have a bunch of terms from query logs that we want to use in the new SharePoint 2013 solution.

So, how can we manually add query suggestions in SharePoint 2013? One way is to import a text file populated with our suggestions using the Query Suggestion Settings page under Central Admin -> Search Service Application. The procedure for using this page is described stepwise on this page on TechNet.

To automate the procedure of adding Query suggestions, we’re going to use PowerShell. There are actually two ways to do this:

1. Using the Import-SPEnterpriseSearchPopularQueries cmdlet

This allows you to import a text file containing your query suggestions and has basically the same effect as using the Query Suggestion Settings Admin page in Central Admin.

Looking more closely at this cmdlet, we investigate the following two parameters of special interest.

ResultSource: This is a pointer to a Search Result Source where your suggestions will show up to users. As an example, if you specify “Local SharePoint Results” as the result source, suggestions will only show up when searching in the context of that result source.

Web: This is the SPWeb were your suggestions will be scoped at. Say you specify the Web of your Enterprise Search Center here, suggestions will only show up when in context of that web, sub web sites or other web sites within the same site collection will not automatically inherit query suggestions from the parent site.

The main difference between using this cmdlet and the Query Suggestion Settings page for importing a text file with Query suggestions, is that suggestions uploaded using the Query Suggestion Settings page will always be scoped at a Search Service Application level, meaning that suggestions will be listed for users across all site collections and web sites using the same Search Service Application. While if using the cmdlet, you must specify a Web as the scope.

Ok, let’s look at how the Import-SPEnterpriseSearchPopularQueries cmdlet can be used.

$ssap = Get-SPEnterpriseSearchServiceApplicationProxy

$web = Get-SPWeb -Identity http://hostname/[site]

$owner = Get-SPEnterpriseSearchOwner -Level SPWeb -SPWeb $web

$manager = New-Object Microsoft.Office.Server.Search.Administration.Query.FederationManager –ArgumentList $ssap

$source = $manager.GetSourceByName("Local SharePoint Results", $owner)

Import-SPEnterpriseSearchPopularQueries -SearchApplicationProxy $ssap -Filename C:\query-suggestions.txt -ResultSource $source -Web $web

$tj = Get-SPTimerJob -type "Microsoft.Office.Server.Search.Administration.PrepareQuerySuggestionsJobDefinition"

$tj.RunNow()

My query-suggestions.txt file looks like this:

Where each line consists has this format:

[Query Suggestion Terms,Query Count,Click Count,LCID]

2. Using the New-SPEnterpriseSearchLanguageResourcePhrase cmdlet

This allows you to add a single language resource entry at a time.

The types of language resource phrases available are:

- QuerySuggestionBlockList – a query suggestion entry that will be blocked from showing up. For example secret codes, offensive words, etc…

- QuerySuggestionAlwaysSuggest – a query suggestion that will always be shown. Ie. it is not personal to specific user.

- Nickname – Used for mapping synonyms for people names.

- QuerySuggestionSubstitution – Used for mapping terms. For example, if the phrase is “car”, you can add a mapping to “car or auto”. The phrase “car” will show up as the suggestion, but when the user selects that suggestion, the actual query will be “car or auto”.

 Looking at theavailable parameters on this cmdlet, we notice the following two parameters which grab attention:

Owner: Specifies the scope at which the corresponding language resource phrase is created.

The available scopes are: SPWeb, SPSite, SPSite Subscription and SSA.

 SourceId: Specifies the Id of the Search Result Source where the language resource pharse will be applied to. This is an optional parameter, so if this is not provided when running the cmdlet, the language resource phrase will be applied to all result sources.

The difference between the two cmdlets, is that the second one does not require an SPWeb as a parameter, hence you can create Query suggestions that can be scoped at different levels. You can scope a suggestion to a site collection, web site or Search Service Application.

Following, we look at some example runs of the New-SPEnterpriseSearchLanguageResourcePhrase cmdlet.

Adding a query suggestion at the SiteCollection level, will allow the query suggestion phrase to show up under all Webs within the site collection.

If we don’t specify the SourceId parameter, the suggestions will be applied to all result sources and show up when searching under any of the verticals in the Enterprise Search Center.

$spsite = Get-SPSite -Identity https://hostname

$ssa = Get-SPEnterpriseSearchServiceApplication

$owner = Get-SPEnterpriseSearchOwner -Level SPSite -SPWeb $spsite.RootWeb

New-SPEnterpriseSearchLanguageResourcePhrase -Name "suggest me" -SearchApplication $ssa -Type QuerySuggestionAlwaysSuggest -Language "en-us" -Owner $owner

$tj = Get-SPTimerJob -type "Microsoft.Office.Server.Search.Administration.PrepareQuerySuggestionsJobDefinition"

$tj.RunNow()

Adding a query suggestion at a Web site level, will restrict the suggestion phrase to show up only under the specified web site.

$spweb = Get-SPWeb -Identity https://hostname

$ssa = Get-SPEnterpriseSearchServiceApplication

$owner = Get-SPEnterpriseSearchOwner -Level SPWeb -SPWeb $spweb

New-SPEnterpriseSearchLanguageResourcePhrase -Name "suggest me" -SearchApplication $ssa -Type QuerySuggestionAlwaysSuggest -Language "en-us" -Owner $owner

$tj = Get-SPTimerJob -type "Microsoft.Office.Server.Search.Administration.PrepareQuerySuggestionsJobDefinition"

$tj.RunNow()

 In a future post, we’ll see how suggestions are being fetched from the UI, and look at using the Search REST API to retrieve query suggestions.

This tool helps you understand and learn how the available parameters on the Search REST service should be formatted.

What does this tool offer:

• Issue HTTP GET or POST search queries.
• See how the different Query parameters are formatted.
• Authenticate using different users to debug security trimming issues.
• Use against your tenant on SharePoint Online and authenticate using your SPO User ID.

Go get it from:

https://sp2013searchtool.codeplex.com/

Big thanks go to my colleagues Dan Gøran Lunde and Murad Sæter for all the cool ideas and support.

As always, Enjoy! 

This post is a continuation to my last one describing how to POST a search request to the postquery method in the Search REST API in SharePoint 2013.

When posting the request, you get a HTTP 403 Forbidden back in response from the Search service with an error message stating “The security validation for this page is invalid and might be corrupted. Please use your web browser's Back button to try your operation again.”

The reason for getting the 403 Forbidden status is mainly because you are missing an important header in your request which is the X-RequestDigest header.

 So what is the X-RequestDigest header all about?

This header contains an essential piece of information known as the form digest. The form digest is an object that is inserted into a page by SharePoint and is used to validate client requests. Validation of client requests is specific to a user, a site and time-frame.

SharePoint relies on the form digest as a form of security validation that helps prevent replay attacks wherein users may be tricked into posting data to the server.

Where do I get the form digest value to include in my request?

As described on this page on MSDN, you can retrieve this value by making a POST request with an empty body to https://site/_api/contextinfo and extracting the value of the “d:FormDigestValue” node in the XML that the contextinfo endpoint returns.

However, I have also noticed that issuing an empty body request to the postquery method itself in the Search Service endpoint at https://site/_api/search/postquery does return the form digest value as a header within the response. So, you can first issue an empty request to postquery, extract the X-RequestDigest header value from the response and use that in subsequent requests.

I have received several question from colleagues and through this blog about using the postquery method in the SharePoint 2013 Search REST API to POST the search query.

Here, I would like to share how that can be achieved. Search in SharePoint 2013 is a key player, and many of the features in SharePoint 2013 are now based on Search as the enabling platform. In that regard, proper documentation about what can be achieved with Search is more important now than ever. Through this blog, I’ll try to share any tips or tricks I come across, and I would kindly ask you to share any suggestions or requests you may have about topics you’d like covered.

OK, enough of the talking, let’s get back to the subject of this post. As noted in my previous post “SharePoint 2013 Search REST API” that describes the different methods and parameters available on the Search REST API, the postquery method can be used to POST the query to the service to overcome URI length restrictions when using the HTTP GET based method query.

The postquery method takes a Microsoft.Office.Server.Search.REST.SearchRequest object as parameter. The SearchRequest class is an internal class in Microsoft.Office.Server.Search.dll.

The HTTP POST request body representing a SearchRequest can be formatted in JSON as follows:

One important thing to note here, is that when formatting the request as JSON, it is important to note that the Accept and ContentType headers must be set to “application/json;odata=verbose”. It is NOT enough with only “application/json”.

And here you can see a full raw HTTP POST request.

In the beginning I thought this wasn’t possible without adding custom JavaScript code to the Display Template.

However, after some digging, this task seems to be a relatively simple one that can be solved by some minor configuration changes.

Follow the steps below to enable hit-highlighting of the value of any property that is returned by search results:

1. Edit the Search Results WebPart.

2. Under Display Templates, select “Use a single template to display items”. This is not what we want… but hold on.

3. The Hit-Highlighted properties (JSON) textbox is now editable, add the MP you want hit highlighting for in there as shown in the screenshot below.

Note: the Managed Property you add in there has to be configured as "retrievable" in your Search Schema, and added to the list of properties to be retrieved in your Display Template.

 4. Click on Apply – this will save the WebPart configuration.

5. Now back to the Display Templates section, select “Use result types to display items” to make the Search Results WebPart use the Result Type configuration to select which Display Template to apply for each result item.

6. Click on Apply or OK.

7. Save the Page to stop editing.

 Now, we'll need to customize our Display Template to extract the hit-highlighted value and display that value to the user.

1. Edit the Display Template; I am going to use the Item_PowerPoint.html template as an example. This file is located at: "https://site/_catalogs/masterpage/Display%20Templates/Search/Item_PowerPoint.html"

2. Use the section below of JavaScript code to get the hit-highlighted value of the required Managed Property. Here in my example, I have configured the "Parent Link" Managed Property to be hit-highlighted.

3. Finally, we add some HTML markup to render the "Parent Link" hit-highlighted value.

The final result can be seen here. I search for "sharepoint document" and you can see how my Parent Link property gets highlighted accordingly.

And below you can download the full version of the Item_PowerPoint.html Display Template used which includes the code snippets mentioned in this post.

Item_PowerPoint_custom.html

Here is an example of query using the Search REST service in SharePoint 2013 with the client type parameter set to "custom":

https://host/site/_api/search/query?querytext='sharepoint is cool'&clienttype='custom'

The Microsoft.Office.Server.Search.Query.QueryLogClientType enumeration defines the valid values for the client type parameter.

If you specify a value that is not valid, your value will be replaced by "Unknown". You should use "Custom" in your custom search application.

So what is the Query Log Client Type parameter useful for?

When viewing Query Health Reports within the Search Service Application; you can filter reports shown according to the available values for client types. This is useful when following trends of usage and load from different search consumers.

How are the values of the Query Log Client Type parameter processed?

The Analytics service is responsible for processing the logged client type values. This happens in the timer job "Analytics Timer Job for Search Service Application Search Service Application". This timer job runs every 10 minutes and collects analytics information for the Search Service Application.

The old search.asmx SOAP web service is marked as deprecated in SP2013, and the new Search REST service is the replacement.

Location of the Search Rest service

The Search REST service is located at the following URI: https://host/site/_api/search

The Search REST API

The Search REST service exposed by the URI specified above provides the following methods or entry points:

Query Parameters

The following table lists the query parameters available for the query entry point:

Suggest Parameters

The following table lists the query parameters available from the suggest entry point:

 References:

• [MS-SRCHCSOM]: Search Client Query Protocol Specification
• Search in SharePoint 2013 on MSDN
• Microsoft.Office.Server.Search.DLL - Reflector is your best friend :).

Search is all around the place now, YEAH!

Links for downloading the bits, documentation and training.

Download Microsoft SharePoint Server 2013 Preview from here.

SharePoint 2013 System Requirements.

SharePoint 2013 Training Videos for IT Pros.

SharePoint 2013 Dev Center.

SharePoint 2013 Preview on Technet.

Sharepoint 2013 Preview on MSDN.

Build apps for Office and SharePoint.

SharePoint 2013 Preview Developer Training Videos.

SharePoint 2013 Preview Marketing Site

I'll keep updating this post with more links to training material continuously. Stay tuned!

The SharePoint Server Search Connector Framework is known to be overly complex, and documentation out there about this subject is very limited. There are basically two types of custom search connectors for SharePoint 2010 that can be implemented; the .Net Assembly Connector and the Custom Connector. More details about the differences between them can be found here. Mainly, a Custom Connector is agnostic of external content types, whereas each .NET Assembly Connector is specific to one external content type, and whenever the external content type changes, the .Net Assembly Connector must be re-compiled and re-deployed. If the entity model of the external system is dynamic and is large scale a Custom Connector should be considered over the .Net Assembly Connector. Also, a Custom Connector provides administration user interface integration, but a .NET Assembly Connector does not.

The XML File Indexing Connector

The XML File Indexing Connector that is presented here is a custom search indexing connector that can be used to crawl and index XML files. In this series of posts I am going to first show you how to install, setup and configure the connector. In future posts I will go into more implementation details where we’ll look into code to see how the connector is implemented and how you can customize it to suit specific needs.

This post is divided into the following sections:

• Installing and deploying the connector
• Creating a new Content Source using the connector
• Using the Start Address of the Content Source to configure the connector
• Automatic and dynamic generation of Crawled Properties from XML elements
• Full Crawl vs. Incremental Crawl
• Optimizations and considerations when crawling large XML files
• Future plans

Installing and deploying the connector

The package that can be downloaded at the bottom of this post, includes the following components:

1. model.xml: This is the BCS model file for the connector
2. XmlFileConnector.dll: This is the DLL file of the connector
3. The Folder XmlFileConnector: This includes the Visual Studio Solution of the connector  

Follow these steps to install the connector:

 1. Install the XmlFileConnector.dll in the Global Assembly Cache on the SharePoint application server(s)

gacutil -i "XmlFileConnector.dll"

 2. Open the SharePoint 2010 Management Shell on the application server.

3. At the command prompt, type the following command to get a reference to your FAST Content SSA.

$fastContentSSA = Get-SPEnterpriseSearchServiceApplication -Identity "FASTContent SSA"

 4. Add the following registry key to the application server

[HKEY_LOCAL_MACHINE]\SOFTWARE\Microsoft\OfficeServer\14.0\Search\Setup\ProtocolHandlers\xmldoc

Set the value of the registry key to “OSearch14.ConnectorProtocolHandler.1”

 5. Add the new Search Crawl Custom Connector

New-SPEnterpriseSearchCrawlCustomConnector -SearchApplication $fastContentSSA –Protocol xmldoc -Name xmldoc -ModelFilePath "XmlFileConnector\Model.xml" 

7. Restart the SharePoint Server Search 14 service. At the command prompt run:

net stop osearch14

net start osearch14

 8.  Create a new Crawled Property Category for the XML File Connector. Open the FAST Search Server 2010 for SharePoint Management Shell and run the following command:

New-FASTSearchMetadataCategory -Name "Custom XML Connector" -Propset "BCC9619B-BFBD-4BD6-8E51-466F9241A27A"

 Note that the Propset GUID must be the one specified above, since this GUID is hardcoded in the Connector code as the Crawled Properties Category which will receive discovered Crawled Properties.

Creating a new Content Source using the XML File Connector

1. Using the Central Administration UI, on the Search Administration Page of the FAST Content SSA, click Content Sources, then New Content Source.

2. Type a name for the content source, and in Content Source Type, select Custom Repository.

3. In Type of Repository select xmldoc.

4. In Start Addresses, type the URLs for the folders that contain the XML files you want to index. The URL should be inserted in the following format:

xmldoc://hostname/folder_path/#x=:doc:id;;urielm=url;;titleelm=title#

The following section describes the different parts of the Start Address.

Using the Start Address of the Content Source to configure the connector

The Start Address specified for the Content Source must be of the following format. The XML File Connector will read this Start Address and use them when crawling the XML content.

xmldoc://hostname/folder_path/#x=:doc:id;;urielm=url;;titleelm=title#

xmldoc

xmldoc is the protocol corresponding to the registry key we added when installing the connector.

//hostname/folder_path/

//hostname/folder_path/ is the full path to the folder conaining the XML files to crawl.

Exmaple: //demo2010a/c$/enwiki

#x=doc:id;;urielm=url;;titleelm=title#

#x=doc:id;;urielm=url;;titleelm=title# is the special part of the Start Address that is used as configuration values by the connector:

x=:doc:id

Defines which elements in the XML file to use as document and identifier elements. This configuration parameter is mandatory.

For example, say a we have an XML file as follows:

Here the value for the x configuration parameter would be x=:document:id

urielm=url

urielm=url defines which element in the XML file to use as the URL. This will end up as the URL of the document used by the FS4SP processing pipeline and will go into the ”url” managed property. This configuration parameter can be left out. In this case, the default URL of the document will be as follows: xmldoc://id/[id value]

titleelm=title

titleelm=title defines which element in the XML file to use as the Title. This will end up as the Title of the document, and the value of this element will go into the title managed property. This configuration parameter can be left out. If the parameter is left out, then the title of the document will be set to ”notitle”. 

Automatic and dynamic generation of Crawled Properties from XML elements

 The XML File Connector uses advanced BCS techniques to automatically Discover crawled properties from the content of the XML files.

All elements in the XML docuemt will be created as crawled properties. This provides the ability to dynamically crawl any XML file, without the need to pre-define the properties of the entities in the BCS Model file, and re-deploy the model file for each change.

This is defined in the BCS Model file on the XML Document entity. The TypeDescriptor element named DocumentProperties, defines an list of dynamic property names and values. The property names in this list will automatically be discovered by the BCS framework and corresponding crawled properties will automatically be created for each property.

The following snippet  from the BCS Model file shows how this is configured:

<!-- Dynamic Properties -->
 <TypeDescriptor Name="DocumentProperties" TypeName="XmlFileConnector.DocumentProperty[], XmlFileConnector, Version=1.0.0.0, Culture=neutral, PublicKeyToken=109e5afacbc0fbe2" IsCollection="true">
 <TypeDescriptors>
 <TypeDescriptor Name="Item" TypeName="XmlFileConnector.DocumentProperty, XmlFileConnector, Version=1.0.0.0, Culture=neutral, PublicKeyToken=109e5afacbc0fbe2">
 <Properties>
 <Property Name="Property" Type="System.String">xxxxx</Property>
 </Properties>
 <TypeDescriptors>
 <TypeDescriptor Name="PropertyName" TypeName="System.String, mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089">
 <Properties>
 <Property Name="PropertyName" Type="System.String">xxxx</Property>
 </Properties>
 </TypeDescriptor>
 <TypeDescriptor Name="PropertyValue" TypeName="System.Object, mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089">
 <Properties>
 <Property Name="PropertyValue" Type="System.String">xxxxxx</Property>
 </Properties>
 </TypeDescriptor>
 </TypeDescriptors>
 </TypeDescriptor>
 </TypeDescriptors>
 </TypeDescriptor>
 <!-- End of Dynamic Properties -->

Example

Say that we have the following XML file to index.

 <feed>
 <doc>
 <title>Wikipedia: Nobel Charitable Trust</title>
 <url>https://en.wikipedia.org/wiki/Nobel_Charitable_Trust</url>
 <abstract>The Nobel Charitable Trust (NCT) is a charity set up by members of the Swedish Nobel family, i.e.</abstract>
 <links>
 <sublink linktype="nav"><anchor>Michael Nobel Energy Award</anchor><link>https://en.wikipedia.org/wiki/Nobel_Charitable_Trust#Michael_Nobel_Energy_Award</link></sublink>
 <sublink linktype="nav"><anchor>References</anchor><link>https://en.wikipedia.org/wiki/Nobel_Charitable_Trust#References</link></sublink>
 </links>
 </doc>
 <doc>
 ...
 </doc>
 </feed>
 

Full Crawl vs. Incremental Crawl

The BCS Search Connector Framework is implemented in such a way that keeps track of all crawled content in the Crawl Log Database. For each search Content Source, a log of all document ids that have been crawled is stored. This log is used when running subsequent crawls of the content source, be it either a full or an incremental crawl.

When running an incremental crawl, the BCS framework compares the list of document ids it received from the connector against the list of ids stored in the crawl log database. If there are any document ids within the crawl log database that have not not been received from the connector, the BCS framework assumes that these documents have been deleted, and will attemp to issue deletion operations to the search system. This will cause many inconsistencies, and will make it very difficult to keep both  the actual dataset and the BCS crawl log in sync.

So, when running either a Full Crawl or an Incremental Crawl of the Content Source, the full dataset of the XML files must be available for traversal. If there are any items missing in subsequent crawls, the SharePoint crawler will consider those as subject for deletion, and og ahead and delete those from the search index.

One possible work around to tackle this limitation and try to avoid (re)-generating the full data set each time something minor changes, would be to split the XML content into files of different known update frequences, where content that is known to have higher update rates is placed in separate input folders with separate configured Conetent Sources within the FAST Content SSA.

Optimizations and considerations when crawling large XML files

When the XML File Connector starts crawling content, it will load and parse found XML files one at the time. So, for each XML file found in the input directory, the whole XML file is read into memory and cached for all subsequent operations by the crawler until all items found in the XML file have been submitted to the indexing subsystem. In that case, the memory cache is cleared, and the next file is loaded and parsed until all files have been processed.

For the reason just described, it is recommended not to have large single XML files, but split the content across multiple XML files, each consisting of a number of items the is reasonable and can be easily parsed and cached in memory. 

Future plans

• Generalize the connector and build a framework around it.
• Add document Processing capabilities to this connector and framework.
• Upgrade the connector to work with the next version of SharePoint.

Feedback is welcomed!

Enjoy.

XmlFileConnector.zip