Nathan's Search Blog https://docs.microsoft.com/archive/blogs/nathan_brackett/feed.xml Blog on all things Microsoft Enterprise Search (by Nathan Brackett - MS Premier Field Engineer) Wed, 17 Apr 2019 04:10:48 GMT en-US hourly 1 Recreating the Popular Searches Web Part in SharePoint 2013 https://docs.microsoft.com/archive/blogs/nathan_brackett/recreating-the-popular-searches-web-part-in-sharepoint-2013 Wed, 11 Nov 2015 10:00:00 GMT https://blogs.msdn.microsoft.com/nathan_brackett/2015/11/11/recreating-the-popular-searches-web-part-in-sharepoint-2013/ If you've upgraded from SharePoint 2010 to 2013 and previously made use of the Popular Searches web part, you may or may not have noticed that it no longer exists. In its place, there are now Popular Items and Recommended Items web parts (see: https://technet.microsoft.com/en-us/library/dn794211.aspx). While these web parts are all well and good, if you are looking to recreate the Popular Searches web part in SharePoint 2013 and don't know where to start then look no further. In this blog I'll quickly walk through how you can do this with fairly minimal customization.

For the uninitiated, the Popular Searches web part simply displayed a number of popular search terms and provided the ability for a user to click on the term and immediately be taken to the search results page for that query. This could be placed anywhere but typically was found on the primary search entry page somewhere near the search box. In order to recreate this we have 2 challenges:

  1. GET THE DATA
  2. DISPLAY THE DATA

STEP 1: GET THE DATA

While Web Analytics no longer exists in SharePoint 2013, the SharePoint UI does track almost all search interactions (queries, clicks, etc). We won't get into much detail about it here (see this for more detail: https://blogs.msdn.com/b/nathan_brackett/archive/2014/07/03/search-reporting-and-sharepoint-2013-introduction.aspx), but we will be leveraging some custom PowerShell scripts found here: https://www.microsoft.com/en-us/download/details.aspx?id=43394

Once downloaded and imported, let's grab a bit of data just to make sure there are no connectivity problems. On our lab environment we can pull down 30 days of data relatively quickly. This will vary based on how much traffic your search center receives:

 

So far so good. Now, since we're looking to get popular searches, let's focus only on searches that are coming directly to our Search Center Site. This will immediately prune out queries that are coming in from other sites and will reduce the complexity of queries that we're running a popular analysis on:

 

At this point, you can either use the popular aggregation functions that exist in the script or roll your own. For the purposes of simplicity and example, we rolled our own. You can of course add any additional logic you'd like to whether it's filtering, parsing the query a bit, etc.:

 

A little utility function like this will group up your raw search entries on QueryHash, which is a numeric representation of the actual query term. It will then sort them out by occurrence and choose the top X searches by term. PopularSearchEntry in this case is a convenience class to store our data. The Query and Count members are relatively obvious. We'll get into why we need the Url member very soon. Let's take a look at some popular searches based on what we have so far:

Great, this is looking close to the data structure we need to put this together. Now for the Url. The Url in this case is going to be a link that lets the user go directly to a search results page. In our lab, we want that link structure to look something like this: https://nmsp01/sites/ootbsearch/Pages/results.aspx?k=sharepoint

 

Now that we have the data prepared, let's move on to the next step.

STEP 2: DISPLAY THE DATA

The data we are looking to display is relatively straightforward. We have a Query Term, optionally a Count, and we want to use our Url field as the link for each item. In the interest of keeping things very simple, we can use an out of the box Content by Query web part. That requires that we push our popular search data into a SharePoint list. In our lab, we created a list with a few custom fields directly on the Search Center site. Also, don't forget to disable crawling of this list as we don't really need this data being surfaced in search results.

Populating this data can be automated using whatever mechanism you prefer. PowerShell can be used either on server side or using CSOM. We just need to get the data into a list in order to display it to end users. Eventually, you'll want a list that looks a bit like this:

 

And finally for the end user display. Add in a Content Query web part from the Content Rollups section. You'll want to reference the list you created (in our case, "Popular Queries") and use the Query field as the title and the Link To Search field as the link. You can add this web part to any page you like. In our lab, we added it to the front page of the search center. Clicking on any of the links will take a user directly to the results page using each of the query terms presented:

]]> SharePoint 2013 and CEWS: Searchable Extended File System Properties https://docs.microsoft.com/archive/blogs/nathan_brackett/sharepoint-2013-and-cews-searchable-extended-file-system-properties Wed, 21 Jan 2015 08:44:00 GMT https://blogs.msdn.microsoft.com/nathan_brackett/2015/01/21/sharepoint-2013-and-cews-searchable-extended-file-system-properties/

For more information about CEWS or the CEWS toolkit,visit this resource:

https://social.technet.microsoft.com/wiki/contents/articles/19376.sharepoint-cews-pipeline-toolkit.aspx

 

Out of the box, SharePoint 2013 takes a "best effort" approach at pulling out searchable properties from items during crawl time. This process can vary quite a bit though depending on what type of crawl you have going (SharePoint, File System, Custom, etc...). A recent request I worked on was to enhance the experience around a file system crawl. Checking the properties of a file, there are a bunch of extended properties based on the item type. For something like an image, there are some pretty common properties like Height, Width, Bit Depth, etc. Video files might have a Frame Rate or a Length. As an example, these properties are what I wanted:

 

 

After some research, it seems there are 2 major options available to pick up these properties:

  1. Use an appropriate iFilter for the types of content you are crawling
  2. Use CEWS and a little custom code

The iFilter approach is a fairly robust one. iFilters are run at crawl time to extract all kinds of properties and the logic there is shared across the system (not just for SharePoint). Using CEWS as an approach is less intrusive in this case as it will only impact your SharePoint crawl. In this article, we will take a look at the CEWS approach.

To begin, I needed to figure out how to use C# to access the properties that are available via windows explorer. It turns out that there are some COM-based hooks to get access to this information and it looks a little like this:

 Type shelltype = Type.GetTypeFromProgID("Shell.Application");
 Object shell = Activator.CreateInstance(shelltype); 
 
 FileInfo fi = new FileInfo(@"f:\fs_crawl\videos\drop.avi");
 Folder folder = (Folder)shelltype.InvokeMember("NameSpace", BindingFlags.InvokeMethod, null, shell, new object[] { fi.DirectoryName }); 
 FolderItem item = folder.ParseName(fi.Name); 
 
 for (int i = 0; i < 512; i++) 
 { 
 var dtlDesc = folder.GetDetailsOf(null, i); 
 var dtlVal = folder.GetDetailsOf(item, i); 
 
 if (String.IsNullOrEmpty(dtlVal)) 
 continue; 
 
 Console.WriteLine("{0}:{1}", dtlDesc, dtlVal); 
 }

There's a lot going on in this little snippet, so let's take it apart. The code is essentially creating a hook into the shell and giving it the path to a file (drop.avi). It is then enumerating through the details (read: properties) of the item up to offset 512. Under the hood, each property is stored at a particular offset for different file types. This could be made more efficient by specifying the offsets for known file types rather than enumerating, but not knowing those offsets initially this approach made more sense for the sake of example.

Now that I had the ability to pull properties from file system items, I cleaned up the code and created a new custom processing stage using the CEWS Toolkit. Using CEWS, there is the consideration that the file would NOT live on the same system that the CEWS Service is running on. This is alleviated by the fact that this code snippet works with a UNC path. In my lab, I run the CEWS Service under the same user account that performs the crawl. This guarantees that my CEWS stage has the same rights to access files as the crawler does.

Bringing this all together, here are the managed properties I created to support this:

 

As input, I primarily needed the Path and IsDocument. I included a check to avoid trying to get any properties from folders. As an enhancement, a trigger could be created that only allowed items into CEWS when IsDocument is true.

I created a number of new Managed Properties to hold data for image and video sizes and lengths. I made a number of these into refiners as well. Once it was all wired up and crawled, here are the populated managed properties coming back in search:

 

 

 

]]> Search Reporting and SharePoint 2013: Introduction https://docs.microsoft.com/archive/blogs/nathan_brackett/search-reporting-and-sharepoint-2013-introduction Thu, 03 Jul 2014 09:26:00 GMT https://blogs.msdn.microsoft.com/nathan_brackett/2014/07/03/search-reporting-and-sharepoint-2013-introduction/ Reference Script: https://www.microsoft.com/en-us/download/details.aspx?id=43394

This is the first post in a series of posts all about analyzing search usage in a SharePoint 2013 environment. Lots of the concepts here can apply to any search platform, but we’re taking a look specifically at how to extract the raw data from SharePoint 2013 and convert that into actionable intelligence.

Search Reporting and SharePoint 2013: Introduction

So you've just launched a brand new SharePoint 2013 Enterprise Search Center with all the bells and whistles. You've crawled your separate content sources, maybe added some new refiner options, even setup some Promoted Results via Query Rules. Surely all of those features you spent time working on will get plenty of use, right? But how can you measure that? Are users finding what they want? How can you make the most out of your investment?

Fortunately, SharePoint 2013 tracks loads of information about how your users are actually interacting with the system, including but not limited to:

  • Users and Sessions
  • User Queries
  • User Actions (clicks, hovers, etc.)
  • Result Impressions
  • Refiner Usage
  • Promoted Results, Result Blocks, and general Query Rules execution
  • Query Origin (Site and Source)
  • Client Type (REST API, UI, etc.)

All of this tracked information drives the out-of-the-box search reports, which are a great place to start if you want to check out how search is being used:

  

The basics are all here. Top Queries for popular trend tracking; Abandoned Queries for low click through queries; No Results to investigate unfulfilled queries; and Query Rule Usage to track the effectiveness of your custom managed rules. For most administrators, this is plenty of information to effectively manage your search application and it’s a general best practice to review these periodically.

Suppose though that you are looking to get into more detail. Maybe you not only want to see what queries are popular amongst your user base, but you also want to see what those users tend to click on for each particular query. Or even what they don't click on? Are users frequently skipping down to the 6th result in the set before they find what they are looking for? Are they going to the second page of results? What about abandoned searches? Are users frequently rewriting queries in the same way because they didn’t get what they expected the first time?

All of this data is but a SQL query (or two) away. There is currently no programmatic access to this data, but if you are PowerShell savvy, you can get your hands on all of it. Behold, the Link Store Database:

 

 

This database holds everything you need to generate your own enterprise search reports. It can be found in whichever farm holds your Search Service Application. There are a significant number of tables here but we will only work with a handful of them, starting with MSSQLogPageImpressionQuery:

 

 

Every search is eventually persisted in this table with a pageImpressionId that is a unique key that relates to the other tables in this database. Each row here represents a search and contains a bunch of useful details:

  • When the search happened and where it came from
  • Who performed the search and under what session
  • The raw query string
  • Did the search make use of refiners?

This information is the basis for reports on search volume (broken up by site, source, or any other facet) and query popularity, but we’ll need a few more tables to get into some more interesting reports. Let’s take a look at some of the other useful tables in here and see how we can join this together.

 

MSSQLogPageImpressionResult

Contains a reference to each result impression that came back for a particular query. Also tracks how this result was displayed to the user (result block, natural result, etc…).

MSSQLogResultDocs

Contains the URI and Title for each of the result impressions. Joining this table in gives us some human readable information about the results that were displayed for our reports.

MSSQLogPageImpressionQueryRule

Shows you which query rules (if any) were triggered during each search execution. We’ll need this if we want to be able to track the click through rates of our custom query rules.

MSSQLogPageClick

And here it is. This table holds information about what users have clicked on which results per query and what type of click it was.

 

I’ll spare you all the gory details of the actual SQL query and will instead direct you towards a set of example PowerShell scripts that will run this for you: https://www.microsoft.com/en-us/download/details.aspx?id=43394

Wrapping this all up in PowerShell lets you interact with all of this query data on a command line and in a common object format. You can dump all of the data out into CSV for consumption by some other technology or do all of your report processing right inside PowerShell and dump just the aggregated statistics instead.

A quick example:

Here we are extracting all search entries for March 5th. This particular query was for the word “test”. You can see the Site that it came from (an Enterprise Search Center in this case), the Source that was queried, the SessionId, no Refiners or Query Rules this time around, and an array of LinkStoreResult objects. This object returned and all of the ones contained inside of it were defined in our helper scripts and are just meant for example purposes. These objects are not part of the SharePoint 2013 space.

Going one step further, let’s store that entry object in a variable called $entry and take a look at Results that were actually clicked:

 

So this particular query ended up getting a Click action (that was actually a hover over) on the first result in the set (position 0). This was a natural result as well and not inside of a result block or promoted.

With this information available, we can now construct our own “Top Queries and Top Clicks” reports, joining together query popularity and what links are actually getting traffic from those popular queries. We can take a look at what results in high result positions are getting passed over the most. We can setup a report to target only specific query text (outside of popular trends). The list goes on.

In the next post, we’ll throw together an example report using these PowerShell scripts and Excel PivotTables and take a look at Popular Queries and those Popular Clicks. Also, if you’ve got any great ideas for new reports, leave some feedback!

]]>