Whenever I need to test a search use case in SharePoint 2013 I need a clean test environment where I'm free to experiment and do what I want. Over time I've realized that there are quite a few steps involved to create a simple test environment for search. There is just too much default crud in the product. Stuff that gets in my way and annoys me.

The goals of my test environment are:

• A new search center where I can easily change display templates without affecting anything else
• Quick refeed of a limited set of content
• A wildcard search for everything should just return content I added, and only documents
• No checkout, no draft
• No permission issues (it's only me in my local test environment)
• Reset the search navigation

1. Create a new site collection 

Sometimes I create a new site for testing, I add some content and then I realize I can't crawl only this source. I want fine grained control over what I crawl and what I query. The easiest way to crawl a limited set of SharePoint content is to create a new site collection.

• Central Admin ->  Application Management -> Create site collections
• Title:Cars
• URL: https://sp2013/sites/cars
• Template: Team Site

Next, add some content. Sticking to my theme, I search online for some car related documents that I add to my site.

 2. Start a full crawl of your new site collection

You want to start a full crawl now, so the crawl can run while doing the rest of the steps. I usually delete the default "How to use this..." document and then start the full crawl.

• Open Central Admin
• Application Management -> Manage Service Applications -> Search Service Application 1 (or the name of your SSA)
• Under Crawling click Content Sources
• Click New Content Source
• Name: Cars
• Start Addresses: https://sp2013/sites/cars
• Crawl Settings: Only crawl the Site Collection of each start address

Make sure you start a full crawl of your Cars content source. I always start crawls using a PowerShell script that monitors progress and alerts me of any errors immediately.

3. Enable SharePoint Server Publishing Infrastructure

To create a search site you need to make sure SharePoint Publishing Infrastructure is activated on the site collection level.

• On https://sp2013/sites/cars, click Site Settings
• Site Collection Administration -> Site Collection Features
• Activate SharePoint Publishing Infrastructure

4. Create a new search center site

When the publishing infrastructure is enabled, we can create our search center.

• On https://sp2013/sites/cars, click Site Settings
• Site Administration -> Sites and workspaces
• Click Create
• Title:Cars
• URL: https://sp2013/sites/cars/search
• Template: Enterprise Search Center

5. Change the default query

By default the search center will search everything. One of the design goals was to only return documents a limited set of content.

• Edit page, Search Results, Edit Web Part
• Change query
• Query Text: {searchboxquery} contentsource:Cars IsDocument:True

6. Disable annoying draft and checkin warning

 We want to be able to edit web parts and make changes quickly, without constant warnings about draft and checkout. By default you get warnings about page being in draft mode and that it's not checked in/checked out.

• On https://sp2013/sites/cars, click Edit Page
• Library Settings -> Versioning Settings
• Require content approval: No
• Create a version each time: Create major versions
• Require documents to be checked out: No

Now you can edit web parts, click save and your'e done. No more annoying warnings!

7. Remove the default navigation

By default you get search scopes for Everything, People, Conversations and Videos. Typically in a test environment you want to delete these and create your own, for testing purposes.

• Edit Page -> Search Navigation -> Edit Web Part
• Click Search Settings Page
• Delete People, Conversations and Videos
• Edit Everything

Your Search Navigation list should now look something like this.

8. Verify by searching for wildcard

Now your test search site should be free of annoying stuff that gets in your way. Search for wildcard (*) to verify that you get only regular content you added.

Ah, looks clean! I know where all of this comes from. No built-in stuff here to confuse me.

You can repeat this procedure to create multiple test sites. Typically I work with multiple customers and I want separate content and search interface per customer, and this is the easiest way to do it in my opinion.

• Check if I get any warnings or errors
• Check if the Documents Per Second (DPS) / throughput was affected
• Abort the crawl if it results in errors 

In the old days of FAST ESP I would typically start feeding some content to the index, and then monitor using the doclog tool. There is no equivalent tool in SharePoint 2013. The only thing I could find was a CrawlStatus and CrawlState set for a content source. This state and status did not seem to be live, however. In my experience they usually showed me the status of a previous crawl.

My colleague Brian Pendergrass gave me an example of how I could peek directly into the database to get the live crawl status. After some fiddling I managed to get a script working that starts my crawl and then monitors progress with some live statistics until it's done. Here's an example of the script in action:

Here's the full script (content2.ps1). It is also available for download at the end of this article.

param([string]$Name, [switch]$StartFullCrawl)

function getCrawlStatus($ssa)
{
    $ret = $null
    if ($ssa)
    {
        $queryString =  "SELECT ";
        $queryString +=   "MSSCrawlHistory.CrawlId, ";
        $queryString +=   "MAX (MSSCrawlHistory.Status) AS Status, ";
        $queryString +=   "MAX (MSSCrawlHistory.SubStatus) AS SubStatus, ";
        $queryString +=   "MAX (MSSCrawlHistory.StartTime) AS CrawlStart, ";
        $queryString +=   "SUM (MSSCrawlComponentsStatistics.SuccessCount) as Successes, ";
        $queryString +=   "SUM (MSSCrawlComponentsStatistics.ErrorCount) as Errors, ";
        $queryString +=   "SUM (MSSCrawlComponentsStatistics.WarningCount) as Warnings, ";
        $queryString +=   "SUM (MSSCrawlComponentsStatistics.RetryCount) as Retries ";
        $queryString += "FROM ["+$ssa.SearchAdminDatabase.Name+"].[dbo].[MSSCrawlHistory] WITH (nolock) ";
        $queryString += "INNER JOIN ["+$ssa.SearchAdminDatabase.Name+"].[dbo].[MSSCrawlComponentsStatistics] ";
        $queryString += "ON MSSCrawlHistory.CrawlId = MSSCrawlComponentsStatistics.CrawlId ";
        $queryString += "WHERE MSSCrawlHistory.Status Not In (5,11,12)";
        $queryString += "GROUP By MSSCrawlHistory.CrawlId";
        
        $dataSet = New-Object System.Data.DataSet "ActiveCrawlReport"
        if ((New-Object System.Data.SqlClient.SqlDataAdapter($queryString, $ssa.SearchAdminDatabase.DatabaseConnectionString)).Fill($dataSet))
        {
            $ret = $dataSet.Tables[0] | SELECT CrawlId, CrawlStart, Status, SubStatus, Successes, Errors, Warnings
        }
    }
    return $ret
}

function monitorCrawlStatus($ssa)
{
    $cs = getCrawlStatus($ssa)
    if ($cs -eq $null)
    {
        Write-Host  "No active crawls"
    }
    else
    {
        Write-Host "Monitoring crawl status"
        while ($cs = getCrawlStatus($ssa))
        {
            $ts = $cs.CrawlStart
            $tnow =  [system.timezoneinfo]::ConvertTimeToUtc((Get-Date))
            $tdiff = $tnow - $ts
            $seconds = $tdiff.TotalSeconds
            $docs = $cs.Successes
            $dps = 0
            if ($seconds -gt 0)
            {
                $dps = $docs / $seconds
                $dps = [Math]::Round($dps, 2)
            }
            Write-Host ("CrawlId:" + $cs.CrawlId + ", Status: " + $cs.Status + ", Successes: " + $cs.Successes + ", Warnings: " + $cs.Warnings + ", Errors: " + $cs.Errors + ", (" + $dps + " DPS)")
            Start-Sleep 5
        }
    }
}

$ssa = Get-SPEnterpriseSearchServiceApplication
if ($ssa)
{
    if ($Name)
    {
        $contentSource = Get-SPEnterpriseSearchCrawlContentSource -SearchApplication $ssa | ?{ $_.Name -eq $Name}
        if ($contentSource)
        {
            if ($StartFullCrawl)
            {
        
                Write-Host "Starting full crawl"
                Write-Host "Content source $Name"
                $contentSource.StartFullCrawl();
                
                # Let the crawl get some time to start before we monitor it
                Start-Sleep 10
                monitorCrawlStatus($ssa)
            }
            else
            {
                $contentSource
            }
        }
        else
        {
            Write-Error "Can't find content source with name $Name"
        }
    }
    else
    {
        Get-SPEnterpriseSearchCrawlContentSource -SearchApplication $ssa        
        Write-Host ""
        Write-Host "Usage: content.ps1 [-StartFullCrawl] [-Name <contentSource>]"
        
        if ($MonitorActiveCrawls)
        {
            monitorCrawlStatus($ssa)
        }
    }
}
else
{
    Write-Error "Can't find SSA"
}

content2.ps1

I decided to test using the standard books.xml example file found on MSDN. This is a single file with multiple books in it. I split this into one file per book, but I kept the surrounding catalog tag.

Test file 

Create a folder with test files 

I created 4 separate XML files in a regular folder. I used the ID of the book as the filename.

Share the folder

This is my favourite technique for testing something quickly. Dump some files in a folder, share with Everyone (to avoid boring security issues) and start crawl.

Add the content source and crawl.

In Search Administration, Content Sources I create a File Share called xmlbooks. I add the path to my content source start address. Since my test server is called sp2013 the path is file://sp2013/xmlbooks.

 Check the crawl log

 Next I check the crawl log. I always do this when I crawl something new. After all, I don't know what to expect. To my surprise I got 5 success, 0 errors.

Test search

At this point I'm curious about what search actually will return. For this kind of testing I always prefer to use the REST API directly. The SharePoint 2013 Search Query Tool from Nadeem Ishqair is my weapon of choice. I do a simple search for xmlbooks, and I get my 4 documents and their folder in return. That's good. They are actually in the index at least.

What parts of an XML file is searchable out of the box?

Finally we need to answer the question, what is searchable? It turns out it works exactly like I would expect.

• XML tags and attributes are not searchable
  I search for author and catalog and get 0 hits.
• XML data is searchable
  I can search for the names of authors, descriptions of books.
• Teasers in the default search interface show an extract of all the XML data.

Mapping XML to managed properties

Usually you want to make a more intelligent search application than this. For example you want to use a list of XML files with orders, people, books or whatever as input data for your search system. You want to have refiners on price, date, author and genre for your books. And you certainly don't want users to open an XML file when you click on the link in the search results.

What you need is a way to map XML properties to your managed properties. You can do this either via a BCS Connector. Check out Nadeem's XML Connector for SharePoint 2010 or the SharePoint 2013 version on Anders Fagerhaug's blog.

Note: If you don't want to create a BCS Connector, you could always use the Content Enrichment Web Service in SharePoint 2013 and map XML properties to managed properties there. Maybe I'll make a post on this some time in the future.

What's the average size of an office document?

Whenever I ask customer what kind of content they have, the answer is always the same. "Just regular content, very average stuff". In most Enterprise Search scenarios this actually means they have no idea.

So what is average content? We think we know. But, do we have any actual data supporting our thoughts? I decided to collect data from a couple of real-life scenarios. In one case we have about 100 different data sources spread across tens of millions of searchable items. In the other case we have more or less a single content source (SharePoint) with hundreds of thousands of items.

Distribution

This is how the content is distributed across the different types of content. Some observations:

• Most web content is smaller than 200kB
• PowerPoint and PDF consume the most space
• Word documents are most frequent

Conclusion: The average size of an office document is 321 kB.

The next time you need to do some search engine sizing you have some data to back you up. For example you know that:

• 1 million documents consume about 306 GB, or 2.98 TB for 10 million documents
• 40% of space will be used to store PDF documents

Note that the space requirement is for the source content only. In addition you will need space for your index. For example the footprint of an JPEG image in the index will typically be 3% of the image size. For a PDF the average footprint is about 35%. But, more on this in a later post.

We can render content in any way we want by modifying the XSLT stylesheet for the core results web part. Lets edit the style so JPG images include a preview. What we want is something like this:

1. On the search results page, click Edit Page

2. Click Edit Web Part on the Search Core Results web part




3. Click the XSL Editor button to edit the style sheet

4. Copy the stylesheet to your favorite text editor and make the following changes (marked in yellow). Download the full stylesheet here.

style.xsl

1. What image formats are supported?
2. How is image metadata mapped by default?
3. How are updates handled? Will images be reindexed when metadata changes?
4. Will search recognize text and characters in the image (Optical Character Recognition/OCR)?
5. How big can pictures be?

If you are feeling lazy, jump to the end to see the answers.

Metadata

Image files usually have a date and time from when they were taken and possibly some technical information like focal length and other photo geek information. Camera devices with GPS usually store some geographical information about where the image was taken. In Windows, do right-click and properties on an image file to bring up the metadata details.

In this example my image is a screenshot from a page in some documentation. It had very little metadata, so I added a title, subject, tags and some comments. Let's see how it is indexed.

Indexing test files with docpush

Whenever I test search features I use docpush to index content. Unless you are indexing directories with lots of files this is far quicker than using the SharePoint crawler features. I usually include the -C LIVE option to wait for the callback telling me that the content is in the index.

My example files are on a share available to everyone.

docpush -c sp -C LIVE \\danglsp\share\TestImages\example.tif

I get a warning about not having the advanced filter pack enabled, but the content is still indexed.

Let's see what was indexed. In my test environment I have enabled the brilliant show all stylesheet to what is actually indexed. Another way to inspect what gets indexed is to run psctrl doctrace on before you run docpush. Then run doclog -a to see what happened in the processing pipeline. Yet another way is to add a Spy stage to the end of your pipeline.

Well, this isn't looking too good. The title is simple a link to my image. I was expecting it to be the title I set in my metadata. Also, I don't see the subject, tags or any other information from the metadata. The reference to chowinery.com is just a default prefix from docpush, so I don't care about that now. Seems like this was just indexed as a file. Let's enable the advanced filter pack to see if we can get some more metadata indexed.

Testing with Advanced Filter Pack enabled

Let's repeat the test with advanced filter pack enabled. This enables the search engine to get more content from hundreds of filetypes.

cd c:\FASTSearch\installer\scripts

.\AdvancedFilterPack.ps1 -enable

Now repeat the docpush command from above.

 Looking good. No more warning. Searching for it again we see that it is looking a lot better. Our metadata is indexed.

For a full list of supported image file types, see the C:\FASTSearch\etc\formatdetector\converter_rules.xml file. It includes your normal JPEG/JPG, BMP, GIF and TIFF in addition to the more obscure Corel Draw, g3fax and even Harvard Graphics format whatever that is. Note that certain file types are excluded from the index by default. Make sure you enable indexing JPEG and JPG file extensions in Central Administration -> Content SSA -> Manage File Types.

This simple example is not a full image search application. We still need to preview images and do some simple redesign. This is covered in a separate post Tutorial: Render images in search results with FAST Search for SharePoint 2010.

Summary

Let's review the questions from the start of this article.

1. What image formats are supported?
   Basically any file format you have heard about, and then some.
2. How is image metadata mapped by default?
   From our example we saw the default mapping:
   
   title -> title
   subject ->
   rating ->
   tags ->
   comments -> description
   
   
   
   
3. How are updates handled? Will images be reindexed when metadata changes?
   Yes. Changing the metadata will force the image to be reindexed on the next incremental or full crawl.
   
   
4. Will search recognize text and characters in the image (Optical Character Recognition/OCR)?
   Yes, you can do OCR for TIFF files in FAST Search for SharePoint.
   
   
5. How big can pictures be?
   Very big. Pixel data is not stored in the index either, so big pictures are no problem. Still normal file size restrictions apply. Read an interesting discussion about max file size in FAST Search for SharePoint in the MSDN forums.