There is link at the bottom, saying “More”, but clicking this one never yield much result.

The solution is fortunately not far away. Just click “File > Options > Search” and uncheck “Improve search speed by limiting the number of results shown”, like this:

Now, if you go back and do the same result, you should get more results!

One last thing: You may still get the “More” option, but now it refers the messages not cached locally on your machine. Starting from Outlook 2013, the client will by default only cache messages for the last 12 months, but this is easily configurable, as well. See this support article for details: https://support.microsoft.com/kb/2733062

Happy searching!

For the last seven years, since I joined Fast Search & Transfer in 2005, search technology has been my core interest and expertise. It has been a fun journey, bringing me from Melbourne to San Francisco, via Amman and Paris. But just as rewarding, is the journey from novice to expert – from seeing search as “black box” magic, to presenting search technology at conferences like TechEd and The SharePoint Conference.

But, there is a time for everything. And as Office 2013 is appearing, making search more accessible to everyone than ever before, I am leaving the stage. I was lucky enough to be offered a new position here at Microsoft, this time in the group named “Developer Platform Evangelism”. Here, I will be a Technical Evangelist, focusing on helping developers build great apps for Windows 8.

This new role looks like it will fit perfectly with my interests and skills. Design and user interfaces have always been a very interesting topics in my view. At the same time, “Once a developer, always a developer” – nothing is like opening your favorite editor, writing a few lines of code, compile, run and then viewing your creation come alive on the screen. It was great fun with Basic on my Commodore 64, and is way more elegant and intuitive using JavaScript or C# to create blazing apps for Windows 8.

While I will still reply to the odd search question now and then, I look forward to engaging with a new community. Hopefully, I will also gain insights that I find worth sharing on this blog. And if you have any question related to Windows 8 apps, whether it’s how to get started, design discussions or how to get published in the Windows store – please reach out to me, and I will doing whatever I can to help you out.

Thanks for ride, and come join the next wave – Windows 8!

Have a great day everyone!

Background

In our project, we are exposing the same SharePoint farms both outside and inside the company firewalls. For this, we are using Alternate Access Mapping (AAM) with a setup like this:

IMPORTANT NOTE: A dedicated services farm was used for crawling the team sites. As we later understood, it otherwise would have worked. Read on the see why.

Problem

In this scenario, the contextual search features, e.g. “This Site” and “This list” don’t work when accessing the site externally. The users never got any results:

But the user who accessed via the “Default” zone got results:

And, importantly, the external users got search results if the search via the “Search Center”. It was only the contextual search which failed.

Solution

With help from my colleague, Jon Waite, I understood that the problem is due to the dedicated services farm not being aware of the AAM settings on the farm that is crawled. Fortunately, SharePoint has provided an easy solution for giving the crawling farm a hint about this, via “External Resource Mapping”. This is the one link under “Alternate Access Mapping” which I previous never understood the purpose of:

If you click on this link, you get a description which is partly misleading:

It says it can be used for mapping “a resource outside of SharePoint”. More correct would be to say “outside this SharePoint farm”, i.e., on a remote farm, which is exactly what we need. To use it, we fill in values as in the example above, click “Save”. Then click “Add Internal URLs”, where we specify the “Extranet” URL. Again, somewhat confusing naming. The end result should look like this:

And that’s all you need to do! No recrawl or other updates are needed. I love it when things just work!

I thought there had to be a solution to this, and finally the solution struck me. It was a matter of changing the federation location in a particular way. First of all, an example, using the downloadable VM image with SharePoint and FAST. If you search for “contoso”, you get this result:

We want to avoid this, and just get people matches in names here, as shown here:

To do this, we go into Central Administration and navigate to the search application and then “Federated Locations”, as shown here:

Change the “Query Template” under “Location Information” from

{searchTerms}

to

{searchTerms} AND (FirstName:{searchTerms} OR LastName:{searchTerms})

Click OK, and you’re done.

NOTE 1: This syntax seems a little more verbose than necessary, but I found that without the initial “{searchTerms}” entry, nothing was returned.

NOTE 2: There seems to be a small delay before this changed is picked up by the front-end, so please retry after a couple of minutes if it does not work immediately.

In our project, we use the SharePoint 2010 wiki to document our solutions. We needed to export this in PDF format when releasing the solution to production, so that we could pass over a formal documentation set to the operations team, and not just refer to a constantly changing wiki, which also had a number of pages irrelevant to operations. We looked at some 3rd party solutions, such as the Wiki Publisher from Bamboo solutions, but it did not do what we wanted. Fortunately, our project manager, found a solution. We extended this a bit, to better fit our need and to create a professionally looking Word document, which we could also export to PDF, of course.

How to export Wiki to Word

First, we added three additional fields to our Wiki library:

All these fields were added to the standard view, and we created a datasheet view for more efficient setting of these values.

Then we created a new view, called "PrinterFriendlyView" as follows:

1. Remove all columns except "Wiki Content"
2. Set sorting to use the "Printout..." fields described above
3. Set filter to be "Printout in System Documentation" = "yes"
4. Set "Style" to "Newsletter"
5. Set "Item Limit" to 999

Open this new view, and follow these steps:

1. Mark all the text (this is a little awkward, I found it easiest to start from the bottom and go upwards)
2. Open a blank document in Word and paste in all the content
3. You will probably get some warnings, where clicking "No" is probably the best (I didn't test clicking "Yes")
   Warning text: "This operation will require copying many styles. Do you want to use Normal style instead?"
   Warning text:
4. Change "Verdana, font 16" to "Heading 1" style (this may differ for your wiki - hopefully you have some consistent formatting for page headings that you can use)
   a. Open the "Search/replace" dialog in Word
   b. Click on "Replace"
   c. Click on "More"
   d. Click in the "Replace with" box and then click the "Format" button at the very bottom. Choose "Style -> Heading 1"
   e. Click in the "Find what" box and then click the "Format" button at the very bottom. Choose "Font -> Verdana, 16pt"
   f. Click "Replace All"
   
5. Modify the style for "Heading 1", paragraph setting, to include a page break before each occurrence.
6. Go to the start of the document
7. Add a heading:"System Documentation", using "Title" style
8. Add a text underneath: "Table of Contents, using "SubTitle" style
9. Underneath, include a Table of Contents, using these steps (assuming Word 2010):
   a. In ribbon, click "References"
   b. Click "Table of Contents"
   c. At the bottom of the list, click "Insert Table of Contents..."
   d. In the creation dialog, set "Show levels" to 1, and click OK.
   e. Verify that the table of contents makes sense, you may, e.g., have some extra paragraphs marked as "Heading 1" where they should not have been.
10. Choose "Save As" and set format to PDF.

Voila, your nice, professional looking PDF is ready for sending to your operations team!

The PDF will even have clickable links in the table of contents, thanks to the nice Word-to-PDF export functionaly.

As you can see, the site names get trimmed, which doesn't look too good. If you edit the web part, you can increase the number of characters to display:

Now, the strings are no longer trimmed. But if they are too long, a line break is inserted, which can look awkward, too. (And it was much worse in our case, since the strings were so long that they extended outside the web part!)

There is no setting for increasing the width of the web part itself, and we did not have access to edit the page layout or CSS files. So, we turned to JavaScript for a quick and easy solution! We added a "Content Editor" web part to the top of the page, and removed it's styling, so that it was invisible to the user. Then we added the following lines of code:

<script language="javascript">
 var existing = window.onload;
 

 /* Needed to make sure existing SharePoint onload JavaScript still gets executed! */
 window.onload = function() {
 
 if(typeof(existing) == "function"){ existing(); }
 /* resize refiner width */
 document.getElementById('SRCHREF').style.width="250px"; 
 } 

</script>
 

There were three challenges to getting the above code to run properly:

1. Finding out that 'SRCHREF' was the element id of the refinement panel
2. Making sure the code ran after the page was fully loaded, since the refinements are displayed asyncronously
3. Making sure existing "onload" code continued to run, otherwise the ribbon functionality was broken

And now, everything looks just the way we want:

Open a PowerShell on your SharePoint server and run the following commands:

Add-PSSnappin *sharepoint*
New-SPEnterpriseSearchQueryScope -SearchApplication "FAST Query SSA" -Name "Boost aspx" -Description "A scope to boost aspx pages" -DisplayInAdminUI 1 -ExtendedSearchFilter "xrank(size:range(0,max), fileextension:aspx, boost=5000, boostall=yes)"

The key part here is towards the end:

fileextension:aspx, boost=5000

This will add 5000 extra "rank points", which is typically enough to always bring these documents to the top of the result list. TIP: In order to get an idea of what number of points make sense, and also to debug this process, it can be very useful to see the current rank values. This is easily done by applying the first part of my previous posting, to add the "Show all" toggle.

NOTE: This example uses a simple property and a simple value. You are free to boost based on multiple values at the same time, with different boost leves. This is done by putting multiple xrank's inside one another, like this: xrank( xrank(..., author:thomas, boost=500, ...), fileextension:docx, boost=2000, ...). You can also boost based on ranges, e.g. xrank(..., write>2011-01-01, boost=1000), to boost documents written this year. Unfortunately, there is no easy way to say "written last 3 months", but a workaround would be a scheduled script which updated the value, e.g., every night.

OK, let's try it out. Bring up the Search center, run a query, and click "Edit Page":

 Next, click "Edit webpart" on the "Search core results" web part:

Under "Location Properties", fill in the "Scope" field with the name of your scope ("Boost aspx" in this case).

Click "OK" and then click "Save" in the upper left part of the ribbon. Now, you can try running your query, and probably get this:

Hey, that failed! Why? There is a job which runs every 15 minutes, publishing these scopes. Depending on timing, it may take up to 15 minutes before your scope is active. But after that, you should be getting results ranked according to your xrank-FQL:

There are some peculiarities in the FQL above, which may be worth explaining:

size:range(0,max)

is used as a "match all" dummy query. This is due to the way xrank was implemented. Ideally, your end user query should have been there instead, but there is no easy way to achieve that. This works just as well since the end user query is "and'ed" to this xrank by the webpart.

 boostall=yes

is even harder to explain. The default is "no", for performance reasons, since a value of "yes" will give somewhat lower query throughput. But in enterprise search scenarios, the difference will be negligible in the large major of cases.

 Thanks to Vance Miller who first documented the approach!

• lemmatised forms of nouns and adjectives
• refiner values
• tokenization of terms
• etc.

 Fixml is stored in gzip'ed files under %FASTSEARCH/data/data_fixml. Un-gzip'ing them is fairly straightforward using command line tools that come with FAST, but knowing which file to extract is far less trivial. I will show you how you can do this easily. The prerequisites for this approach is:

1. SharePoint access rights to do "Edit page" on the FAST Search Center's result page.
2. File access to the data_fixml folder (on the FAST server(s) doing indexing) 

OK, so let's get started. First, run a query in a FAST Search Center, so you get a results page, and click "Edit Page":

Then, go down to the "Core Search Results" web part and choose "Edit web part":

Scrolling up to the top right-hand corner, you open the "Display Properties" of the web part like this:

Uncheck the "Use localization visualization" to enable editing of the "Fetched Properties" setting. Copy out the text and edit it in an editor to add the last two items ( <Column Name="internalid"/> <Column Name="contentid"/> ), shown in bold here:

<Columns>

<Column Name="WorkId"/>
<Column Name="Rank"/>
<Column Name="Title"/>
<Column Name="Author"/>
<Column Name="Size"/>
<Column Name="Path"/>
<Column Name="Description"/>
<Column Name="Write"/>
<Column Name="SiteName"/>
<Column Name="CollapsingStatus"/>
<Column Name="HitHighlightedSummary"/>
<Column Name="HitHighlightedProperties"/>
<Column Name="ContentClass"/>
<Column Name="IsDocument"/>
<Column Name="PictureThumbnailURL"/>
<Column Name="Url"/>
<Column Name="ServerRedirectedUrl"/>
<Column Name="FileExtension"/>
<Column Name="SpSiteUrl"/>
<Column Name="docvector"/>
<Column Name="fcocount"/>
<Column Name="fcoid"/>
<Column Name="PictureHeight"/>
<Column Name="PictureWidth"/>
<Column Name="internalid"/>
<Column Name="contentid"/>

</Columns>

Note: when pasting the above XML snippet, it will have to all be on one line!

 Next, click on the "XSL Editor" and replace the default XSLT with the contents of the attached zip file.

You are all set, and can click "OK" at the bottom of the web part editor, and then "Save & Close" in the ribbon on the top of the page. Your screen should now look like this:

If you click on the "Show all" link, it will toggle showing all columns returned from FAST. As we just added "internalid" and "contentid", this will appear at the end of the list

:

Next, we need a PowerShell script which is available in the TechNet Script Repository, Get-FastFixml. Download this code and install it somewhere on your FAST indexing server. Then, run it with internalid and contentid for the relevant document as input

:

As you can see, it has written the full Fixml to C:\FASTSEARCH\Var\Fixml.xml. We can open this file in an editor, and see for example how "deep dive" has been lemmatized to "deep deeper deepest" and "dive diva divas":

:

 So, we can try searching for "deeper diva", and see that it returns the expected results, and even highlights "deep dive"!

Reading the fixml contents can be challenging at first. Here is a quick overview of the most important tag name conventions:

I must admit that two of my colleagues did all the hard work here:

• Aaron Grant did the "Show All" toggle XSL
• Brent Groom did the Get-FASTFixml

Thank you very much, guys!

SearchCoreResults_ShowAll.zip

The files contains an exact copy of the input and output to the custom stage, and a useful tip is to always include the URL as input, even when it is not required for the business logic. This makes it much easier to see what sort of documents came with what properties

:

NOTE: I recommend that you familiarize yourself with the code sample in the MSDN documentation (see above link) before continuing. Otherwise, the code may be a little hard to follow.

So, extending on the sample code from the documentation, I wrapped the main logic in inside a parent method.

        // Handle the basic logging and exception handling
        static int Main(string[] args)
        {
            try
            {
                String inputXmlFile = args[0];
                String outputXmlFile = args[1];
 
                WriteLogFile(inputXmlFile, "-input");
                DoProcessing(inputXmlFile, outputXmlFile);
                WriteLogFile(outputXmlFile, "-output");
            }
            catch (Exception e)
            {
                // This will end up in the crawl log, since exit code != 0
                Console.WriteLine("Failed: " + e + "/" + e.StackTrace);
                return 1;
            }
            return 0;
        }

What does this give us:

1. We can log what came in to our process
2. We log what our process produced based on that
3. If processing fails - the actual error will be visible in the UI crawl log

NOTE: An additional tip here is to always include the URL of the as input parameter, even when it is not really needed for processing. Then it will be logged as part of the "input"-logging, and will give you a reference to the document processed.

But, as you may know, the pipeline extensibility stage is not allowed to write to the local file system, except for in one single location:

 <user-home-directory>\appdata\LocalLow

Consequently, this is what our logging routine will look like:

        // Write the input file to a location the application has access to write in.
        static void WriteLogFile(string inputFile, string suffix)
        {
            String pipelineInputData = @"c:\users\" + Environment.UserName + 
                                       @"\appdata\LocalLow\PipelineLog";
 
            // Enable/disable debugging in real-time by creating/renaming the log directory
            if (Directory.Exists(pipelineInputData))
            {
                Directory.CreateDirectory(pipelineInputData);
                string outFile = Path.Combine(pipelineInputData, timestamp + "-" + 
                                       MethodBase.GetCurrentMethod().DeclaringType.Name + 
                                       suffix + ".xml");
                File.Copy(inputFile, outFile);
            }
        }




Some points worth noting:

1. We check for the existence of the logging directory ("PipelineLog"), giving us the ability to turn debugging output on/off in real-time
2. We use a timestamp constant, so that input and output can be easily matched

That's about it, now we just need to define the constants we have been referencing:

        public static readonly Guid CrawledCategorySharepoint = 
            new Guid("00130329-0000-0130-c000-000000131346");
        public static readonly Guid CrawledCategoryUserDefined = 
            new Guid("D5CDD505-2E9C-101B-9397-08002B2CF9AE");
        public static readonly String timestamp = 
            DateTime.Now.ToString("yyyyMMddHHmmss.ffff");




And the main processing will be handled very much like suggested in the MSDN documentation:

        // Actual processing
        static void DoProcessing(string inputFile, string outputFile)
        {
 
            XDocument inputDoc = XDocument.Load(inputFile);
 
            // Fetch the content type property from the input item
            var res = from cp in inputDoc.Descendants("CrawledProperty")
                      where new Guid(cp.Attribute("propertySet").Value).
                                    Equals(CrawledCategorySharepoint) &&
                          cp.Attribute("propertyName").Value == "ows_ContentType" &&
                          cp.Attribute("varType").Value == "31"
                      select cp.Value;
 
            // Create the output item
            XElement outputElement = new XElement("Document");
            string mappedTo = "nocontenttype";
 
            // Add a crawled property if a content type was present
            if (res.Count() > 0 && res.First().Length > 0)
            {
                mappedTo = "hascontenttype";
            }
 
            outputElement.Add(
                new XElement("CrawledProperty",
                    new XAttribute("propertySet", CrawledCategoryUserDefined),
                    new XAttribute("propertyName", "mycontentcheck"),
                    new XAttribute("varType", 31), mappedTo)
                    );
 
            outputElement.Save(outputFile);
        } 

Attached is the complete code.

The corresponding contents of pipelineextensibility.xml is as follows:

<PipelineExtensibility>
  <Run command="HasContentTypeCheck.exe %(input)s %(output)s">
    <Input>
      <CrawledProperty propertySet="00130329-0000-0130-c000-000000131346" varType="31" propertyName="ows_ContentType"/>

      <!-- Included for debugging/traceability purposes -->
      <CrawledProperty propertySet="11280615-f653-448f-8ed8-2915008789f2" varType="31" propertyName="url"/>

    </Input>
    <Output>
      <CrawledProperty propertySet="d5cdd505-2e9c-101b-9397-08002b2cf9ae" varType="31" propertyName="mycontentcheck"/>
    </Output>
  </Run>
</PipelineExtensibility>
  

Credits go to my colleague Barry Waldbaum for coming up with the inital solution!

Disclaimer:
This code is dedicated to the public domain as-is without warranty of any kind.

HasContentTypeCheck.cs