Introduction

In my previous blog post about search scopes I described how to manage scopes with FAST Search for SharePoint (FAST Search). In this blog post I’ll try to describe how to programmatically set a filter on a scope, how to fetch the scope rules for a scope and outline how you can convert it’s scope rules into FAST Query Language (FQL) filters.

Setup development environment

See my blog post about accessing FAST admin features for how to set up the development environment, for this exercise it’s similar, except you don’t need the Microsoft.SharePoint.Search.Extended.Administration.dll assembly

Implementation

Updating the scope filter value

First I’ll show how you can fetch a site scope and a shared scope and modify the filter parameter, it’s pretty straight forward. For both site scopes and shared scopes you need to fetch a ScopesManager, the only difference is how you fetch it. When you have the ScopesManager you can fetch a named scope from it. This code assumes you have a site called ‘fast’ with the scope “MySimpleSiteScope” and a shared scope “MySimpleSharedScope”

using System;

using Microsoft.Office.Server.Search.Administration;

using Microsoft.SharePoint;

namespace FASTScopeFilters

{

    class Program

    {

        static void Main(string[] args)

        {

            try

            {

                UpdateSiteScope();

                UpdateSharedScope();

            }

            catch (Exception e)

            {

                Console.Out.WriteLine("Exception caught: " + e);

                Environment.Exit(1);

            }

        }

        private static void UpdateSiteScope()

        {

            const string siteCollectionUrl = "https://localhost/fast/";

            const string scopeName = "MySimpleSiteScope";

            using (var site = new SPSite(siteCollectionUrl))

            {

                var scopesManager = new RemoteScopes(SPServiceContext.GetContext(site));

                var scope = scopesManager.GetScope(new Uri(site.Url), scopeName);

                Console.Out.WriteLine("Site scope filter value before update: {0}", scope.Filter);

                scope.Filter = "title:site";

                scope.Update();

            }

        }

        private static void UpdateSharedScope()

        {

            const string scopeName = "MySimpleSharedScope";

            foreach (SearchServiceApplication ssa in SearchService.Service.SearchApplications)

            {

                ScopesManager scopesManager = new Scopes(ssa);

                var scope = scopesManager.GetSharedScope(scopeName);

                Console.Out.WriteLine("Shared scope filter value before update: {0}", scope.Filter);

                scope.Filter = "title:shared";

                scope.Update();

            }

        }

    }

}

Converting a scope’s rules into a FQL filter string

Say you already have an existing Search Server scope defined, and you want the rules of this scope converted into an FQL filter string that matches this scope. If we were to do that in the UpdateSiteScope() method above, the code would have to be updated to look something like this:

private static void UpdateSiteScope()

{

    const string siteCollectionUrl = "https://localhost/fast/";

    const string scopeName = "MySimpleSiteScope";

    using (var site = new SPSite(siteCollectionUrl))

    {

        var scopesManager = new RemoteScopes(SPServiceContext.GetContext(site));

        var scope = scopesManager.GetScope(new Uri(site.Url), scopeName);

        Console.Out.WriteLine("Site scope filter value before update: {0}", scope.Filter);

        var searchApp = (SearchServiceApplicationProxy)

            SearchServiceApplicationProxy.GetProxy(SPServiceContext.GetContext(site));

        // status code != 0 means the scope doesn't exist

        int statusCode;

        var rulesForScope = searchApp.GetRulesInfo(scope.ID, out statusCode);

        var scopeHelper = new ScopeHelper(rulesForScope);

        scope.Filter = scopeHelper.GenerateFastScopeFilter();

        scope.Update();

    }

}

I’ll leave it as an exercise for the reader to figure out the code in ScopeHelper to generate the FQL from scope rules, but it’s not very difficult given you know how the different scope rules map to FQL terms, which I’ll describe how to do next.

Converting a scope’s rules into an FQL filter

A search scope rule can have three behaviors; Include, Exclude or Require.  Include represents logical "OR", Require represents logical "AND" while Exclude represents logical "NOT". When converting scope rules into FQL statements all rules are converted to property restriction queries based on rule type.

Property Query rules

Property query scope rules are created the same way you would write a property restriction in your query.

The Include property/value pair includeprop/includeval is translated into the FQL statement:

includeprop:”includeval”

Whereas the Exclude property/value pair excludeprop/excludeval is translated into the FQL not statement:

not(excludeprop:”excludeval”)

A combination of an Include/Require and an Exclude rule is translated into the following FQL:

andnot(includeprop:”includeval”,  excludeprop:”excludeval” [,excludeprop2:”excludeval2”, …])

Web Address rules

Web address scope rules are converted into property restrictions using the managed properties "path" and "domain", depending on the type of the rule.

For Folder rules the property used in the property restriction is the managed property “path”.

An UrlScopeRule that has a folder rule type will look like this:

path:starts-with(“https://mysharepoint/sites/mysite/myfolder/”)

For Domain and Hostname rules the property used in the property restriction is the managed property “domain”, but with a little difference in usage depending on rule type.

An UrlScopeRule that has a domain rule type will look like this:

domain:”mydomain”

An UrlScopeRule that has a hostname rule type will look like this:

domain:starts-with(“myhost”)

The Include/Exclude/Require statements are similar as for Property Query rules.

Content Source rules

In a FAST Search for SharePoint setup you will have two search service applications (SSA): one for content and one for query. Since you set up content sources on the content SSA and scope rules on the query SSA, there is unfortunately no information in the query SSA about the content sources in the content SSA. To achieve content source rule filters the approach is to use property query scope rules instead, where the managed property is “ContentSource” and the value is the name of the content source:

contentsource:"local sharepoint sites"

All Content rules

“All Content” rules are not needed in FAST Search for SharePoint and can be ignored. But do note that you can attach a filter also to an “All Sites” type of scope should you want to.

Putting it all together

A scope sometimes consist of more than one rule, which means your FQL filter will grow more complex the more rules you have. In the following section I’ll try to outline how rules are combined into FQL filters below, reading the FQL chapter here is recommended although the examples below should be understandable without it.

Single rule examples

Property Query Rule

Web Address Rule

Domain rule

Hostname rule

Only Include Rules

Two Include rules are combined into an or() statement.

Only Require Rules

Two require rules are combined into an and() statement.

Only Exclude Rules

Two or more exclude rules are OR’ed together.

Include and Require Rules

Combining an Include and a Require rule results in an and() if there is only one include rule.

Two include rules and one require rule results in an and() between the include rules or()’ed together and the require rule.

Include/Require and Exclude rules

With Include/Require and Exclude, the first statement of and andnot() are the Include/Requires while the rest are the Excludes.

Include, Require and Exclude rules

Summary

You can create FQL filters that match existing scope rules programmatically if you already have scopes defined and are migrating to using FAST Search for SharePoint. In case you’re using both SharePoint Search and FAST Search you can keep scope rules and filters in sync with each other. Do note that in case you’re only using FAST Search you don’t have to create any rules for your scopes, all you need to do is define the scope and then set a filter term, either using PowerShell or doing it programmatically.

Introduction

The FAST Search administration features are public with SharePoint Server 2010. All FAST admin services are accessed through the FASTAdminProxy property on the SearchServiceApplication or the SearchServiceApplicationProxy. You can among other things manage keywords and the schema. The documentation will contain examples on how to use the different contexts to perform administration tasks, so I won’t focus much on that. The examples I’ve seen do however not show how to access these contexts in a SharePoint setting, so that’s the main purpose of this blog post.
In this blog post I’ll create a console application that creates a keyword with a best bet using the KeywordContext class.

Setup development environment

Setting up Visual Studio is pretty straight forward if you have Visual Studio 2010 RC running on the same box as SharePoint.

• Create a new console application that targets x64 platform
• Add a reference to the following binaries, located in "%CommonProgramFiles%\microsoft shared\web server extensions\14\isapi\"
• Microsoft.SharePoint.dll
• Microsoft.Office.Server.Search.dll
• Microsoft.SharePoint.Search.Extended.Administration.dll

In addition, the user you’re running as must be part of the FASTSearchAdministrators group on the FAST Search for SharePoint installation you have configured.

You should now be ready to start coding.

Implementation

Below is code that gets the FASTAdminProxy’s KeywordContext and from this retrieves the search setting group for the given site collection. With a reference to the search setting group keywords can be added. What the code examples I’ve seen so far have lacked is how you get hold of the FASTAdminProxy. This is shown in the first few lines of the code: You get hold of the SearchServiceApplicationProxy associated with the site which you want to manage keywords for. When you have the SSAProxy you can get a reference to the FASTAdminProxy on this, if it’s configured with FAST Search. If the SSA is not configured with FAST Search, the FASTAdminProxy reference will be null.

Here’s the code, note that most error checking has been omitted for readability.

// Copyright © Microsoft Corporation.  All Rights Reserved.

// This code released under the terms of the

// Microsoft Public License (MS-PL, https://opensource.org/licenses/ms-pl.html.)

using System;

using Microsoft.Office.Server.Search.Administration;

using Microsoft.SharePoint;

namespace FASTAdminOperations

{

    class Program

    {

        static void Main(string[] args)

        {

            try

            {

                const string siteCollectionUrl = "https://localhost/fast/";

                using (var site = new SPSite(siteCollectionUrl))

                {

                    var guid = site.ID.ToString();

                    var ssaProxy =

                        (SearchServiceApplicationProxy)

                        SearchServiceApplicationProxy.GetProxy(SPServiceContext.GetContext(site));

                    var adminProxy = ssaProxy.FASTAdminProxy;

// If the SSA is not configured with FAST Search it will be null 

                    if (adminProxy == null)

                    {

                        Console.Out.WriteLine("No FASTAdminProxy found for site " + siteCollectionUrl);

                        Environment.Exit(1);

                    }

                    // KeywordContext gives you access to the keyword functionality of FAST Search

                    var keywordContext = adminProxy.KeywordContext;

                    // get or create a search setting group

                    var ssgs = keywordContext.SearchSettingGroups;

                    var searchSettingGroup = ssgs.ContainsSearchSettingGroup(guid)

                                                        ? ssgs.GetSearchSettingGroup(guid)

                                                        : ssgs.AddSearchSettingGroup(guid);

                    // setup keyword and best bet data

                    const string keywordTerm = "britney";

                    const string keywordDefinition = "Britney Jean Spears (born December 2, 1981) is an American singer, " +

                        "songwriter, dancer, actress, author and entertainer.";

                    const string bestBetTitle = "Britney Spears - official web site and blog";

                    var bestBetUri = new Uri("https://www.britneyspears.com");

                    const string bestBetDescription = "Welcome to the official Britney Spears website featuring Britney " +

                        "spears music, Britney Spears lyrics, Britney Spears photos, Britney Spears news and more.";

                    // fetch all keywords for this search setting group

                    var keywords = searchSettingGroup.Keywords;

                    // Let's just remove the keyword we want to add in case it exists

                    if (keywords.ContainsKeyword(keywordTerm))

                        keywords.RemoveKeyword(keywordTerm);

                    // Add a keyword and assign a definition

                    var keyword = keywords.AddKeyword(keywordTerm);

                    keyword.Definition = keywordDefinition;

                    // Associate a time limited best bet with the keyword

                    var bestbet = keyword.BestBets.AddBestBet(bestBetTitle, bestBetUri);

                    bestbet.Description = bestBetDescription;

                    bestbet.StartDate = DateTime.Now;

                    bestbet.EndDate = DateTime.Now.AddDays(7.0);

                }

            }

            catch (Exception e)

            {

                Console.Out.WriteLine("Exception caught: " + e);

                Environment.Exit(1);

            }

        }

    }

}

Summary

Given you’re part of the FASTSearchAdministrators group you can programmatically access the FAST Search administration features through the FASTAdminProxy.

When using search scopes in a FAST Search Server for SharePoint 2010 (FAST Search) installation there are some things that are nice to know to leverage the full potential of scopes. In this post I’ll try to shed some light on how scopes are used with FAST Search as the back-end.

FAST Search supports SharePoint Search Scopes

FAST Search supports all scopes that SharePoint Search exposes. These are shared scopes, site scopes and contextual scopes (e.g. scope:”this site”).

Search scopes with FAST Search for SharePoint are FQL filters

In FAST Search a search scope is just a pre-defined property filter specified in FAST Query Language (FQL) syntax that can be configured via PowerShell only OOB – the scope object itself is public though, so there’s nothing in the way of creating custom scope filter tools should one want to do that. The filter property is attached to the same search scope entity used by SharePoint Search. This means that the scopes you define to be used with FAST Search will show up in the UI in either central admin or site collection admin, but all you can do in terms of scopes administration is either to create or remove the scope itself, more about this a little later. To add, change or remove filters you need to use PowerShell.

Creating a search scope for use with FAST Search

As mentioned, there is no UI to administer scopes in an FAST Search setting, you need to use PowerShell. Let’s start with ways of creating a scope filter to be used with FAST Search. All examples use an SSA named “Search Service Application”, make sure you’re using the correct name for your SSA when creating scopes.

Creating a scope setting the filter on creation

In the SharePoint 2010 Management Shell type the following:

New-SPEnterpriseSearchQueryScope -SearchApplication "Search Service Application" -Name MySimpleScope -Description "My Simple Scope" -DisplayInAdminUI 1 -ExtendedSearchFilter ‘title:vista’

This will create a scope named “MySimpleScope” with a FQL filter on title containing the word ‘vista’. ExtendedSearchFilter is the parameter to make a note of here.

Please note that the scope must be compiled before it can be used in your queries.

Creating a scope and setting the filter after creation

In the SharePoint 2010 Management Shell type the following:

New-SPEnterpriseSearchQueryScope -SearchApplication "Search Service Application" -Name MySimpleScope –Description "My Simple Scope" -DisplayInAdminUI 1

$scope = Get-SPEnterpriseSearchQueryScope -SearchApplication "Search Service Application" -Identity "MySimpleScope”

$scope.Filter = "title:vista"

$scope.Update()

Creating a site scope

To create a site scope, add the –OwningSiteUrl switch to your command line:

New-SPEnterpriseSearchQueryScope -SearchApplication "Search Service Application" -Name MySimpleScope -Description "My Simple Scope" -DisplayInAdminUI 1 -ExtendedSearchFilter ‘title:vista’ –OwningSiteUrl “https://myhost/sites/mysite/”

You can also create a scope through the UI, but you must use PowerShell to set the filter after creating the scope.

Reviewing your scope filters after creation

To print the filter of a shared scope, do the following:

$scope = Get-SPEnterpriseSearchQueryScope -SearchApplication "Search Service Application" -Identity "MySimpleScope"

$scope.Filter

The same for a site scope, just add a –Url switch:

$scope = Get-SPEnterpriseSearchQueryScope -SearchApplication "Search Service Application" -Identity "MySimpleScope" –Url “https://myhost/sites/mysite/”

$scope.Filter

Update status=empty/Items=empty?

If you look in the UI after creating the FAST Search scope filters, you’ll notice that the scopes you have created are reported as empty with the number of items reported as empty as well.

The reason these scopes are reported as empty is because of the nature of how scopes are implemented. Remember that a scope filter is attached to the same scope entity as used by SharePoint Search, so what you’re seeing in the UI are the metrics of the SharePoint Search scopes, not the FAST Search filter. I know this is a little confusing, but it’s important to understand that there is (probably) nothing wrong with your scope filter even if the scope is reported as empty.

Scope rules have no influence on FAST Search queries

So what happens if you add scope rules to the empty scopes in the UI?

Scope rules will be used, but only for SharePoint Search queries. If you have setup FAST Search you will likely have two configured Search Service Applications (SSA): One query SSA with a connection to the FAST Search back-end as well as SharePoint (people) search and one ExtendedConnector SSA for feeding to FAST Search. For more information on Search Service Applications see Arnt’s blog post The two types of Search Service Applications (in a SharePoint 2010 deployment with FAST Search Server).

You can of course add content sources to your query SSA, in which the crawled content from these content sources will go into a SharePoint Search index. That’s when your scope rules come into play.

If you choose to have a SharePoint index on the SSA that is configured with FAST Search please note the following:

1.       Numbers reported in the UI reflect the items in the SharePoint Search index

2.       If you aim for using the same scopes with both SharePoint Search and FAST Search remember to update the scopes you crate with filters using PowerShell. My next blog post will go more in depth on how you can convert scope rules into FQL filters both manually and programmatically.

Specifying which fulltext index to search

FAST Search scope filters have one additional capability; you can in the filter specify which non-default fulltext index (=composite field if you’re familiar with FAST ESP terminology) you want to query. To specify the fulltext index as part of, or the whole scope filter use “FullTextIndex=” as part of the filter. Note that the you separate the FullTextIndex parameter and the value with equals (=).

New-SPEnterpriseSearchQueryScope -SearchApplication "Search Service Application" -Name MySimpleScope -Description "My Simple Scope" -DisplayInAdminUI 1 -ExtendedSearchFilter ‘title:vista,FullTextIndex= mycustomfulltextindex’

Create a scope named “MySimpleScope” with a FQL filter on titles containing the word ‘vista’ searching the fulltext index named ‘mycustomfulltextindex’.

Using the scope in a query:

microsoft scope:mysimplescope

Fulltext index can also be a managed property

The value behind the FullTextIndex= can also be a managed property, i.e. if you specify say ‘title’ as the fulltext index your queries will be limited to this managed property.

This is best demonstrated with these two scopes:

Scope1:

New-SPEnterpriseSearchQueryScope -SearchApplication "Search Service Application" -Name TitleVistaScope -Description "My Simple Scope" -DisplayInAdminUI 1 -ExtendedSearchFilter ‘title:vista’

Scope2:

New-SPEnterpriseSearchQueryScope -SearchApplication "Search Service Application" -Name FullTextIndexTitleScope -Description "My Simple Scope" -DisplayInAdminUI 1 -ExtendedSearchFilter ‘fulltextindex=title’

Now with these two scopes all these queries will return the same results:

Query1: title:vista

Query2: scope:titlevistascope

Query3: vista scope:fulltextindextitlescope

Advantages using scope filters

·         A scope filter can be any FQL statement, which opens up for very powerful query expressions like xrank, sorting etc.

·         After the initial compilation of the scope no additional compilation is necessary if changing the filter syntax. However there is an internal cache that needs to update with the new filter string – the update interval of the cache is 5 minutes (not ready immediately as I originally wrote).

·         When using different fulltext indexes you can get the same recall for a query, but different rankings depending on which fulltext index you use.

·         Use of scopes in search center is the same for FAST Search and SharePoint Search; both products support global scopes, site scopes and contextual scopes. Adding a dropdown and search tab backed by scopes is same for both products.

Limitations with scope filters

There are some limitations on how you can use search scopes with a FAST Search for SharePoint back-end.

You do not get stemming on filters

Unfortunately, stemming is not applied on terms used in a scope filter. This means that if you issue a query to test the effect of a scope filter, make sure stemming is turned off to see how the filter behaves.

You cannot OR scopes filters together

With FAST Search multiple scopes are AND’ed together regardless of operator used, this means that the query scope:myscope OR scope:myotherscope  is rewritten to the FQL filter and(scope:myscope, scope:myotherscope) Where the scopes are replaced with their respective FQL terms of course

No UI support

As mentioned earlier FAST Search scopes with no rules are reported as empty and scope rules used both for FAST Search and SharePoint Search will report item counts matching the SharePoint index, not the FAST Search index.