Search scopes in FAST Search for SharePoint (Part 2)

 

Introduction

In my previous blog post about search scopes I described how to manage scopes with FAST Search for SharePoint (FAST Search). In this blog post I’ll try to describe how to programmatically set a filter on a scope, how to fetch the scope rules for a scope and outline how you can convert it’s scope rules into FAST Query Language (FQL) filters.

Setup development environment

See my blog post about accessing FAST admin features for how to set up the development environment, for this exercise it’s similar, except you don’t need the Microsoft.SharePoint.Search.Extended.Administration.dll assembly

Implementation

Updating the scope filter value

First I’ll show how you can fetch a site scope and a shared scope and modify the filter parameter, it’s pretty straight forward. For both site scopes and shared scopes you need to fetch a ScopesManager, the only difference is how you fetch it. When you have the ScopesManager you can fetch a named scope from it. This code assumes you have a site called ‘fast’ with the scope “MySimpleSiteScope” and a shared scope “MySimpleSharedScope”

using System;

using Microsoft.Office.Server.Search.Administration;

using Microsoft.SharePoint;

namespace FASTScopeFilters

{

    class Program

    {

        static void Main(string[] args)

        {

            try

            {

                UpdateSiteScope();

           UpdateSharedScope();

            }

            catch (Exception e)

            {

                Console.Out.WriteLine("Exception caught: " + e);

                Environment.Exit(1);

            }

        }

        private static void UpdateSiteScope()

        {

            const string siteCollectionUrl = "https://localhost/fast/";

            const string scopeName = "MySimpleSiteScope";

            using (var site = new SPSite(siteCollectionUrl))

            {

                var scopesManager = new RemoteScopes(SPServiceContext.GetContext(site));

                var scope = scopesManager.GetScope(new Uri(site.Url), scopeName);

                Console.Out.WriteLine("Site scope filter value before update: {0}", scope.Filter);

                scope.Filter = "title:site";

                scope.Update();

            }

        }

        private static void UpdateSharedScope()

        {

            const string scopeName = "MySimpleSharedScope";

            foreach (SearchServiceApplication ssa in SearchService.Service.SearchApplications)

            {

                ScopesManager scopesManager = new Scopes(ssa);

                var scope = scopesManager.GetSharedScope(scopeName);

                Console.Out.WriteLine("Shared scope filter value before update: {0}", scope.Filter);

                scope.Filter = "title:shared";

                scope.Update();

            }

        }

    }

}

Converting a scope’s rules into a FQL filter string

Say you already have an existing Search Server scope defined, and you want the rules of this scope converted into an FQL filter string that matches this scope. If we were to do that in the UpdateSiteScope() method above, the code would have to be updated to look something like this:

private static void UpdateSiteScope()

{

    const string siteCollectionUrl = "https://localhost/fast/";

    const string scopeName = "MySimpleSiteScope";

    using (var site = new SPSite(siteCollectionUrl))

    {

        var scopesManager = new RemoteScopes(SPServiceContext.GetContext(site));

        var scope = scopesManager.GetScope(new Uri(site.Url), scopeName);

        Console.Out.WriteLine("Site scope filter value before update: {0}", scope.Filter);

        var searchApp = (SearchServiceApplicationProxy)

            SearchServiceApplicationProxy.GetProxy(SPServiceContext.GetContext(site));

        // status code != 0 means the scope doesn't exist

        int statusCode;

        var rulesForScope = searchApp.GetRulesInfo(scope.ID, out statusCode);

        var scopeHelper = new ScopeHelper(rulesForScope);

        scope.Filter = scopeHelper.GenerateFastScopeFilter();

        scope.Update();

    }

}

I’ll leave it as an exercise for the reader to figure out the code in ScopeHelper to generate the FQL from scope rules, but it’s not very difficult given you know how the different scope rules map to FQL terms, which I’ll describe how to do next.

Converting a scope’s rules into an FQL filter

A search scope rule can have three behaviors; Include, Exclude or Require. Include represents logical "OR", Require represents logical "AND" while Exclude represents logical "NOT". When converting scope rules into FQL statements all rules are converted to property restriction queries based on rule type.

Property Query rules

Property query scope rules are created the same way you would write a property restriction in your query.

The Include property/value pair includeprop/includeval is translated into the FQL statement:

includeprop:”includeval”

Whereas the Exclude property/value pair excludeprop/excludeval is translated into the FQL not statement:

not(excludeprop:”excludeval”)

A combination of an Include/Require and an Exclude rule is translated into the following FQL:

andnot(includeprop:”includeval”, excludeprop:”excludeval” [,excludeprop2:”excludeval2”, …])

Web Address rules

Web address scope rules are converted into property restrictions using the managed properties "path" and "domain", depending on the type of the rule.

For Folder rules the property used in the property restriction is the managed property “path”.

An UrlScopeRule that has a folder rule type will look like this:

path:starts-with(“ https://mysharepoint/sites/mysite/myfolder/ ” )

For Domain and Hostname rules the property used in the property restriction is the managed property “domain”, but with a little difference in usage depending on rule type.

An UrlScopeRule that has a domain rule type will look like this:

domain:”mydomain”

An UrlScopeRule that has a hostname rule type will look like this:

domain:starts-with(“myhost”)

The Include/Exclude/Require statements are similar as for Property Query rules.

Content Source rules

In a FAST Search for SharePoint setup you will have two search service applications (SSA): one for content and one for query. Since you set up content sources on the content SSA and scope rules on the query SSA, there is unfortunately no information in the query SSA about the content sources in the content SSA. To achieve content source rule filters the approach is to use property query scope rules instead, where the managed property is “ContentSource” and the value is the name of the content source:

contentsource:"local sharepoint sites"

All Content rules

“All Content” rules are not needed in FAST Search for SharePoint and can be ignored. But do note that you can attach a filter also to an “All Sites” type of scope should you want to.

Putting it all together

A scope sometimes consist of more than one rule, which means your FQL filter will grow more complex the more rules you have. In the following section I’ll try to outline how rules are combined into FQL filters below, reading the FQL chapter here is recommended although the examples below should be understandable without it.

Single rule examples

Property Query Rule

Rule Name	Rule1
Filter Behavior	Include
Property	contentclass
Value	I1
FQL Filter	contentclass:”I1”

Web Address Rule

Rule Name	Rule1
Filter Behavior	Include
Type	Folder
Value	https://sitecollection/
FQL Filter	path:starts-with("https://sitecollection/")

Domain rule

Rule Name	Rule1
Filter Behavior	Include
Type	Domain
Value	domain.com
FQL Filter	domain:"domain.com"

Hostname rule

Rule Name	Rule1
Filter Behavior	Include
Type	Hostname
Value	hostname
FQL Filter	domain:starts-with("hostname")

Only Include Rules

Rule Name	Rule1	Rule2	Combined
Filter Behavior	Include	Include
Property	contentclass	contentclass
Value	I1	I2
FQL Filter	contentclass:”I1”	contentclass:”I2”	or(contentclass:"I1", contentclass:"I2")

Rule Name	Rule1	Rule2	Combined
Filter Behavior	Include	Include
Type	Folder	Folder
Value	https://sitecollection/	https://sitecollection2/
FQL Filter	path:starts-with("https://sitecollection/")	path:starts-with("https://sitecollection/")	or(path:starts-with("https://sitecollection/"), path:starts-with("https://sitecollection2/"))

Two Include rules are combined into an or() statement.

Only Require Rules

Rule Name	Rule1
Filter Behavior	Require
Property	contentclass
Value	R1
FQL Filter	contentclass:”R1”

Rule Name	Rule1	Rule2	Combined
Filter Behavior	Require	Require
Property	contentclass	contentclass
Value	R1	R2
FQL Filter	contentclass:”R1”	contentclass:”R2”	and(contentclass:"I1", contentclass:"I2")

Two require rules are combined into an and() statement.

Only Exclude Rules

Rule Name	Rule1
Filter Behavior	Exclude
Property	contentclass
Value	E1
FQL Filter	not(contentclass:”E1”)

Rule Name	Rule1	Rule2	Combined
Filter Behavior	Exclude	Exclude
Property	contentclass	contentclass
Value	E1	E2
FQL Filter	contentclass:”E1”	contentclass:”E2”	or(not(contentclass:"I1"), not(contentclass:"I2"))

Two or more exclude rules are OR’ed together.

Include and Require Rules

Rule Name	Rule1	Rule2	Combined
Filter Behavior	Include	Require
Property	contentclass	contentclass
Value	I1	R1
FQL Filter	contentclass:”I1”	contentclass:”R1”	and(contentclass:"I1", contentclass:"R1")

Combining an Include and a Require rule results in an and() if there is only one include rule.

Rule Name	Rule1	Rule2	Rule3	Combined
Filter Behavior	Include	Include	Require
Property	contentclass	contentclass	contentclass
Value	I1	I2	R1
FQL Filter	contentclass:”I1”	contentclass:”I2”	contentclass:”R1”	and(or(contentclass:"I1", contentclass:”I2”), contentclass:"R1")

Two include rules and one require rule results in an and() between the include rules or()’ed together and the require rule.

Include/Require and Exclude rules

Rule Name	Rule1	Rule2	Combined
Filter Behavior	Include	Exclude
Property	contentclass	contentclass
Value	I1	E1
FQL Filter	contentclass:”I1”	contentclass:”E1”	andnot(contentclass:"I1", contentclass:"E1")

Rule Name	Rule1	Rule2	Rule3	Combined
Filter Behavior	Include	Exclude	Exclude
Property	contentclass	contentclass	contentclass
Value	I1	E1	E2
FQL Filter	contentclass:”I1”	contentclass:”E1”	contentclass:”E2”	andnot(contentclass:"I1", contentclass:"E1", contentclass:”E2”)

Rule Name	Rule1	Rule2	Rule3	Rule4	Combined
Filter Behavior	Include	Include	Exclude	Exclude
Property	contentclass	contentclass	contentclass	contentclass
Value	I1	I2	E1	E2
FQL Filter	contentclass:”I1”	contentclass:”I1”	contentclass:”E1”	contentclass:”E2”	andnot(or(contentclass:"I1", contentclass:”I2”), contentclass:"E1", contentclass:”E2”)

Rule Name	Rule1	Rule2	Rule3	Rule4	Combined
Filter Behavior	Require	Require	Exclude	Exclude
Property	contentclass	contentclass	contentclass	contentclass
Value	R1	R2	E1	E2
FQL Filter	contentclass:”R1”	contentclass:”R2”	contentclass:”E1”	contentclass:”E2”	andnot(and(contentclass:"R1", contentclass:”R2”), contentclass:"E1", contentclass:”E2”)

With Include/Require and Exclude, the first statement of and andnot() are the Include/Requires while the rest are the Excludes.

Include, Require and Exclude rules

Rule Name	Rule1	Rule2	Rule3	Combined
Filter Behavior	Include	Require	Exclude
Property	contentclass	contentclass	contentclass
Value	I1	R1	E1
FQL Filter	contentclass:”I1”	contentclass:”R1”	contentclass:”E1”	andnot(and(contentclass:"I1", contentclass:"R1"), contentclass:"E1"))

Rule Name	Rule1	Rule2	Rule3	Rule4	Combined
Filter Behavior	Include	Include	Require	Exclude
Property	contentclass	Contentclass	contentclass	contentclass
Value	I1	I2	R1	E1
FQL Filter	contentclass:”I1”	Contentclass:”I2”	contentclass:”R1”	contentclass:”E1”	andnot(and(or(contentclass:"I1", contentclass:”I2”), contentclass:"R1"), contentclass:"E1"))

Summary

You can create FQL filters that match existing scope rules programmatically if you already have scopes defined and are migrating to using FAST Search for SharePoint. In case you’re using both SharePoint Search and FAST Search you can keep scope rules and filters in sync with each other. Do note that in case you’re only using FAST Search you don’t have to create any rules for your scopes, all you need to do is define the scope and then set a filter term, either using PowerShell or doing it programmatically.