Knut Brandrud's dev blog https://docs.microsoft.com/archive/blogs/knutbran/feed.xml FAST Search Server for SharePoint Wed, 17 Apr 2019 03:23:06 GMT en-US hourly 1 How to view the status of the SharePoint 2013 search service https://docs.microsoft.com/archive/blogs/knutbran/how-to-view-the-status-of-the-sharepoint-2013-search-service Fri, 30 Nov 2012 06:41:00 GMT https://blogs.msdn.microsoft.com/knutbran/2012/11/30/how-to-view-the-status-of-the-sharepoint-2013-search-service/ As promised at the SharePoint Conference, here is a useful script that gives you the status of the search service in SharePoint Server 2013.

The script provides the following information:

  • Key topology status data from Get-SPEnterpriseSearchStatus
  • Information about primary role for index and admin components
  • Job status for jobs run by the Analytics Processing Component
  • Detailed indexer information
    • Re-partitioning status/progress
    • Master merge in progress
    • Index replica out of sync
  • Crawl status (crawling/idle/paused)
  • Number of indexed documents
  • HA status for topology, indicating which components/partitions that does not have active redundant components
  • Host controller repository status (for search dictionaries)
  • Overall indication of system health for the search topology

Get-SPSearchTopologyState.ps1

]]> FAST Search for SharePoint extensibility updates in SP1 https://docs.microsoft.com/archive/blogs/knutbran/fast-search-for-sharepoint-extensibility-updates-in-sp1 Tue, 12 Jul 2011 12:21:00 GMT https://blogs.msdn.microsoft.com/knutbran/2011/07/12/fast-search-for-sharepoint-extensibility-updates-in-sp1/ SP1 is out, and there are some improvements and fixes relevant for the developer. This is a brief summary, with links to additional information.

  1. More flexible property extraction. FAST Search for SharePoint enables you to create your own property extractors. In this way you can extract names and concepts from the crawled content and add the normalized extracted values in new or existing managed properties. This is useful if you enable query refinement on the managed properties.
    With SP1, there is no longer any restrictions on number of custom property extractors. Another improvement is that you can configure case insensitive property extractors. For most use cases this makes the configuration much simpler.
    Note that you now use a separate configuration file for property extractors. If you already have created custom property extractors, you can easily migrate to the new configuration format if desired (but existing configuration will still be supported for backwards compatibility).
    For more details, see updated documentation on MSDN: Creating a custom property extractor.
  2. Configurable title and date extraction for Word and PowerPoint documents. This is an alignment with the existing behavior in SharePoint Server search. After installing SP1, the title and date for these documents are retrieved from the content of the documents (if found), rather than the document metadata. Document metadata is often misleading, as users often writes new documents based on other documents or templates, and do not update the metadata. This is normally the desired behavior, unless your organization have policies and processes for keeping document metadata up to date. If you want the old behavior, you can configure this in optionalprocessing.xml. For more details, see updated documentation on MSDN: Configuring Optional Item Processing.
  3. Correct handling of refiner names with uppercase letters. In the RTM version you must specify refiner names in lowercase, even if the associated managed property contains uppercase letters (such as "Write"). This fix applies to the refinement panel web part configuration, query web service and query OM. There is also a similar fix for managed property names in scope filters.
  4. Using Keyword Syntax queries with non-default full-text index. This means you specify the full-text index as a property filter in the query. This will now return correct results.

 

See also the full list of improvements and fixes: Summary of FAST Search Server 2010 for SharePoint SP1 improvements and fixes.

]]> Date and time in FAST Search Server queries and results - part 2 https://docs.microsoft.com/archive/blogs/knutbran/date-and-time-in-fast-search-server-queries-and-results-part-2 Fri, 10 Jun 2011 07:45:00 GMT https://blogs.msdn.microsoft.com/knutbran/2011/06/10/date-and-time-in-fast-search-server-queries-and-results-part-2/ I discussed some locale aspects of date/time queries in a previous post, and was asked about how to specify date/time property filters in queries.

FAST Query Syntax (FQL)

You must specify datetime property restrictions using an explicit range in UTC timezone and syntax.

and(string("test", mode="or"),
       write:range(min, 2011-06-09))

and(string("test", mode="or"),
       write:range(min, 2011-06-09, to="lt"))

 and(string("test", mode="or"),
       write:range(min, 2011-06-09T00:00:00Z))

 Items updated before, but not including, June 09, 2011, 00:00:00 UTC. If you leave out the time component, it defaults to 00:00:00.
and(string("test", mode="or"),
       write:range(min, 2011-06-09, to="le"))		 Items updated before, and including, June 09, 2011, 00:00:00 UTC.

and(string("test", mode="or"), write:2011-06-09))

and(string("test", mode="or"), 
       write:range(2011-06-09T00:00:00Z,
                         2011-06-10T00:00:00Z))

Items updated between (including) June 09, 2011, 00:00:00 UTC and (not including) June 10, 2011, 00:00:00 UTC.

Note that the first syntax expression is a special case, where the expression means the full day of 2011-06-09. This is a shortcut for the full range. For all explicit range expressions the datetime is interpreted as an exact date/time (down to the second), even if omitting the time component.
and(string("test", mode="or"),
       write:2011-06-09T13:45:00Z))		 Items updated June 09, 2011, 13:45:00 UTC

 

Keyword Query Syntax

For Keyword Syntax, the date expression must be specified according to your locale and timezone. This is an important difference from FQL!

You can specify explicit dates, and use property operators (:, = <, <=, etc.). The query examples below search for items containing the word 'test' (using a Norwegian locale):

test write:"2011-06-09"
test write="2011-06-09"
test write='2011-06-09'		 Items updated June 09, 2011
test write="2011-06-09 11:45:00"

Items updated June 09, 2011.

Specifying a time component is allowed, but the time component is discarded (you can only restrict the query by full days).
test write>="2011-06-09" Items updated June 09, 201,1 or later
test write<"2011-06-09" Items updated before June 09, 2011

 

You can also use these handy shortcuts:

test write:"yesterday"
test write="yesterday"

The time from the beginning of the day until the end of the day that precedes the current day.

test write:"this week"

The time from the beginning of the current week until the end of the current week. The week begins on monday.

test write:"this month"

The entire month that includes the current date.

test write:"last month"

The latest entire month that precedes the current month.

test write:"this year"

The entire year that includes the current date.

test write:"last year"

The latest entire year that precedes the current year.

 

Additional information:

 

 

]]> Some hints on testing custom managed properties and queries https://docs.microsoft.com/archive/blogs/knutbran/some-hints-on-testing-custom-managed-properties-and-queries Fri, 01 Apr 2011 01:52:00 GMT https://blogs.msdn.microsoft.com/knutbran/2011/04/01/some-hints-on-testing-custom-managed-properties-and-queries/ When you customize your FAST Search Server deployment, there are certainly things that may go wrong. Sometimes it is convenient to reduce the number of potential error sources and test that the custom managed property mapping works properly.

It is possible to submit test documents directly to the FAST Search Server deployment (in case the issue is in the crawling). You can submit test items using 'docpush', and you can also set up the item processing pipeline so that you can submit test documents as XML. By using XML test documents, you can make your own crawled properties and map to managed properties the way you want. This is explained in the following TechNet Wiki article: How to create a FAST Search for SharePoint test document using XMLMapper. Note that 'docpush' is only supported for testing, you should not use it in a production environment.

There are some useful query test tools provided by the community that you can use to test your customizations. One good example is https://blogs.technet.com/b/speschka/archive/2010/08/15/free-developer-search-tool-for-sharepoint-2010-search-and-fast-search-for-sharepoint.aspx

Another simple way to test queries, is to use a PowerShell script that access the Query OM. In this way, you can verify what's returned from the OM, and hence figure out if the problem is in the back-end or in the Web Part (or your custom query app). The attached PowerShell script runs a query via the KeywordQuery interface, and returns the selected custom managed property.

usage: propertytest.ps1 <URL> <Hits> <Property> <Query>

  • <URL> - Specifies the URL for the query front-end web server of the SharePoint farm.
  • <Hits>  - Number of hits to return.
  • <Property>  - Specifies the name of a managed property. Prints the content of the property and associated query refiner data. Use lowercase only, even if the property name contains uppercase letters.
  • <Query> - The query string in FQL syntax. Use single quotes if it contains spaces or double quotes.

If there are no refiners defined for the property, all OOB refiners are printed.

The script is also useful for other test purposes:

  • If you have specified a query refiner for the property, it will also provide the refinement data. This is useful for testing custom property extractors and refiners.
  • The script will also return the rank value, in case you are testing relevancy features.
  • The script accepts queries according to the FAST Query Language (FQL) syntax. This enables you to easily test advanced FQL query features. If you are not familiar to FQL: For basic property filters and single term queries, the syntax issimilar to the standard SharePoint Keyword Syntax you type in any SharePoint query box.
  • It is easy to test other OM features via PowerShell.

There is also another way to test queries. FAST Search Server provides an internal test query interface on the query processing server. This is explained in Test feeding and searching on FAST Search Server 2010 for SharePoint. This works fine if you index items with no access right restrictions, which is the case when you use 'docpush'. But for SharePoint data it will not work without some security tweaks, as this interface does not enable any authentication. Hence, it may be more convenient to test using the PowerShell script described in this blog.

propertytest.zip

]]> Date and time in FAST Search Server queries and results https://docs.microsoft.com/archive/blogs/knutbran/date-and-time-in-fast-search-server-queries-and-results Wed, 30 Mar 2011 04:41:00 GMT https://blogs.msdn.microsoft.com/knutbran/2011/03/30/date-and-time-in-fast-search-server-queries-and-results/ Handling date and time can be tricky on a query interface. If you create an end-user search interface, you may want to adapt to the user's timezone. You may decide to determine the user's timezone based on the locale, which makes sense in certain cases, but not always. You could use IP address, etc., but this may not work well in many corporate environments. My office is in Oslo, but the gateway out of the corporate network happens to be in Ireland. Hence, many applications tend to believe that I'm in Ireland.

If you look at how the SharePoint search web parts works, you will see that they are timezone aware. In the following search result, there are two test documents with different date/time in the write managed property. The first date is just showing the content of the write managed property. In the index, all datetime properties are represented as UTC. The second date is added by the web part, and is adjusted to my locale. I'm located in Oslo, which is one hour ahead of Greenwich.

In FAST Search for SharePoint, you can choose between two different query languages. The Keyword Syntax is the default query language in SharePoint. When using this language, you should be aware that all dates are adjusted according to the locale of the user. When using the Query OM, this is determined by the culture property. If you use the FAST Query Language (FQL), all dates should be specified in UTC syntax, and the search client should perform the necessary mapping. This is in most cases the preferred approach for search driven applications.

If you use query refinement, there are some other concerns. The default statistical binning in FAST Search Server will not give you datetime bins with reasonable date boundaries. What you typically want instead, is buckets such as 'today', 'yesterday', 'last week', 'last month', etc. You can do that as well, but then you need to create the (non-overlapping) bins per query in the client, based on the time when the user performs the query. Then you can combine these to provide possibly overlapping bins such as  'yesterday' and 'last week'. This is described in more detail in Query Refinement. If you want to adapt the bins to the user's time zone, you must specify the bins on date and hour boundaries to reflect change of day in the user's timezone, but specified in UTC.

See also part 2, which discusses datetime query syntax.

]]> Content API for FAST Search Server 2010 for SharePoint https://docs.microsoft.com/archive/blogs/knutbran/content-api-for-fast-search-server-2010-for-sharepoint Fri, 18 Mar 2011 09:07:00 GMT https://blogs.msdn.microsoft.com/knutbran/2011/03/18/content-api-for-fast-search-server-2010-for-sharepoint/ FAST Search for SharePoint provides multiple options for crawling and retrieving content for indexing:

  • When you use the FAST Content SSA, the feeding/crawling will work in the same way as for SharePoint Server search. This is the recommended approach in most cases.
  • FAST Search for SharePoint also supports additional indexing connectors, to solve more advanced needs. For more information, see  Plan for crawling and federation (TechNet). There is also a table that provides a comparison of the connectors.
  • If you have a custom content source, you can in most cases retrieve data for indexing using the BCS framework.

If you have more specialized needs, there is one more option. The FAST Content API enables you to push content from a client .NET application. This API is mainly provided to enable customers of previous FAST products to migrate their custom content connectors and use them with FAST Search for SharePoint. You may also use this API for other custom applications, but note that there are certain limitations on the support for the API - see the download page for details.

The API provides an asynchronous interface for pushing items to the FAST Search farm from a remote client application. The API communicates with the FAST Search Content Distributor, using WCF. The API is based on .Net Framework 3.5.

If you are familiar with the FAST ESP .NET Content API, you will find it easy to use, but there are some differences:

  • The routing attributes are not supported
  • You can add multiple attachment documents for each item
  • You can add a security descriptor for each item, using a  Security Descriptor Definition Language string that describes the security/allowed access for an item
  • The API works with crawled properties, and in SharePoint the crawled properties must have a name and a GUID. When you add a crawled property to an item, you can set the crawled property GUID for the IDocumentFeeder instance, or for the property instances

By using the Content API, you create your own custom crawled properties. In the FAST Search Server farm, you should create a new crawled property category for the new crawled properties:

    $guid = [guid]::NewGuid()

    $cat = New-FASTSearchMetadataCategory -Name xmlprops -Propset $guid

    Set-FASTSearchMetadataCategory -Category $cat

    $guid

Note the GUID value. This is a unique value that you need to use in the API client.

In most cases it is convenient to use one crawled property category for all crawled properties used in the API, and then map the crawled properties to managed properties within the index schema. In that case, you only need to specify the GUID in IDocumentFeeder (PropertySet). You can also specify crawled property category for individual properties. When you create a 'Document', and wants to specify a unique GUID for the crawled properties, you can do that by adding elements of type Property (e.g. StringProperty), and add the GUID you have created above using PropertySet. 

You must also create a unique content collection for the Content API client. Do not use the default 'sp' collection.

For detailed documentation of the API interfaces, refer to the .chm help file that is part of the download.

Download: Content API for Microsoft FAST Search Server 2010 for Sharepoint

     

]]> Customize your search result sorting https://docs.microsoft.com/archive/blogs/knutbran/customize-your-search-result-sorting Tue, 15 Mar 2011 13:08:00 GMT https://blogs.msdn.microsoft.com/knutbran/2011/03/15/customize-your-search-result-sorting/ One of the nice features if you create your own query client with FAST Search for SharePoint is all the options to customize search result sorting. The most flexible option is the 'sort by formula' feature. As you may guess, this feature enables you to define your own formula defining how to sort your query results. Why would you want to do that? The product already gives you the ability to sort by relevance, which is computed based on a number of factors related to how well the query match your content. You can also sort by the value of managed properties, such as dates, names, etc. However, in some cases you may have a more specific sorting requirement that is not covered by numeric or text sorting.

The probably most relevant use case for this feature is sorting by geographical distance. Given that you can supply your own position with latitude/longitude for the query, and the indexed items also includes this information, you can create a formula that can sort by distance from your own position to the position given in the indexed item.

Another possible use case is to sort by the value of a managed property, but map the values to given buckets before sorting. Then you can use this in a multi-level sorting specification, and apply some other sorting mechanism for values within the same bucket.

You can find more detailed syntax description, examples, and information on how to use the feature from the query web service in this article:

]]>