•  "Failed to connect to SERVER Failed to initialize session with document engine: Unable to resolve Contentdistributor."

We found that FAST certificate had expired. We updated the certificate and crawl ran without issue. however, next day crawl started behaving as earlier. this time the error logged in eventlog was different:

•  "Failed to connect to SERVER. Failed to initialize session with document engine: Unable to find master Contentdistributor.  Ensure that all Contentdistributors in the ESP installation are listed when creating icontent_manager."

On investigating we found that during the updation of the certificate, new certificate was generated on the admin node and non-admin node was still retaining the old certificate.

Issue:

      Certificate was not properly updated in all the FAST non-admin nodes.

Resolution :

• Exported the certificate from admin node 
• imported into non-admin node where secondary content distributor was running 
• stopped fast services on non-admin node 
• ran installer\scripts\replacedefaultcertificate.ps1 -thumbprint "certthumbprint" 
• started fast services on non-admin node

More details on managing certificates can be found here.

We performed an incremental crawl and amazingly we got some of the data searchable i.e. whatever data was re-crawled was not retrievable. Looking at the indexer info we got something similar to below details:

<documents size="26854297742.000000" total="XXXXX" indexed="YYYYY" not_indexed="ZZZZZ"/>

Here XXXXX = YYYYY + ZZZZZ.

This clearly showed that there were some issue with the indexes. We needed to re-index the data to get the issue fixed. we used the below command for the same:

We will be using XMLHttpRequest object for our request and response.

We need to undertand the below XML structures for moving ahead:

1] SOAP request for search service. Here, the request XML is for 'QueryEX' method. 'queryXml' will contain the actual query packet for search, as indicated by string.

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:xsi="https://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="https://www.w3.org/2001/XMLSchema" xmlns:soap="https://schemas.xmlsoap.org/soap/envelope/">
  <soap:Body>
    <QueryEx xmlns="https://microsoft.com/webservices/OfficeServer/QueryService">
      <queryXml>string</queryXml>
    </QueryEx>
  </soap:Body>
</soap:Envelope>

 Let's assume this is represented by variable 'requestXML'.

2] SOAP response for search service. Here, the response XML is for 'QueryEX' method. 'QueryExResult' will contain the result, as indicated by xml.

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:xsi="https://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="https://www.w3.org/2001/XMLSchema" xmlns:soap="https://schemas.xmlsoap.org/soap/envelope/">
  <soap:Body>
    <QueryExResponse xmlns="https://microsoft.com/webservices/OfficeServer/QueryService">
      <QueryExResult><xsd:schema>schema</xsd:schema>xml</QueryExResult>
    </QueryExResponse>
  </soap:Body>
</soap:Envelope>

  Let's assume this is represented by variable 'responseXML'.

3] FQL Query Packet for search, indicating the search criteria, refiners, sorting, etc.

<QueryPacket Revision=\"1000\">
  <Query>
    <Context>
      <QueryText language=\"en-US\"type=\"FQL\">string(\"-\",Mode=\"AND\")</QueryText>
    </Context>
    <SupportedFormats Format=\"urn:Microsoft.Search.Response.Document.Document\"/>
    <ResultProvider>FASTSearch</ResultProvider>
    <Range>
      <StartAt>1</StartAt>
      <Count>20</Count>
    </Range>
    <EnableStemming>true</EnableStemming>
    <EnableSpellCheck>Suggest</EnableSpellCheck>
    <IncludeSpecialTermsResults>true</IncludeSpecialTermsResults>
    <IncludeRelevantResults>true</IncludeRelevantResults>
    <ImplicitAndBehavior>true</ImplicitAndBehavior>
    <TrimDuplicates>true</TrimDuplicates>
    <Properties>
      <Property name=\"Rank\"/>
      <Property name=\"Title\" />
      <Property name=\"Author\"/>
      <Property name=\"Size\" />
      <Property name=\"Path\" />
      <Property name=\"Write\"/>
      <Property name=\"SiteName\" />
      <Property name=\"HitHighlightedSummary\"/>
      <Property name=\"HitHighlightedProperties\"/>
      <Property name=\"ContentClass\"/>
      <Property name=\"IsDocument\"/>
      <Property name=\"Url\" />
      <Property name=\"FileExtension\"/>
      <Property name=\"SpSiteUrl\"/>
      <Property name=\"docvector\"/>
      <Property name=\"fcocount\" />
      <Property name=\"fcoid\"/>
    </Properties>
  </Query>
</QueryPacket>

 Let's assume this is represented by variable 'queryPacket'. this will replace the value string in step [1] for requestXML.

We can write a function incorporating all the above for calling the service and getting the response as follows:

function RetrieveSearchData()

{

    //get the request object

    var requestObject = new XMLHttpRequest();

    //Initializes request and specifies the method, URL, and other information for the request.

    requestObject.open("POST", "https://SERVER:PORT/_vti_bin/search.asmx", false);

    //set request header

    requestObject.setRequestHeader("Content-Type", "text/xml; charset=utf-8");

    //Get FQL queryPacket along with the SOAP envelope

    var packet = requestXML;

    //Sends an HTTP request to the server and receives a response

    requestObject.send(packet);

    //process responseXML

}

There are multiple ways to achieve the above processing in AJAX, ASP.net or traditional C#. This example is just one of the ways to process SharePoint webservice in javascript code.

In windows 8, access to folder system is provided by api's that are async in nature and 'Task' based. Therefore, we will use 'async' methods for this operation.

        private async Task<SampleData> ReadFromConfigurationFile()

        { 

         try

            {

                //get the package reference

                Windows.ApplicationModel.Package package = Windows.ApplicationModel.Package.Current;

                //get installed location detail

                Windows.Storage.StorageFolder storageFolder = package.InstalledLocation;

               // Use serializer class

                XmlSerializer serializer = new XmlSerializer(typeof(SampleData));         

                try

                {

                    //file variable

                    StorageFile sampleFile = null;

                     try

                    {

                       //Get the file

                        sampleFile = await storageFolder.GetFileAsync("ConfigurationStored.xml");

                    }             

                    catch (FileNotFoundException ex)

                    {

                        //handle exception

                    }                 

                  //open file in read mode

                   var file = await sampleFile.OpenAsync(FileAccessMode.Read);   

                   //Read stream         

                   Stream inStream = Task.Run(() => file.OpenRead()).Result;       

                   return (SampleData)serializer.Deserialize(inStream);

                }           

               catch (Exception ex)

                {

                 //handle exception 

                }

            }

           catch (Exception ex)

            {

             //handle exception

            }       

         return null;

        }

Note the use of await and async pattern for handling the asynchronous operation.

If you want to modify the default value and write the file back, open file in write mode and after modification save the same at the installed location.

More sample on file access can be found at https://code.msdn.microsoft.com/windowsapps/File-access-sample-d723e597.

The core function of BDC is to provide connectivity support to the following types of external systems:

• Databases

• Web/WCF services

• Microsoft .NET Framework connectivity assemblies

• Custom data sources

There are scenarios when we wish to do custom implementation to achieve the desired processing. This can be achieved by developing Visual Studio based code solutions. We add up code to do the required processing for implementing List\Item operations. These solutions work great and provide a lot of flexibility.

However, in scenarios where BCS is used to pull data and index in SharePoint for implementing search driven applications, we need constant updation of data from the external system. In such systems we need incremental update to work to ensure CRUD operations. In such a scenario we need to create solutions that implement ChangedIDEnumerator\DeletedIDEnumerator apart from general List (IDEnumerator), Item (ReadItem) operations.

IDEnumerator - Used for list operation

ChangedIDEnumerator - Identifies the modified items

DeletedIDEnumerator - Identifies the deleted items

ReadItem - used for Item operation

apart from the above we may need to implement streamaccessor method for ingesting streams for indexing like documents.

Here, we will have a look on the methods and the mapping in model file. 

IDEnumerator

Syntax: public static IEnumerable<EntityType> IDEnumerator()

Returns an enumerator of EntityType. This will have all the items that need to be indexed. This can be infered as the finder method.

Model file mapping:

            <Method IsStatic="false" Name="IDEnumerator">
              <Parameters>
                <Parameter Direction="Return" Name="returnParameter">
                  <TypeDescriptor TypeName="System.Collections.Generic.IEnumerable`1[EntityType Class details]" IsCollection="true" Name="ListName">
                    <TypeDescriptors>
                      ...
                    </TypeDescriptors>
                  </TypeDescriptor>
                </Parameter>
              </Parameters>
              <MethodInstances>
                <MethodInstance Type="IdEnumerator" ReturnParameterName="returnParameter" Default="true" Name="EnumeratorName" DefaultDisplayName="DisplayName">
                  <Properties>
                    <Property Name="RootFinder" Type="System.String">x</Property>                    
                  </Properties>
                </MethodInstance>
              </MethodInstances>
            </Method>

ChangedIDEnumerator

         <Method IsStatic="false" Name="ChangedIDEnumerator">
              <FilterDescriptors>
                <FilterDescriptor Name="LastRunDate" Type="InputOutput">
                  <Properties>
                    <Property Name="SynchronizationCookie" Type="System.String">LastModifiedDate</Property>
                  </Properties>
                </FilterDescriptor>
                <FilterDescriptor Name="FilterDescriptor" Type="Timestamp" />
              </FilterDescriptors>
              <Parameters>
                <Parameter Name="@LastRunDate" Direction="InOut">
                  <TypeDescriptor Name="LastRunDateTypeDescriptor" TypeName="System.DateTime" AssociatedFilter="LastRunDate">
                    <Interpretation>
                      <NormalizeDateTime LobDateTimeMode="Local" />
                    </Interpretation>
                  </TypeDescriptor>
                </Parameter>
                <Parameter Direction="Return" Name="returnParameter">
                  <TypeDescriptor TypeName="System.Collections.Generic.IEnumerable`1[EntityType Class details]" IsCollection="true" Name="ListName">
                    ...
                  </TypeDescriptor>
                </Parameter>
              </Parameters>
              <MethodInstances>
                <MethodInstance Type="ChangedIdEnumerator" ReturnParameterName="returnParameter" Default="true" Name="OperationName" DefaultDisplayName="DisplayName" />
              </MethodInstances>
            </Method>

DeletedIDEnumerator

        <Method IsStatic="false" Name="DeletedIDEnumerator">
              <FilterDescriptors>
                <FilterDescriptor Name="LastRunDate" Type="InputOutput">
                  <Properties>
                    <Property Name="SynchronizationCookie" Type="System.String">LastModifiedDate</Property>
                  </Properties>
                </FilterDescriptor>
                <FilterDescriptor Name="FilterDescriptor" Type="Timestamp" />
              </FilterDescriptors>
              <Parameters>
                <Parameter Name="@LastRunDate" Direction="InOut">
                  <TypeDescriptor Name="LastRunDateTypeDescriptor" TypeName="System.DateTime" AssociatedFilter="LastRunDate">
                    <Interpretation>
                      <NormalizeDateTime LobDateTimeMode="Local" />
                    </Interpretation>
                  </TypeDescriptor>
                </Parameter>
                <Parameter Direction="Return" Name="returnParameter">
                  <TypeDescriptor TypeName="System.Collections.Generic.IEnumerable`1[EntityType class details]" IsCollection="true" Name="ListName">
                    <TypeDescriptors>
                      ...
                    </TypeDescriptors>
                  </TypeDescriptor>
                </Parameter>
              </Parameters>
              <MethodInstances>
                <MethodInstance Type="DeletedIdEnumerator" ReturnParameterName="returnParameter" Default="true" Name="OperationName" DefaultDisplayName="DisplayName" />
              </MethodInstances>
            </Method>

       <Method IsStatic="false" Name="ReadItem">
              <Parameters>
                <Parameter Name="ID" Direction="In">
                  <TypeDescriptor Name="ID" TypeName="System.String" IdentifierEntityName="ENTITY" IdentifierEntityNamespace="NAMESPACE" IdentifierName="ID">
                  </TypeDescriptor>
                </Parameter>
                <Parameter Name="returnParameter" Direction="Return">
                  ...
                </Parameter>
              </Parameters>
              <MethodInstances>
                <MethodInstance Name="ReadItem" Type="SpecificFinder" ReturnParameterName="returnParameter" Default="true" DefaultDisplayName="DisplayName">
                  <Properties>
                    <Property Name="LastDesignedOfficeItemType" Type="System.String">None</Property>
                    <Property Name="WindowsSecurityDescriptorField" Type="System.String">SecurityDescriptor</Property>
                  </Properties>
                </MethodInstance>
              </MethodInstances>
            </Method>

Here, SecurityDescriptor is used for implementing security at record level.

StreamAccessor

Syntax: public Stream GetStream(string ID)

GetStream can be used if we intend to ingest documents in the crawl process. ID is the unique identifier.

Mapping in Model:

          <Method IsStatic="false" Name="GetStream">
              <Parameters>
                <Parameter Name="ID" Direction="In">
                  <TypeDescriptor Name="ID" TypeName="System.String" IdentifierEntityName="ENTITY" IdentifierEntityNamespace="NAMESPACE" IdentifierName="ID" />
                </Parameter>
                <Parameter Name="returnParameter" Direction="Return">
                  <TypeDescriptor Name="returnParameterTypeDescriptor" TypeName="System.IO.Stream" />
                </Parameter>
              </Parameters>
              <MethodInstances>
                <MethodInstance Name="MethodName" Type="StreamAccessor" ReturnParameterName="returnParameter">
                  <Properties>
                   ...
                  </Properties>
                </MethodInstance>
              </MethodInstances>
            </Method>

More on custom connectors can be found here.

Another great post on BCS can be found here.

Set-ExecutionPolicy : Windows PowerShell updated your execution policy successfully, but the setting is overridden by a policy defined at a more specific scope. Due to the override, your shell will retain its current effective execution policy of "Unrestricted". Type "Get-ExecutionPolicy -List" to view your execution policy settings. For more information, please see "Get-Help Set-ExecutionPolicy.

"Set-ExecutionPolicy" cmdlet is used for changing the user preference for the Windows PowerShell execution policy. details can be found here.

If the current logged-in user is an administrator on the system with proper access but still getting the above issue, following blogpost instruction may be helpful in re-setting group policy.

However, if the issue persists and "Get-ExecutionPolicy -List" command shows something similar as shown below:

    MachinePolicy = RemoteSigned
    UserPolicy       = Undefined
    CurrentUser     = Unrestricted
    LocalMachine  = RemoteSigned

Execute the below commands in powershell commandline as an administrator:

Set-ExecutionPolicy "RemoteSigned" -Scope Process -Confirm:$false

Set-ExecutionPolicy "RemoteSigned" -Scope CurrentUser -Confirm:$false

This will set the execution policy for the given scope.

On executing "Get-ExecutionPolicy -List", we should now see the below:

    MachinePolicy = RemoteSigned
    UserPolicy       = Undefined
    Process           = RemoteSigned
    CurrentUser     = RemoteSigned
    LocalMachine  = RemoteSigned

For example, in indexed data few words like "ABC" are very common, phrase searches involving these words may show less relevant results than expected. Suppose we search with ABC XYZ or "ABC XYZ" or "XYZ ABC", ideally we should get different result set based on the existence of these phrases in indexed data. However, "Phrase" search behavior changes for very frequent indexed keywords like ABC and it treats the Phrase "ABC XYZ" as ABC AND XYZ.

Expected behavior can be ensured by changing PositionStopWordThreshold and StopWordThreshold values for RankProfile in FAST admin node. New value should be changed based on the specific scenario of the target farm and volume of data. Default value for Rank profile are listed below:

Follow this msdnlink for details.

 In order to do the changes, need to execute below in FAST Server management powershell:

$RankProfile = Get-FASTSearchMetadataRankProfile -Name <RankProfileName>

$RankProfile.StopWordThreshold = <new integer value>

$RankProfile.PositionStopWordThreshold  = <new integer value>

$RankProfile.Update()            

Restart  FAST services.

There are few downside to these changes and must be analyzed before the changes( Follow this msdnlink for details)

• Increase in CPU usage
• Slower Search Performance

In eventviewer of the crawl component observed error: "The search service stopped the filter daemon because it was consuming too many resources. A new daemon will automatically be started, and no user action is required."

This got resolved following the suggested changes for increasing registry values for DedicatedFilterProcessMemoryQuota, FilterProcessMemoryQuota, FolderHighPriority.

Add-PSSnapin : No snap-ins have been registered for windows powershell version 2

This was resolved by changing the path of the powershell.exe used to initiate the management shell. In properties -> Shortcut -> Target, powershell.exe was pointing to sysWOW64 folder. I changed the same to point to System32 [c:\windows\system32\windowspowershell\v1.0\powershell.exe].