Richard Palawan

Microsoft Corporation, April 2012

Background

When setting up an enterprise search service, it is necessary to apply specific access restrictions different parts of the crawled content. When crawling database content into FAST Search for SharePoint 2010, using the BDC framework is the preferred method for implementing item-level security in FAST Search. However, there are conditions under which using the FAST JDBC Database Connector is more appropriate. Refer to online documentation for reference; e.g. https://msdn.microsoft.com/en-us/library/ff383278.aspx .

There are a few references and samples that describe the process of implementing item-level security using BDC. However, there is little documentation on how to achieve the same result using the JDBC Database connector. In this writing, we go over a JDBC-based implementation.

Security Model – Item Level

The architecture of FAST provides a means of storing a Security Descriptor (SD) for each index record. The Security Descriptor contains two lists: Granted SID, Denied SID, where SID is a Security Identifier of a security principal such as a user or a group. The SID can refer to varying security providers such as Active Directory (AD), Forms Authentication, and Windows Live.

Search Results Trimming

When a call is received by the FAST query processor, it also contains a claims ticket indicating the identity of the end user and the user's Security Groups (SG). Since each record in the search index may contain a Security Descriptor, the FAST query engine intersects the user's identity and associated SG's against the record's access control list (ACL), and if access is granted, and access is not denied, the record is included in the result set; otherwise, it is skipped.

Injecting the Security Descriptor as Part of the Database Record

In order to implement this using the JDBC database connector, we need to assign the information in the security descriptor directly to a special property in the FAST engine named docacl. The feed processing pipeline recognizes the special property and does not convert the value into a crawled property, but instead assigns the value directly to the docacl field in the index.

Generating a Security Descriptor

Here is the description of the docacl value that holds the security descriptor information according to MSDN (https://msdn.microsoft.com/en-us/library/ee626835(office.12).aspx):

"Attribute key: docacl

Attribute type: cht::documentmessages::string_attribute

Attribute value:

The docacl attribute MUST contain a list of space separated entries in the following format:

<deniedRightFlag><userStoreID><securityIdentifier>. <deniedRightFlag> is "9" if the rest indicates a deny permission or "" if it is a grant permission. <userStoreID> is the user store identifier of the user/group given in <securityIdentifier>. <securityIdentifier> is the user/group security principal identifier that was granted or denied permission to the item. A docacl value MUST only contain alphanumeric characters. If the <securityIdentifier> contains other characters, the <securityIdentifier> MUST be encoded with a base-32 variant of [RFC4648] using an alphabet with a-z and 1-6 and no equal sign (=) padding at the end."

On any server where FAST Search is installed, we get two PowerShell cmdlets that encode and decode security principal entries according to the base32 encoding scheme mentioned above.

Get-FASTSearchSecurityEncodedSid

https://technet.microsoft.com/en-us/library/ff393821.aspx

Get-FASTSearchSecurityDecodedSid

https://technet.microsoft.com/en-us/library/ff393823.aspx

Here is an encoding example:

Get-FASTSearchSecurityEncodedSid -SID S-1-5-21-255901184-1604012920-1887928677-2217443

which results in the following encoded string:

aecqaaabgsakfiaaaakazoppz3exg16345io23kqyaa

Note that it probably would be easier to use the -Users option and the domain user name such as domain1\user1.

*Note: All SID and encoded strings used in this writing are intentionally invalid. They are used only to illustrate expected formats.

In order to inject this as the Security Descriptor, you would need to add the User Store ID, and possibly the Denied flag if this is a principal that is denied access. If this an Active Directory account, the User Store ID would be 'win'. The denied flag is '9', pre-pended to the overall string.

Example of an allowed account:

winaecqaaabgsakfiaaaakazoppz3exg16345io23kqyaa

Example of a disallowed account:

9winaecqaaabgsakfiaaaakazoppz3exg16345io23kqyaa

If you need to set multiple entries of disallowed and allowed entries, concatenate the encoded strings using a space separator, starting with the disallowed list:

9winaecqaaabgsakfiaaaakazoppz3exg16345io23kqyaa winaecqaaabgsakfiaaaakazoppz3exg16345io23kqyaa

If you need to look at actual FAST encoded strings for your security principals, you run a sample collection using the BDC SQL provider and capture the strings in the generated FiXML files, search for 'docacl'.

Adding the docacl as a Select Field in the JDBC Database Connector Configuration XML File

Once you obtain the values from running Get-FASTSearchSecurityEncodedSid, pre-pending the User Store ID and optionally the denied flag, and concatenating multiple encoded SID's into a single string, the result is ready to be inserted into the FAST index along with the rest of the document content. We achieve this by adding a new field named docacl to the SQL Select statement in the JBDC Database Connector configuration file.

In the following example, I need to set the same value of the Security Descriptor to all of the documents in a particular content source. In the JDBC Database Connector configuration XML file, I modified the SQL statement to add the docacl field as follows:

…

<parameter name="JDBCSQL" type="string"> -<description>

<![CDATA[This or JDBCSQLFile must be provided. SQL query to crawl against. Note, any valid SQL is valid here. <br>Use %TIMESTAMP% where last crawl time gets inserted as a datetime value. <br>Use %TIMESTAMPSEC% where last crawl time gets inserted as number of seconds since epoch <br> Examples: <br> Oracle: SELECT * FROM tableName WHERE dateField > TO_TIMESTAMP('%TIMESTAMP%','yyyy-MM-DD"T"hh24:mi:ss')&nbsp; <br>Note that the time stamp format used must be as indicated here. <br>MS SQL Server: SELECT * from tableName WHERE dateField > convert(datetime,'%TIMESTAMP%',126)<br>select * from employees <br>Default: (none)]]>

</description> -<value>

<![CDATA[SELECT [field1],[field2] ,…,[fieldn], '9winaecqaaabgsakfiaaaakazoppz3exg16345io23kqyaa winaecqaaabgsakfiaaaakazoppz3exg16345io23kqyaa' AS [docacl] FROM [server1].[sch1].[table1]]]>

</value>

…

where the value in quotes '9winaecqaa … winaecqaaa…' was generated by running the FAST cmdlet and concatenating the output as described earlier.

If you need to set a different value for each document, you would have to set the Security Descriptor encoded value in the source database table.

This is a complete implementation of a JDBC Database Connector based item-level security.

Remember to open up a PowerShell window from the Start menu:

Start --> Fast Search Server 2010 for SharePoint (right click --> Run as Administrator)

The Short Version

1. Add the SharePoint PowerShell cmdlets

   Add-PSSnapin Microsoft.SharePoint.PowerShell -ErrorAction SilentlyContinue

2. Create and configure the Content Source (enter a URL that doesn't mind you crawling it. Perhaps your blog page?)

   $contentSSA = "FASTContent"

   $startaddress = [enter a URL here]

   $contentsourcename = "Web site crawl"

   $contentsource = New-SPEnterpriseSearchCrawlContentSource -SearchApplication $contentSSA -Type Web -name $contentsourcename -StartAddresses $startaddress -MaxSiteEnumerationDepth 0

3. Start the crawl

   $contentsource.StartFullCrawl()

   $contentsource.CrawlStatus

   1. Execute $contentsource.CrawlStatus again.
   2. Wait. Wait. Wait. Wait.
   3. Execute $contentsource.CrawlStatus again.
   4. Wait. Wait. Wait. Wait.
   5. Keep executing $contentsource.CrawlStatus until the status changes to CrawlCompleting and then Idle
4. Execute a search

The Long Version

Again, there really isn't any reason to go over all the steps as they don't really change from step to step. I do want to clarify a few things.

• As we installed the advanced filter pack in a previous post we don't need to do that again.
• In the line of PS that creates the content source

  $contentsource = New-SPEnterpriseSearchCrawlContentSource -SearchApplication $contentSSA -Type Web -name $contentsourcename -StartAddresses $startaddress -MaxSiteEnumerationDepth 0

  It is interesting to note that the New-SPEnterpriseSearchCrawlContentSource cmdlet defaults to the Custom crawl rule which will read all pages and all links found at the starting URL. In order to mimic the behavior of the previous blog post (and avoiding crawling the world) we set MaxSiteEnumerationDepth to zero which causes the crawler to read the content at the site we started at rather than allowing the crawler to go into ADD mode becoming easily distracted and chasing down every car that goes by.

Thanks

Much thanks to Runar Olsen and Ben Berenstein for letting me bounce ideas off of them (you can both take your helmets off now).

References

SharePoint Server 2010 Search: Windows PowerShell Cmdlets

Crawl before you walk. Walk before you run. Run so you don't get eaten by a lion.

Let's get to it.

What we are doing this time

1. Create a new content source: a crawl of an arbitrary external web site
2. Run the crawl
3. Check the results

The Short Version

1. Create a new content source: a crawl of an arbitrary external web site
   1. Start --> SharePoint 2010 Central Administration --> Manage Service Applications --> FASTContent --> Crawling --> Content Sources --> New Content Source
      1. Name: Web site crawl
      2. Content Source Type: Web sites
      3. Start Addresses: [your favorite web site. Just one for now. Preferably not a commercial site. How about your favorite blogger? Preferably something small]
      4. Crawl Settings: Only crawl within the server of each start address
      5. Crawl Schedule: None
      6. Start Full Crawl: [check] Start full crawl of this content source
2. Run the crawl
   1. Click OK
   2. Wait
   3. Refresh
   4. Repeat b and c until the status of the crawl is Idle
3. Check the results
   1. Go to https://intranet.contoso.com/fast-search
   2. Search for #
   3. Take a look at the Site refiner. You should see one additional site: the site you crawled.

The Long Version

There really isn't a Long Version. The basic configuration of the web crawler is straightforward and pretty much covers it. If you need different behavior then please add complexity incrementally so that you can confirm the behavior you are looking for. Perhaps we will cover crawl rules in a future post.

In my case I crawled a WordPress site that added an alleged 948 pages to the index. That, of course, is a lie. There are a lot of duplicate documents as the crawler crawls the originating page and well as any links on that page; that means tag links, RSS feed information and as well as links associated with pages. A link is a link is a link even if the content it points to had been crawled before.

Notice that many of the documents have duplicates.

Also take a look at the Company refiner.

Those companies were not there before. The Company entity extractor works automatically and has done its job in this case (sometimes too well as names like Galactica and JSP are listed as companies when they are not).

Are there wrinkles to this? Absolutely! But remember: the point of this series is to get you up and running as quickly as possible. Once something simple works add complexity slowly (and that's two adverbs worth of advice).

How do we do this with PowerShell? Read Part 3a.

Next: Crawling SharePoint 2010 using the SharePoint 2010 Connector

References

Add, edit or delete a content source (FAST Search Server 2010 for SharePoint)

Get the details and register here.

Adding *Your* Business Value With FAST
Search for SharePoint Pipeline Extensibility

It’s well known
that SharePoint provides a powerful set of APIs for integrating custom code into your business’s SharePoint applications .  But FAST Search For SharePoint (FS4SP) provides a new capability – a window into your data between the crawl and the index.  Getting programmatic access to your data at that point opens up a world of possibilities.

This session will provide an overview of what this extensibility looks like, recommendations on how to implement it effectively, and a discussion of what you can do with it to add your specific business value to your search applications.

Speaker Information
Steve Kuenzli, is an Architect at Microsoft, specializing in Search.  His work in Enterprise Search dates back over the ten years in both the public and private sectors.

The meeting will be held in the Microsoft offices at 1290 Avenue of the Americas. Register early!

I hear you ask: but how would I do that in PowerShell? What if my GUI stops working and I just have to get my Fast Search Center created, and content source created and configured?

I am so glad you asked. Leo Souza wasn't that happy as I asked him and he basically gave me the information I am about to transcribe. Aren't I lucky to have such a good friend?

What we did in Part 2 breaks down into 3 parts:

1. Install the Advanced Filter Pack. Since we did that in PowerShell we have already crossed that line.
2. Add the SharePoint PowerShell cmdlets
3. Create the Fast Search Center
4. Create and configure a Content Source
5. Start the crawl
6. Execute a search

Remember to open up a PowerShell window from the Start menu:

Start --> Fast Search Server 2010 for SharePoint (right click --> Run as Administrator)

Now let's do each of the above steps in turn.

1. Install the Advanced Filter Pack
   1. From the PowerShell window:

   > cd ..\installer\scripts

   > .\AdvancedFilterPack.ps1 -enable

   [answer y to the only question it asks]

   There is no need to restart the machine or any of the services.

   

2. Add the SharePoint PowerShell cmdlets

   Add-PSSnapin Microsoft.SharePoint.PowerShell -ErrorAction SilentlyContinue

3. Create the Fast Search Center

   $template = Get-SPWebTemplate "SRCHCENTERFAST#0" # Want to know more? Go to https://technet.microsoft.com/en-us/library/ff678213.aspx

   New-SPWeb -Url "https://intranet.contoso.com/fast-search" -Name "TechNet VM Fast Search Page" -Template $template

4. Create and configure a Content Source

   $contentSSA = "FASTContent"

   $startaddress = "\\VBOXSVR\neurobiology-files"

   $contentsourcename = "File system crawl"

      

   $contentsource = New-SPEnterpriseSearchCrawlContentSource -SearchApplication $contentSSA -Type file -name $contentsourcename -StartAddresses $startaddress

5. Start the crawl

   $contentsource.StartFullCrawl()

   $contentsource.CrawlStatus

   1. Execute $contentsource.CrawlStatus again.
   2. Wait. Wait. Wait. Wait.
   3. Execute $contentsource.CrawlStatus again.
   4. Wait. Wait. Wait. Wait.
   5. Keep executing $contentsource.CrawlStatus until the status changes to CrawlCompleting and then Idle
6. Execute a search

   $searchSite = Get-SPWeb https://intranet.contoso.com/fast-search

   $keywordQuery = New-Object Microsoft.Office.Server.Search.Query.KeywordQuery $searchSite

   $keywordQuery.ResultTypes = [Microsoft.Office.Server.Search.Query.ResultType]::RelevantResults

    

   $keywordQuery.EnableFQL = 1

   $keywordQuery.RowLimit = 10

      

   $keywordQuery.QueryText = "#"

      

   $results = $keywordQuery.Execute()

      

   Write-Host Total hits: $results[1].TotalRows

   $resultsTable = $results.Item([Microsoft.Office.Server.Search.Query.ResultType]::RelevantResults)

      

   $resultsDataTable = $resultsTable.Table

   $resultsDataTable | Format-Table -AutoSize -Property title, url

So by rights you should create two versions of the script:

1. One that executes the code generically with things like the Fast Search Center URL, file location and content source name as incoming arguments
2. A script that calls the generic script from step 1 with the proper values for the URL, file location and content source name.

The separation gives you the flexibility to use the first script in lots of situations and the second one to execute a particular task without all typing. Get to it!

If something doesn't work please let me know. You'd be surprised where I am at any given time writing this blog (and this time I am at TechReady14 sitting in a Starbucks making sure this goes out in one piece.).

References and Thanks

Create a new FAST Search Center Site Collection using PowerShell by Brent Groom (thanks, Brent!)

FAST Search for SharePoint 2010–Using PowerShell to execute search by Satish TK (Thanks, Satish!)

Testing Custom Query Features (FAST Search Server 2010 for SharePoint) (oh, author of unknown origin: thanks!)

Scripted deployment reference (SharePoint Foundation 2010) 

The code

[Not sure how to run these scripts? Check out Running Windows PowerShell Scripts at TechNet. You want to make them more useful? Slice and dice them even more. I won't be doing that...]

CreateContosoFastSearchCenter.ps1

CreateAFastSearchCenter.ps1 "https://intranet.contoso.com/fast-search" "TechNet VM Fast Search Page" "CONTOSO\Administrator"

CreateAFastSearchCenter.ps1

# CreateAFastSearchCenter.ps1 <Site URL> <Page Title>

function usage($cmd) {
   write-host ""
   write-host "Create a FAST Search Center"
   write-host "Create a FAST Search Center using the incoming URL and name"
   write-host ""
   write-host "usage: $cmd <Site URL> <Page Title> <Owner Alias>"
   write-host " <Site URL> - Specifies the new FULL URL for the site (for example, https://intranet.contoso.com/fast-search)"

   write-host " <Page Title> - The title to give this search center"

   write-host " <Owner Alias> - DOMAIN\alias of the site owner"

   exit
}

# argument checks
switch ($args.count) {
   3 { $siteurl = $args[0]
       $pageTitle = $args[1] }

   default { usage $myinvocation.mycommand.definition }
}

# Put this here or execute it before you run the script

Add-PSSnapin Microsoft.SharePoint.PowerShell -ErrorAction SilentlyContinue

$template = Get-SPWebTemplate "SRCHCENTERFAST#0"

write-host "Creating the Fast Search Center at" $siteurl

New-SPWeb -Url $siteurl -Name $pageTitle -Template $template

write-host "Finished!"

ConfigureContosoFileFASTContentSourceAndStartAFullCrawl.ps1

ConfigureAFileFASTContentSourceAndStartAFullCrawl.ps1 "File system crawl" "\\VBOXSVR\neurobiology-files"

ConfigureAFileFASTContentSourceAndStartAFullCrawl.ps1

# ConfigureAFileFASTContentSourceAndStartAFullCrawl.ps1 <ContentSourceName> <FileSystemStartAddress>

function usage($cmd) {

   write-host ""

   write-host "Configure A File System Content Source And Start A Full Crawl"

   write-host ""

   write-host "usage: $cmd <ContentSourceName> <FileSystemStartAddress>"

   write-host " <ContentSourceName> - Name to give this content source"

   write-host " <FileSystemStartAddress> - Starting point for the file system crawl"

   exit

}

# argument checks

switch ($args.count) {

   2 { $contentSourceName = $args[0]

       $startAddress = $args[1] }

   default { usage $myinvocation.mycommand.definition }

}

# Put this here or execute it before you run the script

Add-PSSnapin Microsoft.SharePoint.PowerShell -ErrorAction SilentlyContinue

$contentSource = New-SPEnterpriseSearchCrawlContentSource -SearchApplication "FASTContent" -Type file -name $contentSourceName -StartAddresses $startAddress

$contentsource.StartFullCrawl()

$contentsource.CrawlStatus

$sleepTime = 30

while ($true)

{

   Write-Host "Current status: " $contentsource.CrawlStatus

   if ($contentsource.CrawlStatus -eq "Idle")

   {

     break

   }

   write-host "Will check status again in" $sleepTime "seconds..."

   Start-Sleep -seconds $sleepTime

}

write-host "Finished!"

ExecuteContosoSimpleFQLQuery.ps1

ExecuteASimpleFQLQuery.ps1 "https://intranet.contoso.com/fast-search" $args[0]

[To execute the "# AND isDocument=true" query use "# and isDocument:true" as the script assumes the use of FQL not the SharePoint default of KQL.]

ExecuteASimpleFQLQuery.ps1

# ExecuteASimpleFQLQuery.ps1 <URL> <Query>

function usage($cmd) {
   write-host ""
   write-host "Execute a simple FQL keyword query"
   write-host ""
   write-host "usage: $cmd <URL> <Query>"
   write-host " <URL> - URL to the Fast Search Center"

   write-host " <Query> - Simple FQL to execute"

   exit
}

# argument checks
switch ($args.count) {
   2 { $url = $args[0]

       $query = $args[1] }
   default { usage $myinvocation.mycommand.definition }
}

# Put this here or execute it before you run the script

Add-PSSnapin Microsoft.SharePoint.PowerShell -ErrorAction SilentlyContinue

# argument checks

$searchSite = Get-SPWeb $url

$keywordQuery = New-Object Microsoft.Office.Server.Search.Query.KeywordQuery $searchSite

$keywordQuery.ResultTypes = [Microsoft.Office.Server.Search.Query.ResultType]::RelevantResults

$keywordQuery.EnableFQL = 1

$keywordQuery.RowLimit = 10

$keywordQuery.QueryText = $query

$results = $keywordQuery.Execute()

Write-Host Total hits: $results[1].TotalRows

$resultsTable = $results.Item([Microsoft.Office.Server.Search.Query.ResultType]::RelevantResults)

$resultsDataTable = $resultsTable.Table

$resultsDataTable | Format-Table -AutoSize -Property title, url

There are a number of ways to make the folders disappear.

In a standard search result using my sample corpus:

I specifically clicked on Site --> \\vboxsvr to narrow the results to just my content and the first 55 results are folders. Bleech.

There are a number of ways to fix this:

1. Write a query to tell the system to ignore folders
2. Configure the Search Core Results web part to tells the system to ignore folders
3. Write a custom web part to submit a query to tell the system to ignore folders

I sense a pattern. No matter how you try to do this it will always end with sending a query to Fast telling the system to ignore the folders. Freaky.

Let's fix the obnoxious folder behavior.

1. The first way is rather primitive and does not sit well with the average user of search. It involves understanding query syntax and Boolean logic and perhaps even advanced arithmetic. We have already used this in the last post.

   In order to have the search results return only documents you add isDocument=true. The word AND is capitalized so the query uses it as an operator and not as part of the keyword.

1. The second way involves editing the Search Core Results web part.
   1. In the Fast Search Center select Site Actions --> Edit Page.
   2. Scroll down to the Search Core Results web part and select Edit Web Part.

      

   3. Scroll back to the top and upper right hand corner of the page to find the configuration for this web part.

      

   4. Open Results Query Options and enter in the Append Text To Query field: isDocument=true.

      

   5. Click OK.
   6. Click Save and Close on the page.
   7. Rerun the query and see just documents returned! Good times!
2. Write a custom web part: not happening here. Maybe in another post.

Thanks for Leo Souza for doing the legwork on this one.

The Reader's Digest version of the process is as follows:

• Configure a connector to read your content repository
• Index a sample of your content
• Create a Fast Search Center
• Execute a query

Is it really that simple in real life? From a mechanical perspective: yes. From a business solution perspective: probably not. Questions someone needs to ask are things like:

• What problem are you trying to solve?
• Which refiners (metadata categories) would make searching easier for your users?
• Have you indexed the correct corpus of content?
• Is a plain vanilla result page sufficient for your users or is a search-driven application appropriate?

These are questions equivalent to discovering the Meaning of Life so I will leave them for another day (or never, whichever comes first).

Once our content is indexed we will search it using a new search page. We will create a new Fast Search Center once everything is crawled and searchable.

The Short Version

• Install the Advanced Filter Pack
• Create a new Fast Search Center
• Open the Central Administration page.
• Open the Content Sources page, configure and run a new crawler on your content
• Execute a search in the Fast Search Center with keyword #
• Copy any other files you have into the folder holding the documents to be indexed (in this case, neurobiology-files).
• Return to the browser with the Central Administration page and hover your cursor over the File System Crawl label. An arrow will appear next to it.
• Click the arrow to open the popup menu. Click Start Full Crawl.
• Refresh.
• Wait.
• Refresh.
• Wait.
• Refresh.
• Wait.
• When the Status turns to Idle return to the Fast Search Center page and rerun the query with # (hash).
• Click on the refiner Site --> vboxsvr. The total results will be the total number of documents, the root folder and any sub-folders. My total for 80 documents, 1 root folder, and 149 sub-folders is 230 (if the number haven't changed give SharePoint more time; it will catch up).

The Long Version

• Install the Advanced Filter Pack
  • Start --> Fast Search Server 2010 for SharePoint (right click --> Run as Administrator)
  • From the PowerShell window:

    > cd ..\installer\scripts

    > .\AdvancedFilterPack.ps1 -enable

    [answer y to the only question it asks]

    There is no need to restart the machine or any of the services.

  

• Create a new Fast Search Center
  • Open IE
  • Site Actions --> New Site --> Search --> FAST Search Center
    • Title: TechNet VM Fast Search Page
    • URL: fast-search
  • Click Create
  • When the Fast Search Center page opens enter a keyword of # (hash, or sharp if you're a music major). If you have not ben experimenting with the VM a result total of 95 documents should be returned.

  

• Open the Central Administration page.
  • Start --> SharePoint 2010 Central Administration

    

  • Click on SharePoint 2010 Central Administration. This will open the main admin page in another IE window.
• Open the Content Sources page, configure and run a new crawler on your content
  • General Application Settings (the list to the left) --> Search --> Farm Search Administration --> FASTContent --> Crawling (list to the left) --> Content Sources

    

  • Click on New Content Source
    • Name: File system crawl
    • Content Source Type: File Shares
    • Start Addresses: \\VBOXSVR\neurobiology-files
    • Crawl Settings: Crawl the folder and all subfolders of each start address
    • Crawl Schedules: None [Full and Incremental]
    • Content Source Priority: Normal
    • FileStart Full Crawl: [check the box]

    As it so happens I have 1 plain text file in the folder named neurobiology-files. I moved all of the folders so that the 1 file is sitting there waiting to be crawled. More files later; first let's prove we can index 1 file.

  • Click OK to start the crawl. The Status column should read Starting. Click the Refresh link every so often until the Status changes from Starting to Crawling Full to Completing to Idle. This will take 3-4 minutes.

    

• Execute a search in the Fast Search Center with keyword #
  • Return to the IE at https://intranet.contoso.com/fast-search and re-execute the search using # (hash).
  • A result set with the full contents of the collection (in this case collection sp) is displayed. The results will be 2 more than before (97 vs. 95).
  • Under the Site refiner click vboxsvr. The results will now number 2 (one file and one folder). Success!
• Copy any other files you have into the folder holding the documents to be indexed (in this case, neurobiology-files).
• Return to the browser with the Central Administration page and hover your cursor over the File System Crawl label. An arrow will appear next to it.
• Click the arrow to open the popup menu. Click Start Full Crawl.

  

• Refresh.
• Wait.
• Refresh.
• Wait.
• Refresh.
• Wait.
• When the Status turns to Idle return to the Fast Search Center page and rerun the query with # (hash).
• Click on the refiner Site --> vboxsvr. The total results will be the total number of documents, the root folder and any sub-folders. My total for 80 documents, 1 root folder, and 149 sub-folders is 230 (if the number haven't changed give SharePoint more time; it will catch up).

  

  If you want to be really cool then change the query to:

  # AND isdocument=true

  Click on Site --> vboxsvr and you should see just the number of documents you indexed minus the folders they live in (if you had subfolders to begin with).

  

  How cool is that?

Examine your handy work. Take a good look at the page. Look at the refiners. Try various searches based on the keywords you know should be in the content. Look at the Sort By.

Want to know how to configure the search results to ignore folders? Read Part 2a.

How would you do this using PowerShell? Read Part 2b.

Important pages (your appids will vary)

• Central Admin
  • https://demo2010a:2010/default.aspx
• Farm Search Administration
  • https://demo2010a:2010/searchfarmdashboard.aspx
• Content Sources
  • https://demo2010a:2010/searchadministration.aspx?appid=5c78b149-0a27-44f3-b240-bb7da0f2ab42
• Crawl Logs: if something goes wrong this is the place to start looking
  • https://demo2010a:2010/_admin/search/LogContentSources.aspx?appid=5c78b149%2D0a27%2D44f3%2Db240%2Dbb7da0f2ab42
  • There are a number of useful pages here: specifically URL and Crawl History

Easter eggs

• Want to see the XML output from the Fast QRServer (Query/Response server)? Try this at home, but not at work.
  • URL: https://localhost:13280/
  • FQL Query: meta.collection:sp and #
    • Should return 281 items
  • FQL Query: meta.collection:sp and # and isdocument:true
    • Should return 98 items
• To view the status of the running Fast processes
  • Open the PowerShell window the same way described above and enter on the command line:

    nctrl status

Much thanks to Leo Souza for helping me with answers to various questions/problems I was having that were mostly self-inflicted.

Next: Crawling an External Web Site with the SharePoint 2010 Web Crawler

But we are not doing that this post. Sorry. Life can be full of disappointment. But not to worry.

At this point in the journey we need to tell the VM where to find the files we want to index (we are indexed files for this first iteration). Eventually we will tell the connector, but one step at a time. Let's copy some content into a folder on our local machine, make it accessible by the VM and index it. Sounds simple? It is...but we'll index the content in the next post.

Pre-prep

1. Go get some documents. In my case I pulled some documents from the web using Bing for keyword neurobiology (98 all told in various formats).
2. Copy the files into a local folder (I created a folder named neurobiology-files under fast-technet-files). You will map the folder onto the VM in a later step.

The Short Version

1. Start VirtualBox. Do not start the VM.
2. If you took a snapshot of your VM at the end of the last post then restore it just to be on the safe side.
3. Start the VM.
4. Map the content folder.
   1. VirtualBox Manager --> Settings --> Shared Folders --> Machine Folders --> (right click) Add Shared Folders
      1. Folder Path: [the path to the files you downloaded]
      2. Folder Name: [fills in automatically]
      3. Auto-mount [checked]
      4. Make Permanent [checked]
   2. Click OK and then OK again to close the Settings dialog.
   3. From within the VM open a Windows Explorer and in the Location field type \\VBOXSVR. You should see your folder listed (mine is listed below, but I don't know what you decided to call yours).
   4. Right-click on shared folder and select Map Network Drive…
      1. Drive: X:
      2. Folder: [should already have \\VBOXSVR\your-folder-name]
      3. Click Finish. Explorer should now list any files and folders in your shared folder.

The Long Version

1. Start VirtualBox. Do not start the VM.
2. If you took a snapshot of your VM at the end of the last post then restore it just to be on the safe side.
   1. Click Snapshots (upper right-hand button)
   2. Right click on TechNetVM-clean-install and select Restore Snapshot.
   3. Uncheck Create a Snapshot of the Current Machine State and click Restore.

   

3. Start the VM. The VM will display a dialog stating that the VM is being restored. It will complete in a few seconds.
4. Map the content folder.
   1. VirtualBox Manager --> Settings --> Shared Folders --> Machine Folders --> (right click) Add Shared Folders
      1. Folder Path: [the path to the files you downloaded]
      2. Folder Name: [fills in automatically]
      3. Auto-mount [checked]
      4. Make Permanent [checked]

      

   2. Click OK and then OK again to close the Settings dialog.
   3. From within the VM open a Windows Explorer and in the Location field type \\VBOXSVR. You should see your folder listed (mine is listed below, but I don't know what you decided to call yours).

      

   4. Right-click on shared folder and select Map Network Drive…
      1. Drive: X:
      2. Folder: [should already have \\VBOXSVR\your-folder-name]

         

      3. Click Finish. Explorer should now list any files and folders in your shared folder.

         

It is good to see a folder with content...in my case one file and folders with files.

Once again, high-fives all around. I recommend taking a snapshot (I called my new snapshot TechNetVM-mapping-content-folder-complete).

So far, 4 posts and we still haven't indexed anything. Time to do something about that. Let's configure the connector and see what all the hubbub is about.

Next: Indexing the files (finally!)

Here are our assumptions:

• You are running Windows 7
• You are running VirtualBox
• You have downloaded and extracted the VHD for Fast Search Server 2010 for SharePoint

Let's hook up VirtualBox and the VHD.

The Short Version

Creating the VM

• Start Virtual Box. Notice the friendly animals vying for your attention.
• Clicking on New (upper left hand corner) will open the Create New Virtual Machine dialog. Click Next.
• VM Name and OS Type
  • Name: TechNet FS4SP VM
  • Operating System: Microsoft Windows
  • Version: Windows 2008 (64 bit)

  Click Next.

• Memory

  Enter 8192 in the input field to the right of the slider (or half of whatever memory you do have available).

  Click Next.

• Virtual Hard Disk

  Start-up Disk: checked

  Select Use Existing Hard Disk

  Click the folder icon to the right of drop down and navigate to the location of the VHD. It should be on your desktop in the fast-technet-files\2010-10a\Virtual Hard Disks folder.

  Click Next.

• Summary

  Check the information and, if all looks well, click Create.

• The VirtualBox management console should appear with various configurations listed.
• Select the TechNet FS4SP VM (click on it once).
• Settings --> System --> Processor

  If you have a quad-core box then be a nice guy and give the VM 2 CPUs.

• Settings --> Storage

  Note that the 2010-10a.vhd is under SATA Controller.

• Right-click on 2010-10a.vhd and select Remove Attachment from the popup menu.
• Right-click on IDE Controller and select Add Hard Disk. When the VirtualBox - Question dialog opens click on Choose Existing Disk.
• The File dialog should open in the folder with the 2010-10a.vhd file. Select it and click Open.
• The VHD should now appear under the IDE Controller device. Click OK to accept all of the changes.

Initializing the VM

• Click Start (the big green arrow above the VM entry list).
• As the VM starts up a series of dialog boxes will open. Read them and close them. The VM does take a few minutes to start up.
• When the VM finally says Press Ctrl+Alt+Del to Login...don't. From the VM main menu bar select Machine --> Insert Ctrl + Alt + Del. That will bring you to the login screen.

  The default password for the TechNet Fast Search Server 2010 for SharePoint VM is pass@word1. Drivers will install, a Warm Up script will execute, Lync will open a window and eventually the Warm Up window will close (I promise it will take a long time, but it will finish).

• A Windows dialog has now opened saying you must restart your computer. Click Restart Now and watch your VM reboot.
• When the VM has completed rebooting does the Warm Up window give you an error message in bright red letters complaining about the Forefront Identity Manager Service? If it does you can safely ignore it.
• Install the Guest Additions
  • In the VM window's main menu bar select Devices --> Install Guest Additions. That mounts an ISO image.
  • Within the VM open Windows Explorer. An AutoPlay window will open; Click on Run VBoxWindownsAdditions.exe.
  • When the Welcome dialog open click Next.
  • Choose Install Location: click Next.
  • Choose Components: I suppose you can check Direct 3D support if your machine supports it, otherwise click Install.
  • Would you like to install this device software? Always click Install (you will need to do this 2 times).
  • With Reboot Now selected, click Finish.
  • Get more coffee while you wait for the reboot to complete.
• In the VM edit C:\Windows\System32\drivers\etc\hosts.

  At the bottom of the file add:

  10.0.2.15 intranet

  10.0.2.15 intranet.contoso.com

• Check the SharePoint install on the VM.
  • Open IE within the VM.
  • It should automatically open on https://intranet.contoso.com (look don't touch).
  • Go to https://demo2010a:2010 for the SharePoint administration pages (look don't touch)

The Long Version

Hooking up the VHD to VirtualBox is pretty straightforward. As you would expect, the more resources you can allocate to it the better, but I have it set up with minimum CPU and memory.

Creating the VM

• Start Virtual Box. Notice the friendly animals vying for your attention.

  

• Clicking on New (upper left hand corner) will open the Create New Virtual Machine dialog. Click Next.

  

• VM Name and OS Type
  • Name: TechNet FS4SP VM
  • Operating System: Microsoft Windows
  • Version: Windows 2008 (64 bit)

  Click Next.

  

• Memory

  Enter 8192 in the input field to the right of the slider.

  Click Next.

  

• Virtual Hard Disk

  Start-up Disk: checked

  Select Use Existing Hard Disk

  Click the folder icon to the right of drop down and navigate to the location of the VHD. It should be on your desktop in the fast-technet-files\2010-10a\Virtual Hard Disks folder.

  Click Next.

  

• Summary

  Check the information and, if all looks well, click Create.

  

• The VirtualBox management console should appear with various configurations listed.

  

  Sadly, we are not finished.

• Select the TechNet FS4SP VM (click on it once).
• Settings --> System --> Processor
• If you have a quad-core box then be a nice guy and give the VM 2 CPUs.

  

• Settings --> Storage

  Note that the 2010-10a.vhd is under SATA Controller.

  

     

• Right-click on 2010-10a.vhd and select Remove Attachment from the popup menu.
• Right-click on IDE Controller and select Add Hard Disk. When the VirtualBox - Question dialog opens click on Choose Existing Disk.

  

• The File dialog should open in the folder with the 2010-10a.vhd file. Select it and click Open.

  

• The VHD should now appear under the IDE Controller device. Click OK to accept all of the changes.

  

  Let's check the condition of the VM.

Initializing the VM

• Click Start (the big green arrow above the VM entry list).
• As the VM starts up a series of dialog boxes will open. Read them. Also, if you feel comfortable enough, click the check box that says Do Not Show This Message Again and click OK. The VM does take a few minutes to start up. Be patient.
• When the VM finally says Press Ctrl+Alt+Del to Login...don't. From the VM main menu bar select Machine --> Insert Ctrl + Alt + Del. That will bring you to the login screen.

  

  The default password for the TechNet Fast Search Server 2010 for SharePoint VM is pass@word1 (don't tell anyone, it's a secret). Entering the password and pressing Enter or clicking in the blue and white arrow will (eventually) bring you to your desktop. Drivers will install, a Warm Up script will execute (and will display a warning, multiple times, that it is Waiting for service 'Windows Update(wuauserv): to finish starting…), Lync will open a window (that you can safely Setting icon --> File --> Exit) and eventually the Warm Up window will close (I promise it will take a long time, but it will finish). Go get some coffee if you feel anxious while you wait.

  The Warm Up window finally closed? Not done yet.

• A Windows dialog has now opened saying you must restart your computer. Click Restart Now and watch your VM reboot. Did I mention going for coffee or some other comforting beverages?

• When the VM has completed rebooting, and you have logged in, does the Warm Up window give you an error message in bright red letters complaining about the Forefront Identity Manager Service (perhaps yellow?)? If it does you can safely ignore it.

  Feeling good ?

  Stop celebrating. One more thing to do.

• Install the Guest Additions
  • In the VM window's main menu bar select Devices --> Install Guest Additions. That mounts an ISO image.
  • Within the VM open Windows Explorer. An AutoPlay window will open; Click on Run VBoxWindownsAdditions.exe.

    

  • When the Welcome dialog open click Next.
  • Choose Install Location: click Next.
  • Choose Components: I suppose you can check Direct 3D support if your machine supports it, otherwise click Install.
  • Would you like to install this device software? Always click Install (you will need to do this 2 times).
  • With Reboot Now selected, click Finish.
  • Get more coffee while you wait for the reboot to complete.
• In the VM edit C:\Windows\System32\drivers\etc\hosts.

  At the bottom of the file add:

  10.0.2.15 intranet

  10.0.2.15 intranet.contoso.com

  10.0.2.15 is the static IP of the VM. If you are lucky enough to run this under Windows Server 2008 R2 and Hyper-V then you will not need the above. Running under Windows 7 and VirtualBox requires it.

• Check the SharePoint install on the VM.
  • Open IE within the VM. It should automatically open on https://intranet.contoso.com (look don't touch).

    

  • Go to https://demo2010a:2010 for the SharePoint administration pages (look don't touch)

    

That's it!

Congratulations! You may now celebrate.

This might be a good time to take a snapshot of the VM in its current pristine state so you can restore it if things really go south (Machine --> Take Snapshot). I highly recommend it (in other word: please take a snapshot so you can restore your VM to a clean starting point. I called mine TechNetVM-clean-install. You can find it under C:\Users\[user alias]\VirtualBox VMs\TechNet FS4SP VM\Snapshots not under %DESKTOP%\fast-technet-files\2010-10a\Snapshots).

Shortcuts:

• How to log on to the machine: From the main menu: Machine --> Insert Ctrl + Alt + Del
• Right Ctrl + F: expand the VM window to full screen.
• The Right Ctrl key will release the mouse from the clutches of the VM so you can use it on your desktop (the VM sometimes grabs the mouse when you click in the VM window and it cannot escape from the VM window. The default behavior is to allow you to move the mouse in and out of the VM window unencumbered).

Next: Let's make some content visible to the VM!

Well, if you are reading this then perhaps you were brave enough to download the VHD from TechNet that already has SharePoint 2010 and Fast Search Server 2010 for SharePoint (hereafter referred to as FS4SP). If you have, jump down to the section titled Setting Up the VHD on VirtualBox. If you have not, let's do that right now.

The Short Version

1. Download the VHD files from https://technet.microsoft.com/en-us/evalcenter/ee424282.aspx
2. Copy the files from your local Downloads to %DESKTOP%\fast-technet-files (create the folder)
3. From the %DESKTOP%\fast-technet-files folder double click on 2010-10a.part01.exe
4. When the wizard requests a path to create the VHD files and folders accept the full default path (which should be something like C:\User\[some user alias]\Desktop\fast-technet-files)
5. When the archive creator is done there should be folder named 2010-10a under fast-technet-files.

The Long Version

Downloading the files

When I last check on January 6, 2012 you can download the VHD for the Fast Search Server 2010 for SharePoint Trial from TechNet at https://technet.microsoft.com/en-us/evalcenter/ee424282.aspx. A Windows Live account is required is proceed and download the 24 files that constitute the final VHD.

Copy the files to a folder on your desktop

Recommendation: create a folder named fast-technet-files and move the 24 files into it. It will make the final VHD easier to find (and move, if need be). I created the folder on my desktop.

Creating the VHD

Now that you have 24 files all nicely ensconced on your drive double click on the file named 2010-10a.part01.exe. If you have never created a VHD from a RAR collection then this is your lucky day. That one file will take care of creating a series of folders and the final VHD image.

Here is the picture book version of the creation of the VHD.

Click Run.

Click Accept if you really want to do these exercises.

Click Extract to start the process. Go get coffee or soda or juice or water or whatever you like drinking while your computer is busy doing real work.

Once the extraction process is done, if you created the fast-technet-files folder on your desktop, then the executable will create a folder named 2010-10a directly below fast-technet-files . If you navigate down into 2010-10a\Virtual Hard Disks you will find the actual VHD.

Time to create the VM.

Next: Setting Up the VHD on VirtualBox