Continuous crawl

In SharePoint 2013, you can configure crawl schedules so that crawls are performed continuously. 

What are continuous crawls?

 Continuous crawls help to keep the search index and search results as fresh as possible. This is a new feature in SharePoint 2013.  Because of changes in how the index is created and stored, a document can appear in the index within seconds of going through the content processing component – you no longer have to wait for long index merges until it shows in results.  It also means you can get the latest changes even while a full crawl is starting, so you can see results before full crawl completes.

 Note: It only works with SharePoint sites as content sources.

 How do they work?

 In SharePoint 2010 we have Full Crawls and Incremental Crawls. Both crawls you can schedule and especially the Incremental Crawl is scheduled often. With Incremental Crawl all content changed since last crawl is retrieved and indexed. But the results may not be what you might expect: the index is not as accurate as you hoped it to be.

Let’s examine the below scenario:

You have scheduled to perform an Incremental Crawl every 15 minutes.

 In the first schedule (A) the crawler retrieves a content set and can handle that set within 15 minutes. No problem here. The next schedule (B) is executed and this time the crawler retrieves a bigger content set, but now it cannot be handled within 15 minutes, but it takes 20 minutes. Now a problem arises. Because only one crawler at a time can be executed, the third schedule will be killed immediately. We have to wait for the next window that the schedule takes place. This will be (C) at 45 minutes. So, in this case we had 30 minutes between 2 Incremental Crawls instead of the scheduled 15 minutes. So, the bigger the content set is to handle by the crawler the more risk there is to miss scheduled crawling windows. This results in a lesser up to date search index.

SharePoint 2013

With Continuous crawling you don’t have to schedule anymore. Crawlers are running in parallel and the crawler gets changes from the SharePoint sites every 15 minutes. This is the default setting and it can be changed using PowerShell. So, when a crawl is executed and finished, it continues to crawl immediately in spite of the length. In our previous situation when a crawl is taking more than 15 minutes to handle the content set, another crawler is started.

 It is even possible that a second crawl is started only a few minutes after the first crawl. This way your newly added or changed data is almost immediately available through search.

 By default, a new crawl is run every 15 minutes, but the SharePoint administrator can change this interval using the PowerShell cmdlet Set-SPEnterpriseSearchCrawlContentSource

 When should I use them?

 You should use this feature for your search driven applications. Every time when it matters that new results appear right after they have been created.  

Enabling continuous crawl has the following advantages:

The search results are very fresh, because the SharePoint content is crawled frequently to keep the search index up to date.

 Ex: in social computing whenever there is new feed it can appear fast in the search

The search administrator does not have to monitor changing or seasonal demands for content freshness. Continuous crawls automatically adapt as necessary to the change rate of the SharePoint content.

This new feature is not so important, when you just use a document library to store and exchange documents. In this case the ‘old’ incremental crawl should be enough.

Note: This new feature is not so important, when you just use a document library to store and exchange documents. In this case the ‘old’ incremental crawl should be enough.

 Now, a few things to bear in mind. Continuous crawls can only be enabled for content sources based on SharePoint sites. 

Also, it can be resource intensive for your server running all the crawlers, We need to enable then when we have adequate hardware which can take the load on to the SharePoint servers.

Summary

With continuous crawling, data is almost immediately available for search. No more issues with incremental schedules and big data which can cause missed schedule windows. Using continuous crawls and the Content by Search web part, for example, gives you lots of opportunities to build awesome search-driven solutions.

Host distribution rules removed

In SharePoint Server 2010, Search application administrators could remove items from the index through Central Administration. In SharePoint 2013, you can remove items from the index only by using the crawl logs.

Discovering structure and entities in unstructured content

You can configure the crawler to look for "entities" in unstructured content, such as in the body text or the title of a document.  These entities can be words or phrases, such as product names.  To specify which entities to look for in the content, you can create and deploy your own dictionaries. For locations, you can use the pre-populated location extraction dictionary that SharePoint 2013 provides.

You can store these entities in your index as separate managed properties, and later use those properties for example in search refiners to help users filter their search results.

To improve search relevance, the document parsing functionality in the content processing component analyzes the structure of documents in addition to the contents. Document parsers both extract useful metadata and remove redundant information. For example, parsers extract headings and subheadings from Word documents, as well as titles, dates and authors from within slides in PowerPoint presentations. For HTML content, redundant generic information such as menus, headers and footers are classified as such and removed from document summaries in the search results.

Health Monitoring Reports

In SharePoint 2013, you access health monitoring reports (previously called Search Administration Reports) on the left navigation pane under Crawling or Queries & Results for individual Search service applications. Enhancements to the reports make them easier to interpret.

Query health monitoring reports include:

• Query Latency Report
• Query Latency Report for Index Engine
• Query Latency Report for SharePoint Default IMS Flow
• Query Latency Report for Federation
• Query Latency Report for Local SharePoint Search Results
• Query Latency Report for People Search Results

Crawl health monitoring reports include:

• Crawl Summary Report
• Crawl Rate Report
• Crawl Processing Per Document Report
• Crawl Queue Load Report

New search architecture

Search uses a new, component based architecture that provides maximum flexibility in defining the topology to support search requirements for performance, availability and fault-tolerance.

The crawl component crawls content sources to collect crawled properties and metadata from crawled items. It sends this information to the content processing component.

The content processing component transforms the crawled items so that they can be included in the search index. The component also maps crawled properties to managed properties. In addition, the content processing component interacts with the analytics processing component.

The analytics processing component analyzes the crawled items and how users interact with their search results. The information is used to improve the search relevance, and to create search reports and recommendations.

The index component receives the processed items from the content processing component and writes them to the search index. The component also handles incoming queries, retrieves information from the search index and sends the results back to the query processing component.

The query processing component analyzes incoming queries to help optimize precision, recall (which items are returned in the results) and ranking (the order of those items). The query is then sent to the index component which returns a set of search results for the query. The results can then be further processed before they are presented to the user as the search results for his/her query.

The search administration component runs the required search processes and adds and initializes new instances of search components.

The crawl database contains detailed tracking and historical information about crawled items such as documents and links. The database holds information such as the last crawl time, the last crawl ID and the type of update during the last crawl (add, update, delete).

The link database stores the information extracted by the content processing component and click-through information.

The analytics reporting database stores the results of search and usage analysis, such as the number of times an item has been viewed.

The search administration database stores settings for the Search service application, such as the topology, crawl rules, query rules and the mappings between crawled and managed properties.

Multi-tenant hosting

SharePoint 2013, the search system supports multi-tenant hosting.

1. Plan:

Planning for authentication, performance, capacity and future growth, either at the initial design phase or after deployment is one of the most important exercises of building and maintaining SharePoint 2010.  SharePoint is deceptively complex and lack of planning both in the initial stages as well as the long term can cause pain points within service applications, storage and server resources that start small and become major headaches in the future.

         Resources:

• Initial deployment administrative and service accounts (SharePoint Server 2010)
• Hardware and software requirements (SharePoint Server 2010)
• Plan authentication methods (SharePoint Server 2010)
• Plan browser support (SharePoint Server 2010)
• Performance and capacity management (SharePoint Server 2010)
• SharePoint Server2010 capacity management: Software boundaries and limits

2. Use site quota or the Quota template for Site Collections or Mysites

This is the place which a SharePoint administrator has to keep an eye. The Site collections Can Grow Big and might Cross 100 GB Max. To Maintain proper working environment and to make easy backups it strongly recommended to have limitation on the Site growth.  SharePoint isn’t capable of self-managing the size of its database and with users constantly adding new content the data store will continue to the grow if you don’t impose defined limits. Quotas will really help in maintain the health of the environment and also keep the databases in Supported stated. The site quota template varies according to the environments.

Resources:

• Plan quota management (SharePoint Server 2010)
• Create, edit, and delete quota templates (SharePoint Server 2010)
• Manage site collection storage limits (SharePoint Server 2010)

3.  System Health and Availability of Resources:

When we are looking at SharePoint 2010, Its not he Site collections or the My sites which will consume and store content. There are services which will consume the Storage and resources. A good Administrator will also look into other aspects as when looking at other overheads like Enterprise search, User Profiles, Web Analytics, Health monitoring which will impact performance and latency. The administrator should plan the requirements of the environment.

Resources:

• Monitoring overview (SharePoint Foundation 2010)
• Configuring monitoring (SharePoint Foundation 2010)
• Viewing reports and logs (SharePoint Foundation 2010)
• Solving problems and troubleshooting (SharePoint Foundation 2010)
• Monitor health and performance of a virtual environment (SharePoint Foundation 2010)
• Health Monitoring for SharePoint 2010
• Monitoring overview (SharePoint Server 2010)

4.  Monitor Recycle bins which consumes space outside of Site collection

There are 2 stages for the recycle bin. The first stage is used for user recoverable content and is stored within the site collection. This consumes space against the site collection quota. User recoverable content has a default limit of 30 days, and then it is moved to the second stage recycle bin. The second stage recycle bin is used by the site collection administrator and/support staff to recover content that may have been deleted and/or expired from the first stage recycle bin. It consumes 50% of SharePoint’s total quota size, so you should enforce a quota or it will continue grow out of control.

Resources:

• Configure Recycle Bin settings (SharePoint Foundation 2010)
• Protecting content by using recycle bins
• Protecting content by using versioning

 5.    Site collections, its necessity and performance

The only type of sites that can be moved between content databases are site collections. We can easily achive the stability by managing site collections and database storage can provide flexibility, but also puts additional boundaries within the environment. Maintaining a site collection structure based on teams, projects and departments will allow individual sites to grow organically. It also imposes some security limitations and gives each site collection a different look and feel. if we understand the requirement of a team or project that can really help in maintaining and managing the site collections.  

Resources:

• Plan sites and site collections (SharePoint Server 2010)
• Sites and site collections overview (SharePoint Server 2010)
• Site navigation overview (SharePoint Server 2010)
• Plan site navigation (SharePoint Server 2010)

6.   SharePoint 2010 and User Profile Service

The User Profile service application stores information about users in a central location. Social computing features use this information to facilitate productive interactions which enable users to collaborate efficiently. In order to provision My Sites, enable social computing features such as social tagging and newsfeeds, and create and distribute profiles across multiple sites and farms, you must enable the User Profile service application contain detailed information about individuals in an organization. A user profile organizes and displays all of the properties related to each user together with social tags, documents and other items related to that user. We need to make sure that Proper accounts are added and correct OUs are selected from the domains. We need to monitor whether the user profiles are being imported properly or not. Assigning the correct organizational units/containers within the Active Directory structure will allow the User Profile Service to only target the required users and groups. This will prevent SharePoint from accessing service and other miscellaneous accounts and ensure the optimal performance of the User Profile Service.

Resources:

• Plan for profile synchronization (SharePoint Server 2010)
• Troubleshoot User Profile Synchronization Service start issues (SharePoint Server 2010)
• Troubleshoot profile synchronization configuration issues (SharePoint Server 2010)

7.       External search Vs SharePoint OOB and FAST Search for SharePoint 2010

SharePoint Can index the external sites and internet and provide the query results which is beneficial for an organization to find content in easy and effect way. But it has huge impact on the Performance. if you are using OOB SharePoint Search then the total size varies from 15% to 25% of storage for index file. If you are using FAST Search for SharePoint you can add 10% more for the size of the SharePoint OOB index. These sizes are inductive. But the Administrator has to monitor this carefully and Create content sources and add the content to the environment. But it is always recommended to have reserved space for these content sources when planning the Search.

Resources:

• Plan search topology
• Performance and capacity test results
• FAST Search Server 2010 for SharePoint Capacity Planning
• Search scopes in FAST Search for SharePoint part 1
• Monitor FAST Search Server 2010
• FAST Search Server 2010 for SharePoint Capacity Planning
• Troubleshooting Best Practices for FAST Search Server for SharePoint 2010 Installation
• Troubleshooting: FAST Admin DB

 8.        Understand SharePoint limitations and planning long term goals.

SharePoint 2010 is extremely Scalable and flexible for the deployment. Microsoft has provided some limitations and Boundaries for SharePoint. We need to keep in mind the scenarios and best practices while deploying SharePoint in the Environment. First we need to do lab testing but this will not give actual results. The good starting point is to follow the guidelines while deploying the Customizations, Solutions. This will ensure the System stay in tact and perform as expected.

Resources:

• Performance and capacity management (SharePoint Server 2010)
• SharePoint Server 2010 capacity management: Software boundaries and limits
• Plan search topology
• Plan sites and site collections (SharePoint Server 2010)
• Developer Best Practices Resource Center | SharePoint 2010
• Inside SharePoint: SharePoint 2010
• Governance overview (SharePoint Server 2010) 

9.  Storage for planning is crucial

Data Storage is one of the Key and important thing in SharePoint performance. Poor storage for content databases and Service application databases will really cause performance hit. Understanding the Storage requirements for SharePoint is key. If the SharePoint access the Databases and the file stores properly we can expect the high Performance level over the period of time. The SharePoint content has less impact on the overall performance but it is good to start from here.

Resources:

• Enterprise content storage planning (SharePoint Server 2010)
• Document library planning (SharePoint Server 2010)
• Plan for disaster recovery (SharePoint Server 2010)
• Storage and SQL Server capacity planning and configuration (SharePoint Server 2010)

10.   SQL Server Deployment, the heart and brain of SharePoint.

If you are planning for robust and reliable SharePoint Environment, Plan the SQL server accordingly. A Proper SQL server can Mean the difference between a high performance or a constantly failing systems. Make sure that SQL Deployment is met withe best practices and meets the resources. This includes Memory, CPU and disk space. SQL configuration is tuned for optimum performance in SharePoint Deployment in a Large SQL Cluster Sharing resources. Make sure SharePoint databases are set to simple recovery model and correct collation. Make sure we have the adequate disk space for the log files. Ensure the databases are being taken backup.

 Resources:

• SQL Server 2008 R2 and SharePoint 2010 Products: Better Together (white paper) (SharePoint Server 2010)
• SharePoint 2010 Products, SQL Server, Storage and Databases
• SQL Server and SharePoint Server 2010 Databases
• Storage and SQL Server capacity planning and configuration (SharePoint Server 2010) 

There is also a significant difference in behavior when Office detects that WebDAV is enabled on the server. If WebDAV is enabled then Microsoft Office also has to determine whether or not to open the document as ReadWrite or ReadOnly.

Typically Office on Windows XP uses WinInet to handle the communication with the server and therefore should handle authentication just like IE does; if the server is configured for Windows authentication then you are prompted to access the server, you can probably expect a prompt to open the document.

The expected default behavior is that if the site can be identified as a local intranet site then Windows logon credentials will be automatically sent to the server. A site is typically determined to be identified as local when there is no '.' in the address.  When a '.' exists in the address then some help may be needed in order to identify the site as local intranet; if the site does not show "Local intranet" in the lower, right of the status bar then you may need to check [Sites] for Local intranet on the Security tab of IE's Internet options. If the 'Automatically detect intranet network' is checked and/or the appropriate sub-options are checked then you may need to specifically use the [Advanced] button to add the site to the Local intranet.

It is possible to change the security settings for the other zones to allow User Authentication\Logon to Automatic logon with current user name and password but it is not recommended as that could result in sending credentials in a scenario when they are not secure (such as to a site using Basic authentication without SSL).

Prior to Windows Vista, Microsoft Office would install the Web Extender Client to allow OLE DB to handle the WebDAV protocol because the native Web Client was not robust enough. Since the Vista Web client was enhanced  Office 2007 on Vista will try to use the native WebClient (which uses WinHTTP instead of WinInet) to handle the communication. WinHTTP does not have the concept of zones and requires the configuration of a registry key in order to properly pass credentials automatically for sites with a '.' in the name. This is referenced in KB 943280 (https://support.microsoft.com/?id=943280). The 'Office 2007 prompting on Vista' scenario is also discussed in more detail in the following blog: https://blogs.msdn.com/sharepoint/archive/2007/10/19/known-issue-office-2007-on-windows-vista-prompts-for-user-credentials-when-opening-documents-in-a-sharepoint-2007-site.aspx

We need to keep in mind some of the Prerequisites for this

1. FARM Path level should be Same

2. Soluitons Deployed for the Web applications or sites should be avilable in the Target FARM

3. The account with which we are doing this TASK should be FARM account or Service account which has DBO rights on SQL server

4. Check the Database is not read only.

Server Name: Contoso
Source Site Collection URL: https://contoso.com
Source Site Owner: Contoso\Administrator.
Goto the Command prompt and Change the Directory to below
%Program Files%\Common Files\Microsoft Shared\Web Server Extensions\12\Bin\
Take backup using following command:

 stsadm.exe -o backup
     -url <url>
     -filename <filename>
     [-overwrite]
     [-nositelock]

STSADM.EXE –o backup –URL https://Contoso.com -filename “contoso.bak”

Now we need to restore this to another FARM:
Login to the Server with Service account which has Admin rights on the Server and DBO Rights on SQL server so that we can write to the Database.

On the Destination Server:
Server Name: Tailspintoys
Target Site Collection URL: https://Tailspintoys.com
Service account: Tailspintoys/Administrator
Create New Web Application from Central Administration
Now restore using STSADM command
YOu can use the below Syntax

stsadm.exe -o restore
    -url <url>
    -filename <filename>
    [-hostheaderwebapplicationurl <web application url>]
    [-overwrite]

STSADM.EXE –o restore –URL https://Tailspintoys.com -filename “contoso.bak”

If you already have a site collection with Blank Template on this web applicaton you can use the -Overwrite switch

Change the site collection owner of this newly restored site

1. Access is denied. Verify that either the Default Content Access Account has access to this repository, or add a crawl rule to crawl this repository.

2. If the repository being crawled is a SharePoint repository, verify that the account you are using has "Full Read" permissions on the SharePoint
Web Application being crawled.

3. The item was deleted because it was either not found or the crawler was denied access to it.

Disable the Loop Back Check on the Windows Server, you can find the steps in this KB article
as well https://support.microsoft.com/kb/896861

You Can use the Below steps to resolve the issue

1. Click Start, click Run, type regedit, and then click OK.

2. In Registry Editor, locate and then click the following registry key:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa

3. Right-click Lsa, point to New, and then click DWORD Value.

4. Type DisableLoopbackCheck, and then press ENTER.

5. Right-click DisableLoopbackCheck, and then click Modify.

6. In the Value data box, type 1, and then click OK.

7. Quit Registry Editor, and then restart your computer.

1. Stop FAST Search Server 2010 for SharePoint on all servers in the farm, including the monitoring service.
2. On the administration server:
1. On the Start menu, click All Programs.
2. Click Microsoft FAST Search Server 2010 for SharePoint.
3. Right-click SharePoint 2010 Management Shell, and select Run as administrator.
4. At the command prompt, browse to installer\scripts under the installation folder.
5. Type the following command:

                             .\ReplaceDefaultCertificate.ps1 -generateNewCertificate $true

                         Enter a password for the certificate.

3. Start FAST Search Server 2010 for SharePoint on the administration server.
4. On each non-administration server:
1. On the Start menu, click All Programs.
2. Click Microsoft FAST Search Server 2010 for SharePoint.
3. Right-click SharePoint 2010 Management Shell, and select Run as administrator.
4. At the command prompt, browse to installer\scripts under the installation folder.
5. Type the following command:

                     .\ReplaceDefaultCertificate.ps1 -generateNewCertificate $true

                    Enter the password that you defined for the certificate on the administration server.

          5.  Start FAST Search Server 2010 for SharePoint on all non-administration servers.

The post-setup configuration wizard creates a new database when you configure a single server or administration server. However, you can configure FAST Search Server 2010 for SharePoint to use another existing
database after the configuration wizard has finished.

You can find the FAST settings for the database in <FASTSearchFolder>\etc\hostconfiguration.xml, where <FASTSearchFolder> is the path of the folder where you have installed FAST Search Server 2010

Example: <Drive>\FASTSEARCH\

Below are the Steps to Configure or Create New Database

Make Sure you log into the Machine with Admin rights on FAST box as well as SQL server

            EX: Domain Account which can access FAST, SharePoint and SQL database

Create Empty Database on SQL server

1. Open PowerShell Right click Microsoft FAST Search Server 2010 for SharePoint shell and select Run
   as administrator
2. At the Windows PowerShell command prompt, type the following command

Install-FASTSearchAdminDatabase -DbName <DatabaseName> -dbserver <ConnectionString>

Here <Database name> and <Database String> are replaced with Appropriate values

Example: Database string is in the format <ServerName>\<DBInstanceName>

                   Database name: Unique Name of the Database

On the FAST Search Server 2010 for SharePoint administration server, do the following

1. Stop the FAST Search Sever
2. Update the database server and database name in <FASTSearchFolder>\etc\hostconfiguration.xml
3. Now open PowerShell, Right click Microsoft FAST Search Server 2010 for SharePoint shell and select Run as administrator.
4. Run the CMDLET Set-FASTSearchConfiguration
5. Start the FAST search for SharePoint

Note: If you are using any other connectors like Lotus connector update the Database name and Database server in the <FASTSearchFolder>\etc\dbconfig.xml

1. The incremental or Full Crawls will not start when they are scheduled in SharePoint 2007

2. You alwasy need to start them manually

3. Some times when you browse to browse Search Settings.aspx page you get 403 Forbidden Error

Error Message/Cause:

Most of the time we dont see any error messages, We only face issue that Schedule Crawls will not start automatically

Sometime we see that Folder %WinDir%\Tasks is not added with SharePoint Local group WSS_WPG Where Indexer is Running

Resolution/WorkAround:

1. Login to the MOSS Indexer machine with account which has Administrative previlages

2. Open Command prompt and run the command attrib –s %windir%\tasks.

%Windir% is the Default installation of Windows

3. Open Windows Explorer and go to the %Windir%\Tasks folder And open the properties of the Tasks folder

4. Click the Security tab and Check whether we have WSS_WPG, If it is not there Add the Group with Full Read and Write permissions

5. Now Open IIS Manager and restrat IIS or at the command prompt Type IISRESET so that the Chages will take place.

6. From the command prompt Run the command attrib +s %windir%\tasks so that the view goes back to default

7. Do the above Steps on all the SharePoint servers in the Farm