Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Original KB number: 3019711
Symptoms
Consider the following scenario:
You use the Microsoft SharePoint Server 2013 or SharePoint Server 2010 search engine to crawl various sites.
For those sites, you want to use directives in the Robots.txt file to define the paths that the search engine can crawl.
You set the following directive for the default user-agent of the crawler:
User-Agent: Mozilla/4.0 (compatible; MSIE 4.01; Windows NT; MS Search 6.0 Robot)
In this scenario, the SharePoint Server crawler doesn't apply the directive.
Cause
This issue occurs because the SharePoint Server crawl engine doesn't recognize its default user-agent in the directive.
Resolution
To resolve this issue, use the following directive in the Robots.txt file:
User-Agent: MS Search 6.0 Robot