Guidance: Crawling a site with a host header
When crawling content in a site collection that uses a host header, it will fail with the following error in your crawl logs:
“Access is denied. Check that the Default Content Access Account has access to this
content, or add a crawl rule to crawl this content.
(The item was deleted because it was either not found or the crawler was denied
access to it.)”
This is because Windows Server 2003 SP1 includes a loopback check security feature that is designed to help prevent reflection attacks on your computer. Therefore, authentication
fails if the FQDN or the custom host header that you use does not match the local
computer name.
As well, the installation of the following patches on a Windows 2003 Service Pack 1 machine will trigger the Access Denied
1. KB957097 https://support.microsoft.com/kb/957097/
2. .Net 3.5 SP1 https://support.microsoft.com/kb/959209
On all Web Front Ends that are crawled:
1. Click Start, click Run, type regedit, and then click OK
2. In Registry Editor, locate and then click the following registry key:
3. HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa
Right-click Lsa, point to New, and then click DWORD Value.
4. Type DisableLoopbackCheck, and then press ENTER.
5. Right-click DisableLoopbackCheck, and then click Modify.
6. In the Value data box, type 1, and then click OK
7. Quit Registry Editor, and then restart your computer.
Resolution steps are from:
896861 You receive error 401.1 when you browse a Web site that uses Integrated
Authentication and is hosted on IIS 5.1 or IIS 6
https://support.microsoft.com/default.aspx?scid=kb;EN-US;896861
You can run System.Net tracing (See KB947285) and in the trace you will find that
you are getting 401.1 Invalid Credentials for the web services
../_vti_bin/sitedata.asmx
947285 How to use System.NET tracing to troubleshoot content deployment issues and
search issues in Windows SharePoint Services 3.0 and in SharePoint Server 2007
https://support.microsoft.com/default.aspx?scid=kb;EN-US;947285