Crawling Large Lists in SharePoint 2007
There is a lot of information out there on working with large lists in SharePoint 2007. There is some good info here and here and Steve Peschka has a great whitepaper on working with large lists in SharePoint 2007 that has been out for a while now. One thing that I haven’t seen a lot of information on is crawling large lists. Steve’s whitepaper has a quick reference in his whitepaper saying “If the indexer is timing out when crawling large lists, you can increase the time-out value”. We had been experiencing issues crawling a large pages library of over 4,000 items. We were getting the error “The item may be too large or corrupt.” Increasing the time out seemed to help, but many times the indexer would still not make it through the whole library, we were still looking for a better solution. After working with Microsoft Support we were told to modify three registry values:
Key | Default Value | New Value |
HKLM\SOFTWARE\Microsoft\Office Server\12\Search\Global\GatheringManager\DedicatedFilterProcessMemoryQuota | 104,857,600 | 209,715,200 |
HKLM\SOFTWARE\Microsoft\Office Server\12\Search\Global\GatheringManager\FilterProcessMemoryQuota | 104,857,600 | 209,715,200 |
HKLM\SOFTWARE\Microsoft\Office Server\12\Search\Global\GatheringManager\FolderHighPriority | 50 | 500 |
Modifying these registry values has definitely seemed to do the trick for us so far. This was for x64 machines with a lot of RAM, so this might not be a good idea if you are still on x86 and are RAM constrained. I’d like to hear your results, did this work for you? What are your experiences crawling large pages libraries in SharePoint 2007?
Anonymous
May 04, 2010
This was somehow unmarked as an answer in the M$ forums. These settings seem to fix the problem I was having also. Thanks. Bailed me out of a jam$Anonymous
July 24, 2011
Can you tell me what the dfault values are representing? Are these bytes, Kb or something else? When you say alot of ram how much are you talking about? Thanks!Anonymous
November 04, 2013
After changing these settings, does anything need to be reset? Search Services?