Crawl logs showing Site URL crawled twice successfully. Search result showing duplicate results

Umr 511 Reputation points

Hello ,

when user search for the site name , it shows up twice in the search results. Both the results are for the same SharePoint site home page.
When checked in the crawl logs , i see 2 successful crawled results for the same URL with 2 different ID . If i remove one of them using "Remove item from the index" search results shows one item in the search results but at the next crawl it shows 2 items in the crawl logs and 2 in search results again.
enabling "trim duplicate search results" also does not help. My question is why is it crawling same URL twice in crawl logs with different ID , same URL and same content source as shown in the image below? Its happening on the home page of the sub sites.

Any advice where to further look for this ?
Thanks in advance.


SharePoint Server
SharePoint Server
A family of Microsoft on-premises document management and storage systems.
1,618 questions
SharePoint Server Management
SharePoint Server Management
SharePoint Server: A family of Microsoft on-premises document management and storage systems.Management: The act or process of organizing, handling, directing or controlling something.
2,411 questions
No comments
{count} votes

3 answers

Sort by: Most helpful
  1. Echo Du_MSFT 16,911 Reputation points

    Hello @Umr ,

    If Item ID is different, it means that these are two different items. Although the two items are the same and point to the same page, they will be treated as two items in the crawling.


    If you want to exclude duplicate search results, please do the following steps:

    1.Go to the results.aspx and click "Edit page"

    2.Select Search Results web part and click on "Edit Web Part"


    3.On the Search Results web part, click "Change query" button


    4.On the Build Your Query page, go to the "SETTINGS" page and select "Remove duplicates" checkbox.


    5.Apply it and Save page


    Echo Du


    If an Answer is helpful, please click "Accept Answer" and upvote it.
    Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.

  2. Umr 511 Reputation points

    Hi Echo,

    Thanks for testing and sharing he screenshots. In my case like mentioned earlier "Remove Duplicate" is checked. it is checked by default btw, but I am still seeing duplicate results. the only time I don't see duplicate result when hard remove from index from crawl logs.

    I also tried using api with "trimduplicates=true" and also used search query tool that allow you to see the additional properties of the results. In each case I am seeing link for the same page twice.

    Any other tips, please share.

  3. Umr 511 Reputation points

    Yeah, resetting index might fix lots of issues but in this case thousands of sites home page is dependent on index as they are build on using search web parts. So cant take all of them down for hours. cuz recrawl in this environment can easily take 48 hours to build again.
    Thanks for your suggestions thou. Any other advice will be helpful.