Pagination not working for "page search results" for large sites

Question

Hi.

I use the endpoint https://graph.microsoft.com/v1.0/search/query and paging to get files from a sharepoint folder. I can see from the output that there are 3387 files in the folder. (I have the issue for all large sites)

1. Firstly, I tried to iterate through by incrementing "from" and checking if "moreResultsAvailable" == False (which i guess it should be at, for example, from:3000, and size:500), but it still shows "moreResultsAvailable": true.
2. Then I tried using the "total" and iterate until from + size = total. However I do not get all the files in my site and a lot are duplicates.

It seems like every time I run the endpoint the output is scrambled and follows no consistent pattern when retreiving with "from" and "size", such that there is no way to use the pagination. (I also tried using sorting in the query without any luck).

Btw, the pagination works for smaller sites, but for large sites i get the issue. Any ideas?

My request body is:

{

"requests": [

{

  "entityTypes": [

    "driveItem"

  ],

  "query": {

    "queryString": "Path:https://{domain}/sites/{sitename}/Shared%20Documents AND (filetype:docx OR filetype:pdf)"

  },

  "from": 0,

  "size": 500,

  "trimDuplicates": true,

}

]

}

Answer 1

Hello Victoria.G

Thank you for reaching out to Microsoft Support!!

Pagination Behavior:

• The from and size parameters control pagination, but for large datasets, the API may not always return results in a predictable order.
• Instead of relying solely on from and size, Microsoft recommends using the @odata.nextLink property for pagination

Best Practices for Pagination:

• Use @odata.nextLink to fetch the next set of results dynamically.
• Avoid manually incrementing from as it may not work consistently for large datasets

Reference link: https://learn.microsoft.com/en-us/graph/paging?tabs=http

Handling Duplicate Results

1. Using trimDuplicates:
   • The trimDuplicates property is supported for SharePoint files, but it may not always remove all duplicates.
   • If duplicates persist, consider filtering them out programmatically by tracking unique file IDs.
2. Sorting Issues:
   • Sorting within the query may not always work as expected for large datasets.
   • Try sorting results client-side after retrieval.

Reference link: https://learn.microsoft.com/en-us/answers/questions/590157/whats-the-best-way-to-paginate-large-data-sets-usi

Recommended Approach

1. Modify Your Request:
   • Instead of relying on from and size, use @odata.nextLink for pagination.
   • Ensure trimDuplicates is set to true, but also implement a client-side check for unique file IDs.
2. Check API Limits:
   • Microsoft Graph has limits on the number of results returned per request. If you're hitting these limits, consider breaking queries into smaller chunks

Reference link: https://learn.microsoft.com/en-us/graph/api/resources/search-api-overview?view=graph-rest-1.0

Hope this helps.

If the answer is helpful, please click Accept Answer and kindly upvote it. If you have any further questions about this answer, please click Comment.