How to solve result limit from File Share Indexer in Azure Search AI using Python ?

Question

How to solve result limit from File Share Indexer in Azure Search AI using Python ?

Lyncheese 105

I have a File Share Indexer which made using this tutorial
https://learn.microsoft.com/en-us/azure/search/search-file-storage-integration

My File Share has pdf files and I'm using Python SDK for Azure.

The problem now is,
I need to fetch all of the result at once, and the documents there are exceeding 100k.
Since the result is only limited to 100k using skip parameter.
What way I can use to get all of them ?

I cannot use facet or filter.
Facet cannot be done because the field data is added and filled by the indexer.
Filter also not an option since nothing can be filtered. The pdf file name are random.

I searched for continuation token in Python Azure SDK, but it is not clear enough and I have not find the suitable samples yet. Instead of returning continuation token, the search method throw exception.

Please give me more information or alternative solution for this.

Thank you.

Grmacjon-MSFT 18,896 Reputation points

2024-02-01T21:38:28.1+00:00
Hello @Lyncheese

I understand that you are trying to fetch all the results from your File Share Indexer, but the documents there are exceeding 100k and the result is limited to 100k using the skip parameter. You have mentioned that you cannot use facet or filter since the field data is added and filled by the indexer and the pdf file names are random. You have also searched for continuation token in Python Azure SDK, but it is not clear enough and you have not found any suitable samples yet. One way to fetch all the results at once is to use server-side paging. Azure Search uses server-side paging to prevent queries from retrieving too many documents at once. The default page size is 50, while the maximum page size is 1000 .

This means that by default, Search Documents returns at most 50 results if you don’t specify $top. If there are more than 50 results, the response includes information to retrieve the next page of at most 50 results (see @odata.nextLink and @search.nextPageParameters in the examples below). Similarly, if you specify a value greater than 1000 for $top and there are more than 1000 results, only the first 1000 results are returned, along with information to retrieve the next page of at most 1000 results Here’s an example of how you can use the search method in the Python SDK to fetch all the results using server-side paging:

from azure.search.documents import SearchClient from azure.core.credentials import AzureKeyCredential # Create a SearchClient object search_client = SearchClient(endpoint="your-search-service-endpoint", index_name="your-index-name", credential=AzureKeyCredential("your-search-service-api-key")) # Set the page size to 1000 page_size = 1000 # Initialize the search results search_results = search_client.search(search_text="*", top=page_size) # Get the total count of results total_count = search_results.get_count() # Initialize the results list results = [] # Append the first page of results to the list results.extend(search_results.get_results()) # While there are more results to fetch while len(results) < total_count: # Get the next page of results search_results = search_client.search(search_text="*", top=page_size, skip=len(results)) # Append the next page of results to the list results.extend(search_results.get_results()) # Print the results for result in results: print(result)

In the above example, we set the page size to 1000 and initialize the search results. We then get the total count of results and initialize the results list. We append the first page of results to the list and while there are more results to fetch, we get the next page of results, append it to the list and print the results
Please let us know if you have further questions further questions. Best, -Grace
Lyncheese 105 Reputation points

2024-02-02T00:42:43.4633333+00:00
@Grmacjon-MSFT

I'm not sure you are fully understand my problem.

You mentioned it before
"I understand that you are trying to fetch all the results from your File Share Indexer, but the documents there are exceeding 100k and the result is limited to 100k using the skip parameter."

But using your sample

# While there are more results to fetch while len(results) < total_count: # Get the next page of results search_results = search_client.search(search_text="*",top=page_size,skip=len(results)) # Append the next page of results to the list results.extend(search_results.get_results())

Your code eventually will use, skip parameter with value exceeding 100k.

Facet cannot be use because the data field are produced by File Share Indexer.
There is no data, I can use as facet here.
Grmacjon-MSFT 18,896 Reputation points

2024-02-08T03:04:59.22+00:00

@Lyncheese you're correct, thanks for the clarification. We are checking internally with the engineering team to get an alternative solution for your scenario and will get back to you when we hear back from them. -Grace

Accepted answer

0 additional answers

Your answer

Lyncheese 105 Reputation points

2024-02-02T00:42:43.4633333+00:00

@Grmacjon-MSFT

I'm not sure you are fully understand my problem.

You mentioned it before
"I understand that you are trying to fetch all the results from your File Share Indexer, but the documents there are exceeding 100k and the result is limited to 100k using the skip parameter."

But using your sample

# While there are more results to fetch while len(results) < total_count: # Get the next page of results search_results = search_client.search(search_text="*",top=page_size,skip=len(results)) # Append the next page of results to the list results.extend(search_results.get_results())

Your code eventually will use, skip parameter with value exceeding 100k.

Facet cannot be use because the data field are produced by File Share Indexer.
There is no data, I can use as facet here.
Grmacjon-MSFT 18,896 Reputation points

2024-02-08T03:04:59.22+00:00

@Lyncheese you're correct, thanks for the clarification. We are checking internally with the engineering team to get an alternative solution for your scenario and will get back to you when we hear back from them. -Grace

Answer 1

Grmacjon-MSFT 18,896

Hello @Lyncheese thanks for your patience. we heard from the engineering team. They pointed out that without a filterable or sortable field there's no way to retrieve more than 100k results. Please review the following this documentation: How to work with search results - Azure AI Search | Microsoft Learn

If you have further questions, please let us know.

-Grace

Share via

How to solve result limit from File Share Indexer in Azure Search AI using Python ?

0 additional answers

Your answer