How to add Indexer for 70 million data in Azure cognitive search?

Nagaramya Konudula 1 Reputation point Microsoft Employee
2022-11-25T07:24:28.797+00:00

our DB table contains 70 million data, and Indexer is timing out after 24hrs and getting stopped at 50 million, can you please suggest in which way I can retrieve entire data within less than 24 hrs.

Azure AI Search
Azure AI Search
An Azure search service with built-in artificial intelligence capabilities that enrich information to help identify and explore relevant content at scale.
701 questions
0 comments

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. VenkateshDodda-MSFT 18,176 Reputation points Microsoft Employee
    2022-11-25T18:00:55.157+00:00

    @Nagaramya Konudula Thank you for reaching out to Microsoft Q&A. Apologize for the inconvenience caused on this.

    You can use the index scheduling mechanism for processing large data sets. Indexer processing operates within a 24-hour window. If an indexing job cannot complete within the maximum time allowed, try running it on a schedule. The scheduler keeps track of indexing status. If a scheduled indexing job is interrupted for any reason, the indexer can pick up where it last left off at the next scheduled run.

    You can refer to this documentation on Index large data set in cognitive search and also Indexers limits.

    Feel free to reach back to me if you have any further questions on this.