How to add Indexer for 70 million data in Azure cognitive search?

Question

our DB table contains 70 million data, and Indexer is timing out after 24hrs and getting stopped at 50 million, can you please suggest in which way I can retrieve entire data within less than 24 hrs.

Answer 1

@Nagaramya Konudula Thank you for reaching out to Microsoft Q&A. Apologize for the inconvenience caused on this.

You can use the index scheduling mechanism for processing large data sets. Indexer processing operates within a 24-hour window. If an indexing job cannot complete within the maximum time allowed, try running it on a schedule. The scheduler keeps track of indexing status. If a scheduled indexing job is interrupted for any reason, the indexer can pick up where it last left off at the next scheduled run.

You can refer to this documentation on Index large data set in cognitive search and also Indexers limits.

Feel free to reach back to me if you have any further questions on this.