Realtime Index Updates in Azure AI Search

Question

Hi,

We would like to use Azure AI Search in our project, but we're seeing that there is a limitation that may prevent us from using it.

In our application the user can add/update items. After the addition/update, they can perform a search on the items. When they do perform the search, the changes they've made are not picked up in the search. We can see the additions/updates in the database, but not in the index.

After doing some research, I found that lowest time I can set the schedule to for the indexer is 5 minutes. I also have track changes set on my datasources, but that doesn't appear to have any effect that would produce the expected result.

Is there something that I can do to get these additions and updates picked up in real-time in the index i.e. as soon as they are added to the database?

Thanks

Answer 1

Hi @Bernard Brown , since real-time data synchronization isn’t possible with an indexer you may want to explore a ‘push model’ that simultaneously updates data in both Azure AI Search and your external data source. Basically, you can use the Azure Search REST API or SDK to directly push document changes (adds, updates, deletes) from your application to the search index. This method offers the closest to real-time updates that you're looking to accomplish for your use case.

Here are two helpful Azure docs that can help you get started:

• Tutorial: Optimize indexing with the push API
• Index large data sets in Azure AI Search

Best,

Grace

Answer 2

An indexer in Azure AI Search is a crawler that extracts textual data from cloud data sources and populates a search index using field-to-field mappings between source data and the search index. By default, indexers run on a recurring data refresh schedule, which can be as frequent as every five minutes.

While true real-time indexing isn’t possible with an indexer, you can achieve near real-time updates using the following strategies:

• Push Model: Instead of relying solely on the indexer’s pull model, consider pushing updated rows directly to the index when changes occur in your database.
• Custom Script or Code: Implement a custom script or code that detects changes in your database and triggers an update to the search index immediately.
• Azure Functions or Logic Apps: Use Azure Functions or Logic Apps to monitor changes in your data source and invoke the indexer when updates occur.

If your search service supports availability zones, replicas are automatically provisioned in different physical data centers for extra resiliency. Consider adding multiple replicas to handle higher indexing and query workloads. Availability zones ensure that even if one zone experiences an outage, your search service remains available.

While true real-time indexing isn’t achievable with indexers alone, near real-time solutions can significantly reduce the delay between data changes and index updates. Explore the options mentioned above to tailor your approach based on your project’s requirements! https://learn.microsoft.com/en-us/azure/search/search-reliability

Feel free to post if you have any further questions. If this answer helped, please accept the answer :-)