Is it possible to run an indexer in Azure search as and when there is a change in the blob container (like upload blobs, delete blobs, update blobs) ?

Question

Is it possible to run an indexer in Azure search as and when there is a change in the blob container (like upload blobs, delete blobs, update blobs) ?

Sachitha Chithralekha 20 Microsoft Employee

In azure cognitive search, I have imported data from azure blob storage and indexed it using indexer. Now, in future, as changes are made to the blob container (like adding new documents, deleting existing one) ,is there a way to run indexer automatically only when these changes are made and not set a fixed interval for it to run?

If there is, please provide the steps to achieve it.

SnehaAgrawal-MSFT 22,706 Reputation points Moderator

2023-06-12T10:55:49.84+00:00

@Anonymous

Apologies for late response here!

Once you set up a search index, it serves as a structured catalog of your documents, making it easier to search and retrieve information from them. When you initially create the search index, it includes all the documents available at that time.

However, as time goes on, you might only want to update the search index with new or modified documents, rather than re-indexing all the existing ones.

In the case of documents stored in Azure Storage, the indexing process automatically detects changes. It does this by keeping track of the last update time using the built-in timestamps associated with the objects and files in Azure Storage.

So, whenever a document stored in Azure Storage is modified or a new document is added, the indexer recognizes the change and updates the search index accordingly. This way, the search index remains up to date with the latest version of your documents.

However, there is a slight complication when it comes to deleted documents. The indexer doesn't actively track or detect when a document is deleted from the data sources, such as Azure Storage. This means that if you delete a document from Azure Storage, it will still remain in the search index unless you take further steps.

To avoid having search documents that reference deleted documents, you can implement a "soft delete" strategy. This strategy involves deleting the corresponding search documents first. By removing these search documents, you ensure that they no longer appear in search results. Then, as a second step, you can physically delete the documents from Azure Storage.

By following this "soft delete" strategy, you can keep your search index synchronized with your data sources, ensuring that it only includes existing and relevant documents.

See this detailed document here might be helpful- Change and delete detection using indexers for Azure Storage in Azure Cognitive Search

Please let us know if further query or issue remains.
Sachitha Chithralekha 20 Reputation points Microsoft Employee

2023-06-12T17:27:41.57+00:00

So are you saying that the indexer automatically runs as soon as new blobs are added or updated?
But in my case it does not, I have to manually run the indexer after uploading new docs into the blob container.
Could you mention the steps to automate it ?
SnehaAgrawal-MSFT 22,706 Reputation points Moderator

2023-06-19T04:23:29.4733333+00:00

@Anonymous Just checking if you have chance to see recent response.

Please let us know if further query or issue remains.

1 answer

Your answer

Sachitha Chithralekha 20 Reputation points Microsoft Employee

2023-06-12T17:27:41.57+00:00

So are you saying that the indexer automatically runs as soon as new blobs are added or updated?
But in my case it does not, I have to manually run the indexer after uploading new docs into the blob container.
Could you mention the steps to automate it ?
SnehaAgrawal-MSFT 22,706 Reputation points Moderator

2023-06-19T04:23:29.4733333+00:00

@Anonymous Just checking if you have chance to see recent response.

Please let us know if further query or issue remains.

Answer 1

@Anonymous

Thanks for response! After further more research and discussions with PG it was clear that the built-in indexers are made to either run on schedule or if on demand.

If you need triggered indexing, you would have to build it using Azure logic app Refer - Azure Logic App run trigger when Blob is added or modified - Stack Overflow)

OR Azure Blob storage trigger for Azure Functions | Microsoft Learn

And then using the Push API (Data import and data ingestion - Azure Cognitive Search | Microsoft Learn) so you build your own indexer.

Hope this helps.

Share via

Is it possible to run an indexer in Azure search as and when there is a change in the blob container (like upload blobs, delete blobs, update blobs) ?

1 answer

Your answer