Azure Storage Events - Parquet file content change auto refresh

Question

Hi Guys,

We do have an azure storage event on storage container whenever new file arrive, it triggers the event and move file to different folder, But we are observing, we need an continuous file watch event not on file arrival, but if content of file changes, i don't see any out of the box event trigger available via events, Apart from azure function/logic apps/power apps etc., what options do we have within ADLS2 to accomplish?

Regards,

Navin

Answer 1

@Dondapati, Navin - Thanks for the question and using MS Q&A platform.

Azure Data Lake Storage Gen2 (ADLS Gen2) does not provide an out-of-the-box event trigger for detecting changes in the content of a file. However, you can use Azure Event Grid to monitor changes in the metadata of a file, such as the file size, last modified time, or other properties.

Azure Data Lake Storage (ADLS) Gen2 can now publish events to Azure Event Grid to be processed by subscribers such as WebHooks, Azure Event Hubs, Azure Functions and Logic Apps. With this capability, individual changes to files and directories in ADLS Gen2 can automatically be captured and made available to data engineers for creating rich big data analytics platforms that use event-driven architectures.

The events that will be made available for Azure Data Lake Storage Gen2 are BlobCreated, BlobDeleted, BlobRenamed, DirectoryCreated, DirectoryDeleted and DirectoryRenamed.

Event Grid uses event subscriptions to route event messages to subscribers. This image illustrates the relationship between event publishers, event subscriptions, and event handlers.

To accomplish this, you can create an Event Grid subscription for your ADLS Gen2 storage account and specify the events you want to monitor. When a change occurs in the metadata of a file, such as the file size or last modified time, Event Grid will trigger an event and send a notification to your specified endpoint. You can then use this notification to trigger an action, such as running an Azure Function or Logic App to process the file.

Here are the high-level steps to set up an Event Grid subscription for ADLS Gen2:

1. Create an Azure Function or Logic App to process the file when a change is detected.
2. Create an Event Grid subscription for your ADLS Gen2 storage account and specify the events you want to monitor, such as "Microsoft.Storage.BlobCreated" or "Microsoft.Storage.BlobDeleted".
3. Configure the endpoint for the Event Grid subscription to point to your Azure Function or Logic App.
4. When a change occurs in the metadata of a file, Event Grid will trigger an event and send a notification to your specified endpoint. Your Azure Function or Logic App can then process the file based on the event data.

Note that this approach will not detect changes in the content of a file, only changes in the metadata. If you need to monitor changes in the content of a file, you may need to consider using a third-party tool or building a custom solution.

Hope this helps. Do let us know if you any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.