Azure Storage Events - Parquet file content change auto refresh

Dondapati, Navin 281 Reputation points
2023-10-26T17:45:37.87+00:00

Hi Guys,

We do have an azure storage event on storage container whenever new file arrive, it triggers the event and move file to different folder, But we are observing, we need an continuous file watch event not on file arrival, but if content of file changes, i don't see any out of the box event trigger available via events, Apart from azure function/logic apps/power apps etc., what options do we have within ADLS2 to accomplish?

Regards,

Navin

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,499 questions
{count} votes

1 answer

Sort by: Most helpful
  1. PRADEEPCHEEKATLA 90,246 Reputation points
    2023-10-27T07:56:50.4766667+00:00

    @Dondapati, Navin - Thanks for the question and using MS Q&A platform.

    Azure Data Lake Storage Gen2 (ADLS Gen2) does not provide an out-of-the-box event trigger for detecting changes in the content of a file. However, you can use Azure Event Grid to monitor changes in the metadata of a file, such as the file size, last modified time, or other properties.

    Azure Data Lake Storage (ADLS) Gen2 can now publish events to Azure Event Grid to be processed by subscribers such as WebHooks, Azure Event Hubs, Azure Functions and Logic Apps. With this capability, individual changes to files and directories in ADLS Gen2 can automatically be captured and made available to data engineers for creating rich big data analytics platforms that use event-driven architectures.

    The events that will be made available for Azure Data Lake Storage Gen2 are BlobCreated, BlobDeleted, BlobRenamed, DirectoryCreated, DirectoryDeleted and DirectoryRenamed.

    Event Grid uses event subscriptions to route event messages to subscribers. This image illustrates the relationship between event publishers, event subscriptions, and event handlers.

    Event Grid Model

    To accomplish this, you can create an Event Grid subscription for your ADLS Gen2 storage account and specify the events you want to monitor. When a change occurs in the metadata of a file, such as the file size or last modified time, Event Grid will trigger an event and send a notification to your specified endpoint. You can then use this notification to trigger an action, such as running an Azure Function or Logic App to process the file.

    Here are the high-level steps to set up an Event Grid subscription for ADLS Gen2:

    1. Create an Azure Function or Logic App to process the file when a change is detected.
    2. Create an Event Grid subscription for your ADLS Gen2 storage account and specify the events you want to monitor, such as "Microsoft.Storage.BlobCreated" or "Microsoft.Storage.BlobDeleted".
    3. Configure the endpoint for the Event Grid subscription to point to your Azure Function or Logic App.
    4. When a change occurs in the metadata of a file, Event Grid will trigger an event and send a notification to your specified endpoint. Your Azure Function or Logic App can then process the file based on the event data.

    Note that this approach will not detect changes in the content of a file, only changes in the metadata. If you need to monitor changes in the content of a file, you may need to consider using a third-party tool or building a custom solution.

    Hope this helps. Do let us know if you any further queries.


    If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

    0 comments No comments

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.