Storage event trigger fires twice on one blob (ADF / ADLS Gen2)

Johan á Rogvi-Hansen 0

Hi.

I have multiple ADF pipelines with a storage event trigger, but one of them always fires twice though only one blob is uploaded. It's also the largest one (.csv/blob > 50 mb). I tried modifying the advanced filtering in the event grid subscription, so that I only have CopyBlob, PutBlob, PutBlockList and FlushAndClose, but to no avail. What can I do?

Thank you in advance.

KranthiPakala-MSFT 46,487 Reputation points Microsoft Employee

2023-09-28T21:58:26.9266667+00:00

@Johan á Rogvi-Hansen Thanks for using Microsoft Q&A forum and posting your query.

Could you please confirm if that particular file is being generated by the same process as the other files that triggered only once? If different, could you please confirm what is the process that is generating those events?
Johan á Rogvi-Hansen 0 Reputation points

2023-09-29T09:25:39.2733333+00:00

Hi @KranthiPakala-MSFT

It's six files which are copied from a SharePoint folder to a specified ADLS Gen2 account, each .csv file as a blob to their own folder, by a Power Automate flow. I then created a storage event trigger in ADF, for each folder, which executes a Databricks notebook. For all the other files, triggers and pipelines it works with the exception of this one file, which also happens to be the largest. The trigger fires twice, the second being around 10-15 seconds after the first. So, in short, yes, it's the same process, the main difference being the file size (though not significantly). I hope this answers your question.
KranthiPakala-MSFT 46,487 Reputation points Microsoft Employee

2023-09-29T17:08:34.7533333+00:00

@Johan á Rogvi-Hansen Thanks for additional context.

This sounds strange as all the files being generated by same process and the issue is happening only with the largest file. This may need deeper investigation to get better understanding of the write behavior of Power automate for the large file hence would recommend filing a support ticket for deeper investigation.

Please do share the support ticket number once it is filed so that we can track it internally. In case if you don't have a support plan, please let me know.

Thank you
Johan á Rogvi-Hansen 0 Reputation points

2023-10-02T07:48:25.01+00:00

Hi @KranthiPakala-MSFT

Thank you for your reply. We do not currently have a support plan.
KranthiPakala-MSFT 46,487 Reputation points Microsoft Employee

2023-10-02T16:43:34.46+00:00

@Johan á Rogvi-Hansen Could you please send an email to AzCommunity[at]Microsoft[dot]com with the below details, so that we can enable a one-time-free support for you to work closely on this matter.

Email subject: <Attn - Kranthi : Microsoft Q&A Thread title>

Thread URL: <Microsoft Q&A Thread>

Subscription ID: <your subscription id>

Thank you
Johan á Rogvi-Hansen 0 Reputation points

2023-10-03T14:14:14.43+00:00

@KranthiPakala-MSFT sent :)
Nandan Hegde 32,336 Reputation points MVP

2023-10-04T04:54:56.2366667+00:00

Hey, Just for debugging purpose:
Can you capture the file name that is creating this triggers.
https://stackoverflow.com/questions/62052791/how-to-get-the-name-of-the-file-that-triggered-the-azure-data-factory-pipeline

And confirm whether in both instances the file name is same?
Also leverage Get Meta data activity to check the file size for that trigger and confirm those aspects during the 2 triggers
KranthiPakala-MSFT 46,487 Reputation points Microsoft Employee

2023-10-06T01:20:38.46+00:00

@Johan á Rogvi-Hansen Thanks for sharing the details over email. Please keep this thread posted once the root cause and solution of the issue has been identified as it would help others reading this thread.

Thank you

Share via

Storage event trigger fires twice on one blob (ADF / ADLS Gen2)

Your answer