Azure Storage V2 blob event - Not triggering ADF

Question

I have an Azure Data Factory V2 (X), with a blob event trigger. The blob event trigger works fine and triggers the Data Facotry (X) when I manually upload a file using the storage explorer. But when I use another data factory (Y) to write the file to the same Blob Storage instead of manual write, the event doesn't trigger my Data Factory (X).
I have verified the following:

1. There are no multiple events under the 'Events' blade section of the Blob Storage.
2. There is a System Topic and a Subscription created with the correct filters. I have the 'BeginsWith' with my container name and 'EndsWith' with the file extension 'parquet'.
3. In the Event Subscription filters, I have 'CopyBlob' and 'CreateFile' and the number of filters less than 5

I checked the logs of the Event Topic & Subscription, when the file is written by the ADF (Y) there is no event generated but with the manual upload/write the event gets triggered.

[@KranthiPakala-MSFT], I saw your comments in this thread - adf-event-based-trigger-fires-when-file-created-manually-but-not-when-an-another-pipeline-creates-a

Answer 1

I encountered a similar issue writing CSV data from an application using Azure SDK. I observed that the Storage Events trigger wasn't firing when the application wrote to ADLSG2, but when I manually uploaded the file, the trigger fired.

I compared the properties of the files written by the application and the file I manually uploaded to ADLS and discovered their CONTENT-TYPE properties differed. The file created by the application was an application/octet-stream, while the file I uploaded was a text/csv. Since the application's data was written as a stream, it starts with "0 bytes and file size grows until all the data is written permanently"; consequently, if you created the trigger to "Ignore empty blobs", the file is actually empty at the first instance, and so ignored.

I solved this issue by setting "Ignore empty blobs" to No when creating the trigger.

Hope this helps!

Answer 2

Hi @Satya D ,

As confirmed offline your issue was resolved by using ADLS gen2 instead of regular storage account.

Could you also please confirm if you were able to test copying a sample blob from regular Azure Storage account using ADF event triggers?

Thank you.