Azure Data Factory event storage trigger doesn't run when another pipeline upload new file

Question

I'm using Azure Data Factory and I have a pipeline that creates a file in Blob Storage Account. Storage Account is StorageV2 (general purpose v2).

I have a second pipeline that is subscribing to Storage Event. When I'm adding the file to the specified container manually then trigger runs pipeline. However, when I run the first pipeline and the file is added to the blob automatically, the trigger doesn't run.

How can I resolve this problem? Maybe there is better solution for scenario like that.

Answer 1

Hello @Juszcze, Krzysztof
Thanks for the ask and using the Microsoft Q&A platform .
I did tried the setup and it works for me . Let me share my implementation and then you can compare with yours .

This is the setup for the trigger

In my case the trigger is firing as exptected .

Please do let me know how it goes .
Thanks
Himanshu
Please do consider clicking on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members

Answer 2

Hello @HimanshuSinha-msft .

Yes, my trigger is listening the container test-trigger-sink and this is a container where I copy the data exactly.
I've created a generic data set and it looks like that under the hood:

This concatenate "test-trigger-" with "sink".

And the copy activity works well, the file is added to the container:

I think it's a problem with the event storage trigger which is not run in my second pipeline when the file is upload by another pipeline.
Above it's only a reproduced situation from my project solution where it's also doesn't work.
It looks for me like a bug in ADF or there is some configuration that stopped fire the trigger if the file is upload by pipeline.

Answer 3

I am having a similar issue. The trigger executes successfully when I manually upload a file to the storage account, however, when a file is uploaded to the storage account from using the ftp protocol the trigger is never prompted. I downloaded the file to my local, deleted the file from the storage account then manually uploaded the exact same file to the storage account and the trigger works fine. Very Disappointing. FWIW, I have confirmed the storage account is ADLS Gen2 and Hierarchical namespace is enabled.

Answer 4

Thanks, for the answer @HimanshuSinha-msft . However, your implementation doesn't reflect my situation. I'm talking about a situation when the file is added to a container by activity in Azure Data Factory (ex. Copy Activity).

1. I've created a simple copy activity that uploads a file in the container
2. Next, I've created a storage event trigger
3. Then, I've created a second activity that is listening to the trigger
4. I run the activity of point 1. and the file is created in the container but the trigger not run. When I upload a file from Azure Portal or Azure CLI to a container then the trigger runs the activity (from point 3).

Answer 5

specify file extension for the property "blob path ends with" like .csv or .txt.