Azure data factory trigger on event not working

Question

Azure data factory trigger on event not working

Kataria, Dipesh SITI-ITY/FD 1

Hi,

I am using Azure Data Factory V2 and want to set up a trigger based on a blob creation in a Gen2 ADLS. When I create the trigger, the paths are recognised as it shows me the blobs of interest, however when this gets published the trigger does not fire when a blob is created. I have checked the event against the Gen2 ADLS and I can see that the event is not being recognised in there. I have noticed this bug before. My location is West Europe.

Saurabh Sharma 23,866 Reputation points Microsoft Employee Moderator

2021-02-27T00:36:58.787+00:00

@Kataria, Dipesh SITI-ITY/FD Thanks for using Microsoft Q&A !!

Is trigger working with manual uploads of the blobs ? Also, have you tried using a Blob storage instead of ADLS to check if the triggers are working ?
Dipesh Kataria 1 Reputation point

2021-02-27T08:02:33.257+00:00

Hi,

The "trigger now" functionality does work. But it does not fire on blob creation.

I will try blob storage, but I have read on many threads that this only works with Gen 2 ADLS.
Saurabh Sharma 23,866 Reputation points Microsoft Employee Moderator

2021-02-27T16:42:01.557+00:00

@Dipesh Kataria @Kataria, Dipesh SITI-ITY/FD No I am not talking about Trigger now. How are you uploading the blobs ? Is this through any automated process and have you tried creating the blob manually to see if the published event trigger gets fired ?
Kataria, Dipesh SITI-ITY/FD 1 Reputation point

2021-02-28T11:22:39.467+00:00

Hey,

Yes there is an automated process (and ADF in a different tenant) which writes into this ADLS. I have manually uploaded a file and this did indeed activate the trigger.

What are the next steps now?

Thanks,

Dipesh
Saurabh Sharma 23,866 Reputation points Microsoft Employee Moderator

2021-03-02T02:07:49.65+00:00

@Kataria, Dipesh SITI-ITY/FD
Can you please verify while generating the files clients use the CreateFile and FlushWithClose operations that are available in the Azure Data Lake Storage Gen2 REST API.
Please refer to the thread
If you are writing your own custom logic then we might need to check your code logic to fill in the gaps.
Kataria, Dipesh SITI-ITY/FD 1 Reputation point

2021-03-02T08:28:48.037+00:00

I am not using the Rest API directly, I am using a copy data activity from an ADF.
Saurabh Sharma 23,866 Reputation points Microsoft Employee Moderator

2021-03-03T00:24:34.73+00:00

@Kataria, Dipesh SITI-ITY/FD No, I am talking about the automation method which is generating files at the ADLS Gen2 ? Is it using CreateFile and FlushWithClose operations ? I am also checking this internally for any other suggestions as well.
Dipesh Kataria 1 Reputation point

2021-03-03T07:02:24.003+00:00

Yes, the automation method is just a copy data activity from an ADF in a different tenant.

This is now being investigated with an official support ticket and I will give the answer here when they find one.
Saurabh Sharma 23,866 Reputation points Microsoft Employee Moderator

2021-03-03T17:38:54.327+00:00

@Dipesh Kataria ok, got it. Sure, Thanks.
kris 1 Reputation point

2021-07-19T14:30:28.217+00:00

All, I am seeing this same behavior but only if the blobs are automatically created by another pipeline. If i manually upload to the storage account (v2) then the trigger works.
Jeff Born (J&CLT-ATL) 106 Reputation points

2022-06-17T17:30:11.407+00:00

It's been over a year since this was reported, is there any update on when this will be fixed.

I'm experiencing the same error 16 months later. Trigger files if I place the file there using Storage Explorer, but using ADFv2 Copy Activity to create and populate the file does not trigger the storage event trigger. Please provide an update, like bug/issue link so we can track the progress and upvote.

Your answer

Saurabh Sharma 23,866 Reputation points Microsoft Employee Moderator

2021-02-27T00:36:58.787+00:00

@Kataria, Dipesh SITI-ITY/FD Thanks for using Microsoft Q&A !!

Is trigger working with manual uploads of the blobs ? Also, have you tried using a Blob storage instead of ADLS to check if the triggers are working ?
Dipesh Kataria 1 Reputation point

2021-02-27T08:02:33.257+00:00

Hi,

The "trigger now" functionality does work. But it does not fire on blob creation.

I will try blob storage, but I have read on many threads that this only works with Gen 2 ADLS.
Saurabh Sharma 23,866 Reputation points Microsoft Employee Moderator

2021-02-27T16:42:01.557+00:00

@Dipesh Kataria @Kataria, Dipesh SITI-ITY/FD No I am not talking about Trigger now. How are you uploading the blobs ? Is this through any automated process and have you tried creating the blob manually to see if the published event trigger gets fired ?
Kataria, Dipesh SITI-ITY/FD 1 Reputation point

2021-02-28T11:22:39.467+00:00

Hey,

Yes there is an automated process (and ADF in a different tenant) which writes into this ADLS. I have manually uploaded a file and this did indeed activate the trigger.

What are the next steps now?

Thanks,

Dipesh
Saurabh Sharma 23,866 Reputation points Microsoft Employee Moderator

2021-03-02T02:07:49.65+00:00

@Kataria, Dipesh SITI-ITY/FD
Can you please verify while generating the files clients use the CreateFile and FlushWithClose operations that are available in the Azure Data Lake Storage Gen2 REST API.
Please refer to the thread
If you are writing your own custom logic then we might need to check your code logic to fill in the gaps.
Kataria, Dipesh SITI-ITY/FD 1 Reputation point

2021-03-02T08:28:48.037+00:00

I am not using the Rest API directly, I am using a copy data activity from an ADF.
Saurabh Sharma 23,866 Reputation points Microsoft Employee Moderator

2021-03-03T00:24:34.73+00:00

@Kataria, Dipesh SITI-ITY/FD No, I am talking about the automation method which is generating files at the ADLS Gen2 ? Is it using CreateFile and FlushWithClose operations ? I am also checking this internally for any other suggestions as well.
Dipesh Kataria 1 Reputation point

2021-03-03T07:02:24.003+00:00

Yes, the automation method is just a copy data activity from an ADF in a different tenant.

This is now being investigated with an official support ticket and I will give the answer here when they find one.
Saurabh Sharma 23,866 Reputation points Microsoft Employee Moderator

2021-03-03T17:38:54.327+00:00

@Dipesh Kataria ok, got it. Sure, Thanks.
kris 1 Reputation point

2021-07-19T14:30:28.217+00:00

All, I am seeing this same behavior but only if the blobs are automatically created by another pipeline. If i manually upload to the storage account (v2) then the trigger works.
Jeff Born (J&CLT-ATL) 106 Reputation points

2022-06-17T17:30:11.407+00:00

It's been over a year since this was reported, is there any update on when this will be fixed.

I'm experiencing the same error 16 months later. Trigger files if I place the file there using Storage Explorer, but using ADFv2 Copy Activity to create and populate the file does not trigger the storage event trigger. Please provide an update, like bug/issue link so we can track the progress and upvote.

Share via

Azure data factory trigger on event not working

Your answer