Azure Data Factory is not firing the trigger when I create a blob as output from ETL tool

Azevedo Emanoel 1

Hi,

I'm using a ETL tool to save files into a blob container and it is not firing the trigger that I have created to the process.

When I manually upload the file to the storage, it works fine.

Can someone help?

BhargavaGunnam-MSFT 28,071 Reputation points Microsoft Employee

2024-04-19T17:07:24.86+00:00

Hello Azevedo Emanoel,

Are you using the storage event triggers in ADF? if yes, can you please check the trigger configuration?
Azevedo Emanoel 1 Reputation point

2024-04-19T17:52:19.7766667+00:00

Yes, I'm using storage events.

As I told, when I upload or use AzCopy to upload the files, the trigger works fine. When I create the file as output of my ETL tool, it doesn't fire the trigger.

Find attached the trigger configuration.

Blob Container File:
BhargavaGunnam-MSFT 28,071 Reputation points Microsoft Employee

2024-04-19T18:33:40.7833333+00:00

from the below document:
The Storage Event Trigger currently supports only Azure Data Lake Storage Gen2 and General-purpose version 2 storage accounts.

Can you please confirm if you are using ADLS gen2 or general purpose v2 ?

https://learn.microsoft.com/en-us/azure/data-factory/how-to-create-event-trigger?tabs=data-factory

also can you please check if Soft delete enabled. When the JSON file is created on Storage side, check "show deleted objects"

It's possible that a file with same name was previously deleted and now is waiting for 7days to be flushed.

Check the file path in the container. If the file is located in subfolders, try to use the full path up to that file in the trigger configuration.

Ex. File the file is located in container/folder/subfolder/file.json, use :

Container name : container/folder/subfolder
Azevedo Emanoel 1 Reputation point

2024-04-19T19:03:26.53+00:00

Yes, it is ADLS gen2 and the soft delete is not enabled.

Once again, as I've told, it is not working just when I create the file as output of my ETL tool.

When I copy the file with AzCopy, or upload the file manually, it works with no problem.

The difference I can see now is that, when I upload it manually, the content of the blob is text/plain:

When it is created from the ETL, the content is application/octet-stream:

Could it be the problem?
BhargavaGunnam-MSFT 28,071 Reputation points Microsoft Employee

2024-04-19T19:32:09.37+00:00

Thanks for checking these.

I am not sure "content type" would cause an issue.

Can you please change the ignore Empty Blobs to No and see if it makes any difference.
Azevedo Emanoel 1 Reputation point

2024-04-19T20:09:53.01+00:00

I changed it, but no success. The trigger is not starting.
BhargavaGunnam-MSFT 28,071 Reputation points Microsoft Employee

2024-04-19T22:12:55.6366667+00:00

Could you try using a different storage account? It appears that the current storage account is not generating the event for ADF to process.

If that doesn't help, further investigation from the storage end will be needed. I suggest opening a support case for additional troubleshooting. If you don't have a support plan, please let me know so I can enable a one-time free support request for this case.

I look forward to hearing from you.
Azevedo Emanoel 1 Reputation point

2024-05-19T10:17:04.2333333+00:00

Hi,

I got it resolved creating anew blob storage, thanks.
BhargavaGunnam-MSFT 28,071 Reputation points Microsoft Employee

2024-05-20T18:17:52.56+00:00

Hello Azevedo Emanoel,

I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to "Accept " the answer.

1 answer

BhargavaGunnam-MSFT 28,071 Reputation points Microsoft Employee

2024-05-20T18:17:51.5033333+00:00
Hello Azevedo Emanoel,

I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to "Accept " the answer.

Issue:

Azure Data Factory is not firing the trigger when created a blob as output from ETL tool

Solution:

It appears that the current storage account is not generating the event for ADF to process. The issue was resolved after creating a new blob storage account.

If I missed anything please let me know and I'd be happy to add it to my answer, or feel free to comment below with any additional information.

Please remember to "Accept Answer" if any answer/reply helped, so that others in the community facing similar issues can easily find the solution.
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

Share via

Azure Data Factory is not firing the trigger when I create a blob as output from ETL tool

1 answer