ADF storage event trigger pipeline is not triggered when file is SFTP to Azure Blob Storage.

Shrivastava, Ravi-ZB 0

Hi,

I have a file coming from on-prem app through SFTP to Azure Blob Storage container. I set up an event storage trigger on my pipeline. However the pipeline is not getting started when the file (blob) created on Azure storage container through SFTP. On the other hand when I manually upload a file, it works fine. I searched and found event grid schema to use, but that is not supportive per my application. Please assist what is missing or what need to be done.

Tushar Kumar 3,331 Reputation points MVP

2023-09-25T15:17:38.07+00:00

Do you have the ignore Empty Blob option enabled? if yes can you please set it to No and check.
Shrivastava, Ravi-ZB 0 Reputation points

2023-09-25T15:29:02.7766667+00:00

Hi Tushar, I tried changing the option to ignore empty blob to No, still not working.
Tushar Kumar 3,331 Reputation points MVP

2023-09-25T15:33:04.44+00:00

Can you add a screenshot of your tigger setting masking any sensetive data just for review here!
Shrivastava, Ravi-ZB 0 Reputation points

2023-09-25T15:48:40.93+00:00

Hi Tushar,

Please find below. I just removed subscription and storage account name for screenshot.

Page-1:

Page-2: Checks for file available in container. skipping that screenshot.

Page-3:
Ramya Harinarthini_MSFT 5,351 Reputation points Microsoft Employee

2023-09-26T10:16:47.78+00:00

Shrivastava, Ravi-ZB

Just checking in to see if the below answer helped. If this answers your query, do click "Accept the answer” for the same, which might be beneficial to other community members reading this thread. And, if you have any further query do let us know.

1 answer

Ramya Harinarthini_MSFT 5,351 Reputation points Microsoft Employee

2023-09-25T16:33:59.39+00:00

Shrivastava, Ravi-ZB Welcome to Microsoft Q&A, thank you for posting your here!!

Please make sure you add the sftpcreate and sftpcommit on the Advanced filters on the Event Subscription as shown in the below screenshots.

Navigate to the Azure Storage account -->Events-->Select Event Subscription-->Webhook

Now click on Filters--> In the Advanced Filters section add sftpcreate and sftpcommit API's.

Now try to upload any blob through SFTP and let us know if the trigger runs.

Note: I have tried to reproduce this, and it is working in my test environment

Hope this helps!
Kindly let us know if the above helps or you need further assistance on this issue.

Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.
Please sign in to rate this answer.

2 people found this answer helpful.
Shrivastava, Ravi-ZB 0 Reputation points

2023-09-26T14:55:43.9033333+00:00

Hi Ramya,

As per our organization we are restricted to add filter from portal. Could you please share the azure cli command which can be used. Meanwhile we are researching the other ways to add these sftp filter.

Ramya Harinarthini_MSFT 5,351 Reputation points Microsoft Employee

2023-09-28T10:56:29.0433333+00:00

Shrivastava, Ravi-ZB

Apologies for the delay in response. We have tried to check internally for Az cli commands with other Support Engineers from Data Factory and Event Grid.

az eventgrid event-subscription update --name 1178ee51-3b84-8cf8-a180-82eea8f1be01 --source-resource-id "/subscriptions/subscriptionid/resourceGroups/resourcegroupname/providers/Microsoft.Storage/storageaccounts/Storageaccountname" --endpoint "https://pmwesteurope.svc.datafactory.azure.com:4443/triggerevent/BlobEventsTrigger/newsftptrigger" --advanced-filter data.api StringIn sftpcreate

Source Link:https://learn.microsoft.com/en-us/cli/azure/eventgrid/event-subscription?view=azure-cli-latest#az-eventgrid-event-subscription-update()

In your data factory do you have the validation event
https://learn.microsoft.com/en-us/azure/event-grid/webhook-event-delivery#validation-details

We have tried all possible ways to update the Advanced filter using Azcli however, we failed to validate the Webhook handshake for Data Factory trigger URL AS Http POST request failed with response code Unknown.

For quicker resolution I would recommend you contact support for Data Factory service If you have a support plan. Request you file a support ticket, else please do let us know, we will try and help you get a one-time free technical support. In this case, could you send an email to AzCommunity[at]Microsoft[dot]com referencing this thread as well as your subscription ID. Please mention "ATTN subm" in the subject field. Thank you for your cooperation on this matter and look forward to your reply.

Shrivastava, Ravi-ZB 0 Reputation points

2023-09-28T15:02:02.43+00:00

Thank you. I sent the email to provided email with mentioned details. Could you please do the needful. Thank you.

Jesse Jenken 1 Reputation point

2024-01-18T21:59:36.3366667+00:00

Are you suggesting we add a new Event Subscription, or click into the current one created by ADF? My ADF trigger isn't appearing under the Storage Account Events, is my problem. Where do these ADF triggers get registered?

Shaik, Raheem 0 Reputation points

2024-02-22T09:32:49.96+00:00

This solution is working perfectly for me.

Jesús Eduardo Álvarez Aguilar 0 Reputation points

2024-03-18T15:06:31.34+00:00

Hello, the solution works fine, but I was wondering if there is a filter that I can add so that when someone drops multiple files through the SFTP, the trigger waits until every file is loaded and then acts. Right now I have the issue that my trigger is getting executed depending on the number of files; for example, if someone drops 10 files, it executes 10 times.

Thanks a lot.

Paul Hernandez 676 Reputation points Microsoft Employee

2024-08-15T12:49:09.1566667+00:00

Dear @Ramya Harinarthini_MSFT ,

I tried adding the data.api operations you proposed to the event subscription but I have the following maintenance problem: when the trigger is stopped and restarted, for example, during a factory deployment, the filters I added are gone.
Do you know if there is a well-known reason for that, or if you can share a good workaround for it. At the moment, my customer is just re-adding the filters manually after deployments.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

ADF storage event trigger pipeline is not triggered when file is SFTP to Azure Blob Storage.

1 answer

Your answer