ADF storage event trigger pipeline triggered by SFTP upload in storage account. Filters are deleted if trigger is stopped and start.

Question

Hi everyone,

When you create a storage event trigger in Azure Data Factory, it automatically creates a new Event Grid system topic along with a webhook subscription:

To ensure that the trigger works when you upload files to the storage account using SFTP, you need to add filters for the data.api property with values SftpCommit and SftpCreate:

If the trigger is stop and start again, for instance, during a deployment, these manually added filters are gone:

My questions:

• Is there any way to avoid that?
• If not, what could be a consistent alternative

This is causing some maintenance overhead we would like to avoid.

BR. Paul

Answer 1

@Paul Hernandez

Welcome to the Microsoft Q&A and thank you for posting your questions here

When you stop and start the trigger in Azure Data Factory (ADF), the manually added filters for SftpCommit and SftpCreate are removed, causing maintenance overhead.

Here are a few suggestions respective links attached to it

1. Automation with ARM Templates: Use Azure Resource Manager (ARM) templates to automate the deployment of your ADF triggers. This way, you can include the necessary filters in the template, ensuring they are always applied when the trigger is deployed or redeployed.
2. PowerShell or Azure CLI: Create a script using PowerShell or Azure CLI to reapply the filters whenever the trigger is started. This script can be part of your deployment pipeline to ensure the filters are consistently applied.
3. Event Grid Subscription: Instead of relying solely on ADF triggers, consider creating an Event Grid subscription that listens for the SftpCommit and SftpCreate events and then triggers the ADF pipeline. This approach can provide more control and reduce the dependency on ADF’s built-in triggers.
4. Custom Logic App: Use Azure Logic Apps to handle the event filtering and trigger the ADF pipeline. Logic Apps can provide more flexibility in managing events and can be configured to handle the specific SFTP events.

Implementing one of these solutions should help reduce the maintenance overhead and ensure that your filters are consistently applied

Hope this helps. Do let us know if you any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.