Trigger ADF pipeline based on two events

Question

There is an ADF pipeline which I want to trigger when two events are met. For example, when a file is uploaded to a Storage account and at some point when another file is deleted from a different location.

Can we establish this dependency ?

Answer 1

Hello @Adarsh Pansari, Welcome to Q&A forum!

If your scenario of adding and deleting the files occur at a defined internal, the you may want to first monitor the triggers first and then execute the Main Pipeline which will process the files!!!

Below is one of the few solutions which we can implement the scenario!

1. Trigger 1: Activates when a file is uploaded to a Storage account. This trigger starts a pipeline (Pipeline 1) that records this event to an Azure SQL Database or Table Storage. It will record some information like the filename, time of upload, and an "Upload Event" status.
2. Trigger 2: Activates when a file is deleted from a different location. This trigger also starts a separate pipeline (Pipeline 2) that records this event to the same Azure SQL Database or Table Storage. It will record similar information like filename, time of deletion, and a "Delete Event" status.
3. Create a polling pipeline (Pipeline 3) that periodically checks this Azure SQL Database or Table Storage. This pipeline checks if both the upload and delete events have occurred (based on the status in the database or table). If they have, it triggers the main data processing pipeline and clears the status/events from the SQL Database or Table Storage.
4. Schedule Pipeline 3 using a tumbling window or schedule trigger to run at a frequency that meets your business requirements.

Apart from the above solution, we can even use Logic app to monitor the File additions and deletion and then execute the ADF pipeline!

---

Below Scenario (a different scenario!)will trigger the pipeline when a file is added or deleted but it will not wait for the 2 events and I don't think we have the ADF feature to execute if 2 events are executed concurrently!

We can add multiple Triggers to the same ADF pipeline.

Below is an example of Pipeline which is trigegred by 2 Events: Addition of Files And Deletion of Files

Created 2 Event based triggers!

Monitor the triggered Pipeline
As i have added Files to the input and deleted a file , it triggered the same pipeline.

Created 2 Storage Event Triggers

If the response is helpful, please click "Accept Answer" and upvote it. So that we can close this thread.