Azure Data Factory Pipe line trigger is firing multiple times

Question

Hello:

New to Data Factory, but happy I have gotten a couple of things to work.

I have a pipeline that copies files from a blob "Inbox" to another folder in the data lake.

I expected four files. This works.

I have added an event trigger to the pipeline to fire the pipeline when the files come in. When I copy the files into the folder the pipe line kicks off four times. 'Cause there are four files. But when the multiple pipeline instances run there are lots of errors, 'cause the later runs are looking for files that have already been moved.

Is there a way to delay the trigger. Like if the trigger has fired do not test again for 10 minutes. Or is there a method to tell the pipeline NOT to run if an instance is already running.

Thanks for your attention to this matter.

KD

Answer 1

Please refer the below blog for guidance:

https://datasharkx.wordpress.com/2022/09/24/event-trigger-of-synapse-azure-data-factory-pipeline-on-arrival-of-nth-file-in-azure-blob-storage/

there is something called concurrency:

that ensures only 1 instance of pipeline is running even though there are parallel triggers.

Note : The above blog is for reference and you can update it as per your use case