Is there a way to restrict multiple instances of an ADF pipeline

Question

Hi all,

My scenario is like this. Pipeline p1 is having an event based trigger (a trigger file in azure blob). Now if p1 is running and during that another trigger file comes up, I do not want the second instance to run unless first one is completed. Would it be possible through the azure portal?

Thanks!

Answer 1

Hi @xyz_chatter ,

Thanks for your query. You can try utilizing the Concurrency property in pipeline General settings. This property is used to control the number of simultaneous pipeline runs that are fired. If you set Concurrency = 1 , whenever a new event is triggered it will start the first pipeline run, and if another event is triggered then the trigger runs again and tries to run the pipeline, but since the first run hasn't finished, it will add it to the queue until the previous run of the pipeline in completed. After the first run finishes, it will start to run the second one and so on...

Please note that there is a limit on Queue size of 100, so when using pipeline concurrency of 1, you could have 1 pipeline run InProgress and 100 in the Queue, after that, if they keep triggering they will get error 429.

Hope this helps.

----------

Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.