Avoid queuing pipeline execution. In a scheduled pipeline is there a way to not trigger if previous execution is still in progress

Question

Hi Friends,

We have a scheduled pipeline every 10 mins however sometime execution is not completed within 10 mins. so the pipeline get queued and after reaching max allowed queue number of 100 it starts failing.

Is there a way to check if previous execution is completed then only new triggered execution will start else it will not invoke execution.
Thanks in advance

Answer 1

Hello @Vishnu Tupe (US) ,
Thanks for the question and using MS Q&A platform.

There are a few ways I can read your ask.

At face value "how can I prevent the trigger from starting a new pipeline run, if there is an existing pipeline run in progress."
Ideally, you would use a tumbling window trigger, however you have specified using a scheduled trigger. A tumbling window trigger is preferred because it waits for the last run to finish before the new one starts. A scheduled trigger just starts new pipeline run at the specified time, so the only way to avoid this is to stop/pause the trigger.

Before you jump to conclusions, consider this: there is REST API available to stop/start pause/resume triggers.
If we alter the pipeline such that the first activity executed is a Web activity to make the trigger stop/pause, and the last activity executed is a Web activity to start/resume the trigger, then we effectively stop the trigger's ability to make more pipeline runs while this pipeline is running. Precautions should be taken to handle any activity failures that might prevent the last Web activity from executing. Otherwise the trigger would stay deactivated / inactive.

To ensure the final "Re-start the trigger" activity is run, connect it by both blue on-completion dependency and grey on-skipped dependency. This should cover all possibilities.

If you would like to implement this, there is a prerequisite. You must grant this Data Factory the role "Data Factory Contributor" on the scope of the resource group containing the Data Factory. This is so that the Web activities have proper permissions. If you do not do this, then you will get an error like:

Operation on target Stop trigger failed: {"error":{"code":"AuthorizationFailed","message":"The client 'XXXXXXXX' with object id 'XXXXXXX' does not have authorization to perform action 'Microsoft.DataFactory/factories/triggers/stop/action' over scope '/subscriptions/XXXXXX/resourceGroups/XXXXXXX/providers/Microsoft.DataFactory/factories/XXXXX/triggers/XXXXXX' or the scope is invalid. If access was recently granted, please refresh your credentials."}}  

The calls to stop/pause the trigger, and the calls to start/resume the trigger are very similar. The only difference is in the URL.
start stop

https://management.azure.com/subscriptions/MY_SUBSCRIPTION_ID/resourceGroups/MY_RESOURCE_GROUP_NAME/providers/Microsoft.DataFactory/factories/MY_FACTORY_NAME/triggers/MY_TRIGGER_NAME/stop?api-version=2018-06-01  
  
https://management.azure.com/subscriptions/MY_SUBSCRIPTION_ID/resourceGroups/MY_RESOURCE_GROUP_NAME/providers/Microsoft.DataFactory/factories/MY_FACTORY_NAME/triggers/MY_TRIGGER_NAME/start?api-version=2018-06-01  

For both calls, the method should be POST. While these calls don't expect or care about the body, Data Factory validation expects there to be a body. So put some nonsense in the body.

The Authorization type is System Assigned Managed Identity (unless you made a user-assigned identity for the Factory). This is why I had you grant the Data Factory Contributor role earlier.

The Resource is https://management.azure.com/ just like in the URL.

That's it!

Let me know if you want other insight, this answer was only addressing the stated top-level issue.

Please do let me if you have any queries.

Thanks
Martin

------------------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators