How to automatically stop trigger in azure data factory when pipeline failed

Question

Hi All,

In Azure Data Factory a pipeline is running with trigger enabled.
Here the requirement is that trigger should be stopped automatically when the pipeline gets failed. Until the pipeline issue is resolved trigger should not be enabled. Kindly provide your inputs on this.

Answer 1

Hello @Ravikumar Sivasamy and thank you for your question.

Such a feature is not available in Data Factory, but it sounds very useful. I highly recommend you request it in the feedback forum.

I have been thinking hard about ways to achieve your ask: If a pipeline fails, it disables the trigger. Below I detail two solutions. The second one is better, I think. There are other ways to achieve your ask, but they would require an outside resource to monitor the pipeline.

----------

One solution I thought of goes like this:

Since your requirement was to stop the trigger if the pipeline failed, not a specific activity failed, it would make sense to have the pipeline stop the trigger by default, and only enable it again if all activities are successful. While a pipeline can have logic that allows the pipeline to succeed even if one activity fails, a pipeline cannot report failure if all activities succeed.

Alter the pipeline, placing a web activity at the beginning. This one will make a call to the Data Factory REST api to disable the trigger. Add a web activity at the end after all other activities succeed. This one will make a call to the Data Factory REST api to enable the trigger.
Details on how this works later.

There is a possibility all this switching on and off could mess up timing.

----------

Another possibility, this one easier, goes like the following:

You want to disable the trigger when the pipeline fails, but a pipeline cannot test itself after completion, however a parent-child pipeline relationship does allow testing. When the "wait on completion" feature of Execute Pipeline activity is used, the Execute Pipeline activity takes on the success / failure of the child pipeline. This we can make use of.

Create another pipeline. In this one place an Execute Pipeline activity, pointing at the one you really want to run. Set the "wait on completion" flag in the Execute Pipeline activity. Add a Web activity connected to the Execute Pipeline by an on-failure dependancy. The Web activity will call the Data Factory REST API to disable the trigger. Make the trigger start this parent pipeline, and not start the child pipeline.

----------

About using the web activity to stop the trigger...

There are a few authentication methods. I think MSI is easiest. This will require giving permissions to the Data Factory.
The MSI "Resource" should be set to https://management.azure.com/ (see picture)

For sufficient permissions to allow the Data Factory to edit itself, you will need to go to the resource group access control (IAM) and give the Data Factory the role "Data Factory Contributor". Note this does give it permission to affect other factories in the same resource group.

Here is reference documentation for the call we will be using to stop the trigger. It has a "try it" button which is very useful for building and testing the URL.

Answer 2

@MartinJaffer-MSFT I have same scenario but trigger will stop after pipeline successfully completed.