skip activity in Azure Data Factory debug mode

Question

Basic question
How can I skip an activity within a pipeline in Azure Data Factory if the pipeline runs in debug mode?

Background information
I have a complex pipeline setup (one master pipeline that triggers multiple sub pipelines) which also triggers fail messages if some activities failed. When testing things in debug mode, the fail messages are also triggered. This should not be happening to avoid spam.

Current approach
I could use the system variable @pipeline().TriggerType, which has the value Manual and pass that information as parameter from master pipeline through every single sub pipeline and check for the trigger type before sending the message (if triggerType != Manual). But this would mean a lot of changes and more things to consider when creating new pipelines, because that parameter always needs to be there then.

Does anyone have a better idea? Any idea how I can check in a sub-pipeline if the whole process was initially triggered via a scheduled trigger or as a debug run?

Answer 1

Hi @dominik.braun ,

Welcome to Microsoft Q&A platform and thanks for posting your question here.

As I understand your ask, you want to skip some activities if the pipeline execution trigger type is manual . Since you don't want to create multiple parameters within the child pipelines , you can try following approach :

First of all remove the activity which is triggering the fail message from child pipeline, we need to accommodate it in parent pipeline itself so that the need of passing the parameter doesn't come into picture. Also, the pre-requisite is to have Azure SQL server provisioned so that SQL query can be run.

1. In all the child pipelines, Add a lookup activity that will run an update query in SQL table and update the column called 'status' . If all the activities ran successfully, connect the lookup via success path to update the column to 'Success'. Similarly, have another lookup activity connected via failed path , if activity failed, update the column to 'Failed'

2. In the parent pipeline, after the execute pipeline activity , Use another lookup activity to run a select query and fetch the value from 'Status' column .

3. Use If activity to check if the status column holds 'Failed' and if the pipeline trigger type ='Manual', then in true block, call the activity triggering fail message . In false block, do nothing.

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you.
  Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

Answer 2

Hello @dominik.braun

Currently we can't disable / skip an activity in ADF pipeline during its run

Please submit the feedback for this feature here:

https://feedback.azure.com/forums/270578-data-factory/suggestions/33111574-ability-to-disable-an-activity

You can either follow one of these for now:

1) Manually delete the activity and click debug for execution but don't publish it

2) Create a copy of that pipeline by cloning from original pipeline and delete the activities that you need to skip and save that with a suffix DEBUG which will become easy to identify and then you can run that pipeline whenever you need to debug

3) Perform the steps using parameter as you mentioned

Thanks