How to set 'DependsOn' setting dynamically in copy activity in Data Factory

1rajat 1 Reputation point
2020-06-08T15:59:55.553+00:00

I am trying to set 'DependsOn' property in copy activity using parameters. Is there any way to do it ?

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,171 questions
{count} votes

3 answers

Sort by: Most helpful
  1. Vaibhav Chaudhari 38,681 Reputation points
    2020-06-08T18:49:19.757+00:00

    Not sure if it is possible to make dependsOn dynamic but see if you can use IF condition activity and then use Copy activity in it based on condition.


    If the response helped, do "Accept Answer" and upvote it - Vaibhav

    0 comments No comments

  2. 1rajat 1 Reputation point
    2020-06-09T05:15:02.747+00:00

    Adding IF activity defeats the purpose of limiting total number of activities at 40. have a master ETL pipeline that loads more than 80 tables currently (will increase in near future to be around 160). Currently I have bundled all the activities into stages based on their dependencies and used the bundled pipelines in the master to limit them below 40 activities. However, if I could set 'DependsOn' dynamically, then I can use 'Foreach' to load each table (activity) easily with significantly less number of activity in a single master pipeline. Hope I am explain my issue here. Please let me know if there is a solution for it.

    @MalleswaraReddy : I have the pipelines created, just that I want to be able to use Foreach to load all the tables with their dependencies.

    Thanks.

    0 comments No comments

  3. MartinJaffer-MSFT 26,061 Reputation points
    2020-06-11T21:07:59.67+00:00

    I do not fully understand you situation, but it sounds like you need more options for controlling flow and logic. I have many tricks that may or may not be helpful. I will share one below.
    If I had a diagram of the logic you want to implement, I might be able to suggest a more manageable setup.

    I use the below pattern when I I want to branch my control flow, without using an If Activity.
    9903-logiccontrol.jpg

    I intentionally cause the set variable activity to fail using an expression like @if(equals(pipeline().parameters.DoSucces,true),true,'IMakeError')
    Then I use the dependency conditions to branch my flow. The highlighted 'skipped' dependency is important. Without the highlighted skipped dependency, the pipeline will report failure status.