question

1rajat-2101 avatar image
0 Votes"
1rajat-2101 asked MartinJaffer-MSFT answered

How to set 'DependsOn' setting dynamically in copy activity in Data Factory

I am trying to set 'DependsOn' property in copy activity using parameters. Is there any way to do it ?

azure-data-factory
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

VaibhavChaudhari avatar image
0 Votes"
VaibhavChaudhari answered

Not sure if it is possible to make dependsOn dynamic but see if you can use IF condition activity and then use Copy activity in it based on condition.


If the response helped, do "Accept Answer" and upvote it - Vaibhav

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1rajat-2101 avatar image
0 Votes"
1rajat-2101 answered MartinJaffer-MSFT converted comment to answer

Adding IF activity defeats the purpose of limiting total number of activities at 40. have a master ETL pipeline that loads more than 80 tables currently (will increase in near future to be around 160). Currently I have bundled all the activities into stages based on their dependencies and used the bundled pipelines in the master to limit them below 40 activities. However, if I could set 'DependsOn' dynamically, then I can use 'Foreach' to load each table (activity) easily with significantly less number of activity in a single master pipeline. Hope I am explain my issue here. Please let me know if there is a solution for it.


@MalleswaraReddy : I have the pipelines created, just that I want to be able to use Foreach to load all the tables with their dependencies.


Thanks.


5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

MartinJaffer-MSFT avatar image
0 Votes"
MartinJaffer-MSFT answered

I do not fully understand you situation, but it sounds like you need more options for controlling flow and logic. I have many tricks that may or may not be helpful. I will share one below.
If I had a diagram of the logic you want to implement, I might be able to suggest a more manageable setup.

I use the below pattern when I I want to branch my control flow, without using an If Activity.
9903-logiccontrol.jpg

I intentionally cause the set variable activity to fail using an expression like @if(equals(pipeline().parameters.DoSucces,true),true,'IMakeError')
Then I use the dependency conditions to branch my flow. The highlighted 'skipped' dependency is important. Without the highlighted skipped dependency, the pipeline will report failure status.


logiccontrol.jpg (67.3 KiB)
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@RajatSharma-9776 Did my suggestion help? If you found your own solution, could you please share it for the benefit of others?

0 Votes 0 ·