Run pipeline activity based on condition in ADF.

Question

Run pipeline activity based on condition in ADF.

Nikunj Patel 51

Hello

Below screenshot shows my pipeline structure. The goal of my pipeline, get all list of feeds in for each activity (First screenshot) and run one by one all table feed inside for each activity and create feed file in data lake (Second Screenshot).

Also, We have one SQL table called FeedHistory which schema like below and it mainly logged each feed entry in this table.

FeedId (int)
JobId (int)
FeedStartdate (Datetime)
FeedEndDate (Datetime)
FeedName (varchar)
IsCompleted (Bit)

Inside For each loop we logged entry in table with Is completed =0 and Process End date is NULL. Once copy activity completed then we update that feed with completed to 1 and so on.

Sometime, we facing issue like Copy activity failed due to such reasons and we rerun the pipeline and this time it reprocess all the feeds again and take same amount of time.

To restrict this, my plan is check if feed already processed with isCompleted= 1 then we need to skip those feed inside for each loop. but How can I achieve this I don't know.(I tried lookup ,if condition but it not make sense to me) so can anyone help me or guide me to achieve this scenario.

Thank you in advance.

Answer 1

Pratik Somaiya 4,126

Hello @Nikunj Patel

You can use a lookup activity wherein you would add the query having a where clause to skip rows having isCompleted = 1 like a normal SQL query

Then the output of the Lookup activity can be passed as an input to the ForEach

Let me know of any queries

Nikunj Patel 51 Reputation points

2022-03-28T05:04:18.503+00:00

Thank you @Pratik Somaiya for your response. I thought this logic but it wont helpful in my pipeline structure

Pipeline logic is reads feed data from json and provide input as for each loop and based on that inside feed will process. if I changed then it huge change in my all pipelines. Instead of this you have any idea how can I do this inside for each loop?
Pratik Somaiya 4,126 Reputation points

2022-03-28T05:16:26.89+00:00

@Nikunj Patel : There's an inbuilt @json expression in ADF, can you check if this helps: https://learn.microsoft.com/en-us/azure/data-factory/control-flow-expression-language-functions#json
Nikunj Patel 51 Reputation points

2022-03-28T05:25:31.103+00:00

Ok @Pratik Somaiya I will look this and check is this fit in my pipeline or not?

Also can you please confirm like is there other way we can do inside for each loop or not?
Pratik Somaiya 4,126 Reputation points

2022-03-28T05:42:22.35+00:00

Inside ForEach, you will have to pass FeedIds as input to the activity, for that I think Lookup is the best possible way
Nikunj Patel 51 Reputation points

2022-03-28T05:47:23.027+00:00

Yes Pratik, I tried same. Inside for each loop I can able to pass feed Id in lookup activity and I get Iscompleted flag value either 0 or 1. But how can I restrict the flow if I get Iscompleted flag value to 1 ? which activity or which component I can use in case of lookup return value Iscompleted 1.
Pratik Somaiya 4,126 Reputation points

2022-03-28T06:26:07.04+00:00

@Nikunj Patel :

In this case you can either use If activity or Switch activity

The flow would be as follows:
Nikunj Patel 51 Reputation points

2022-04-01T07:00:30.5+00:00

Instead of If condition I tired with filter but it wont work. Below screenshot shows my pipeline structure.
Also I cant used If condition as IF already used inside for each loop.

Run pipeline activity based on condition in ADF.

1 answer

Activity