Run pipeline activity based on condition in ADF.

Nikunj Patel 51 Reputation points


Below screenshot shows my pipeline structure. The goal of my pipeline, get all list of feeds in for each activity (First screenshot) and run one by one all table feed inside for each activity and create feed file in data lake (Second Screenshot).



Also, We have one SQL table called FeedHistory which schema like below and it mainly logged each feed entry in this table.

FeedId (int)
JobId (int)
FeedStartdate (Datetime)
FeedEndDate (Datetime)
FeedName (varchar)
IsCompleted (Bit)

Inside For each loop we logged entry in table with Is completed =0 and Process End date is NULL. Once copy activity completed then we update that feed with completed to 1 and so on.

Sometime, we facing issue like Copy activity failed due to such reasons and we rerun the pipeline and this time it reprocess all the feeds again and take same amount of time.

To restrict this, my plan is check if feed already processed with isCompleted= 1 then we need to skip those feed inside for each loop. but How can I achieve this I don't know.(I tried lookup ,if condition but it not make sense to me) so can anyone help me or guide me to achieve this scenario.

Thank you in advance.

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
3,805 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
8,472 questions
0 comments No comments
{count} votes

1 answer

Sort by: Most helpful
  1. Pratik Somaiya 4,126 Reputation points

    Hello @Nikunj Patel

    You can use a lookup activity wherein you would add the query having a where clause to skip rows having isCompleted = 1 like a normal SQL query

    Then the output of the Lookup activity can be passed as an input to the ForEach

    Let me know of any queries