Below screenshot shows my pipeline structure. The goal of my pipeline, get all list of feeds in for each activity (First screenshot) and run one by one all table feed inside for each activity and create feed file in data lake (Second Screenshot).
Also, We have one SQL table called FeedHistory which schema like below and it mainly logged each feed entry in this table.
Inside For each loop we logged entry in table with Is completed =0 and Process End date is NULL. Once copy activity completed then we update that feed with completed to 1 and so on.
Sometime, we facing issue like Copy activity failed due to such reasons and we rerun the pipeline and this time it reprocess all the feeds again and take same amount of time.
To restrict this, my plan is check if feed already processed with isCompleted= 1 then we need to skip those feed inside for each loop. but How can I achieve this I don't know.(I tried lookup ,if condition but it not make sense to me) so can anyone help me or guide me to achieve this scenario.
Thank you in advance.