How to pick tables on the basis of batches?

Anmol Ganju 176 Reputation points
2022-07-11T06:40:49.683+00:00

Hi, I have a config table, where I have list of all tables specified with batches (I have a batch column also), we are having batch count till 5, how can I process all of these tables parallelly on the basis of this batch count, for example, if all the 5 batches have list of 20 tables, then batch should run sequentially but the table load should run in parallel, and pipeline should called as completed when all the 5 batches are complete loading data.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,208 questions
{count} votes

1 answer

Sort by: Most helpful
  1. MartinJaffer-MSFT 26,226 Reputation points
    2022-07-13T18:06:50.517+00:00

    Okay @Anmol Ganju . Here is another way to do it. This example (for brevity) does 3 batches, but you can see the pattern to extend to more.

    1. Lookup. Gets the table
    2. Lookup branches into a Filter activity for each batch, to isolate the records for the batch
    3. Each Filter activity has its own corresponding ForEach loop to do all the tables for that batch. This has the benefit of allowing you to fine-tune the exact amount of parallelism.
    4. There is a green on-success dependency from "ForEach Batch 1" to "ForEach Batch 2", and from "ForEach Batch 2" to "ForEach Batch 3". The ForEach only runs when the previous ForEach completes.
    5. Inside the ForEach is the Copy activity

    Picture below.

    220487-image.png


Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.