Do incremental(delta) loads for multiple tables

AzeemK 516 Reputation points
2020-12-14T22:52:09.287+00:00

ADF have built in Template Delta_CopyfromDB using control table ( see attached screen shot)
This template is built in for one table delta load at a time , is there way to use this for multiple tables in DB also Parameters at bottom need to hard coded at run time , is there a way to make them dynamic so they can pick up changing directory every month like Data_Destination_Directory.48191-deltaloads.jpg

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,833 questions
{count} votes

Accepted answer
  1. Nasreen Akter 10,791 Reputation points
    2020-12-15T02:28:54.73+00:00

    Hi @AzeemK ,

    That is an interesting question.

    There are some options, I was thinking about:

    option#1: create an additional SQL table and save all the information there e.g., data_source_table_name, data_source_water_mark_column,..., valid_from_date, valid_to_date etc. Then for each row (i.e., each SQL table information) in the validTimePeriod (i.e., determine from the valid_from_date and valid_to_date), perform the DeltaLoadCopyActivity.

    option#2: for each table, schedule the pipeline at different time with hard-coded parameters. In this case, you might have to update the parameters in each month manually, if the values cannot be calculated by the Expression/function in the trigger.

    ----------

    Please let me know if this helps. If it does, please don't forget to "accept the answer" and "up-vote"! Thank you!

    3 people found this answer helpful.

0 additional answers

Sort by: Most helpful

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.