Incremental Load in Azure Data Factory

Question

Hi,

Requirement: Fetch data from on-premise using ADF and load it into Synapse incrementally.

Problem: We don't have a unique value column to perform upsert taking that column as the key. There is a 'change_date' column indicating the last modified date, we can extract the data from the source based on that date but unable to figure out how to update the already inserted value in synapse which has been modified recently in the source.

Thank you.

Answer 1

Try implementing incremental data load using Change tracking method as explained here:

• https://learn.microsoft.com/en-us/azure/data-factory/tutorial-incremental-copy-change-tracking-feature-portal#overview

Similar thread

• https://stackoverflow.com/questions/61473405/incremental-load-without-date-or-primary-key-column-using-azure-data-factory

----------

Please don't forget to Accept Answer and Up-vote if the response helped -- Vaibhav

Answer 2

Hi @39861377 ,

Thankyou for using Microsoft Q&A platform and posting your query.

As per your query, it looks like you want to perform incremental load in destination table . The source dataset is having a watermark column called 'change_date' which stores the last modified date value.

For this requirement, I would suggest you to :

1. Use look up activity to fetch the max(change_date) value from target table .
2. Use Copy activity with dynamic query as 'Select * from <SourceTable> where change_date>lookup.output value' and copy the incremental data in stage table.
3. Write a stored procedure to :
   a. Delete from target table where change_date>=Change_date from stage table
   b. Insert all records from stage table to target table

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you.
  Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators