You should have a column to track changes, and you can set up Change Data Capture (CDC) for your process.
Loading huge tables , how to load only new rows
Lisa Kruger
60
Reputation points
Hi All,
I have to load tables between two Azure SQL Databases. Both are on different servers. I have created a pipeline to bulk load the tables between the databases. The issue here is I have around 7 big tables and , currently the pre copy script is TRUNCATE and INSERT. I have excluded these 7 tables from the bulk load process. My question is for example I have a table (product) in the source with 32 million rows and next day it has 32.2 million rows, is there a way to just load the difference (200K rows), please advice.
Thanks