Loading huge tables , how to load only new rows

Lisa Kruger 60 Reputation points
2023-07-17T16:35:24.6166667+00:00

Hi All,

I have to load tables between two Azure SQL Databases. Both are on different servers. I have created a pipeline to bulk load the tables between the databases. The issue here is I have around 7 big tables and , currently the pre copy script is TRUNCATE and INSERT. I have excluded these 7 tables from the bulk load process. My question is for example I have a table (product) in the source with 32 million rows and next day it has 32.2 million rows, is there a way to just load the difference (200K rows), please advice.

Thanks

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,215 questions
{count} votes

1 answer

Sort by: Most helpful
  1. Vahid Ghafarpour 22,425 Reputation points
    2023-07-17T19:04:47.02+00:00

    You should have a column to track changes, and you can set up Change Data Capture (CDC) for your process.

    https://learn.microsoft.com/en-us/sql/relational-databases/track-changes/about-change-data-capture-sql-server?view=sql-server-ver16


Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.