how to replicate ongoing changes of on prem database to data lake?

Question

Like AWS DMS handles ongoing replication and update bucket with respective files, what is available in azure to manage ongoing changes occurring at on-prem sql database?
Initially it requires to do a full load in my data lake storage which can be achieve by copy data functionality available in ADF.
Next it is required to capture all changes I have in respective CDC tables in SQL database. How to get those?

Answer 1

Hi @Seuli Das Majumder ,

Once you have done a full load in your ADLS using copy activity, there are a couple of options to do an "Incremental Copy" listed out here :

https://learn.microsoft.com/en-us/azure/data-factory/tutorial-incremental-copy-overview

You can set up a pipeline that has an incremental copy setup using one of the options mentioned in the above doc. Additionally, you can set up a Tumbling window trigger to run at a fixed frequency to copy the incremental records :

https://learn.microsoft.com/en-us/azure/data-factory/how-to-create-tumbling-window-trigger

Hope this helps.