question

SeuliDasMajumder-3800 avatar image
0 Votes"
SeuliDasMajumder-3800 asked ChiragMishra-MSFT commented

how to replicate ongoing changes of on prem database to data lake?

Like AWS DMS handles ongoing replication and update bucket with respective files, what is available in azure to manage ongoing changes occurring at on-prem sql database?
Initially it requires to do a full load in my data lake storage which can be achieve by copy data functionality available in ADF.
Next it is required to capture all changes I have in respective CDC tables in SQL database. How to get those?

azure-data-factoryazure-functionsazure-data-lake-storageazure-migrateazure-database-migration
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

ChiragMishra-MSFT avatar image
0 Votes"
ChiragMishra-MSFT answered ChiragMishra-MSFT commented

Hi @SeuliDasMajumder-3800,

Once you have done a full load in your ADLS using copy activity, there are a couple of options to do an "Incremental Copy" listed out here :

https://docs.microsoft.com/en-us/azure/data-factory/tutorial-incremental-copy-overview

You can set up a pipeline that has an incremental copy setup using one of the options mentioned in the above doc. Additionally, you can set up a Tumbling window trigger to run at a fixed frequency to copy the incremental records :

https://docs.microsoft.com/en-us/azure/data-factory/how-to-create-tumbling-window-trigger

Hope this helps.

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.


Hi @SeuliDasMajumder-3800,

Just wanted to check - was the above suggestion helpful to you? If yes, please consider accepting it as answer. This would help other community members reading this thread.

1 Vote 1 ·