Data Migration through ADF while the application is running

Question

Data Migration through ADF while the application is running

Rahul 251

Hi,

I am planning to perform the Data Migration from oracle (on-prem) to azure sql database.

Database Size- 1.5 TB.

Data Migration Throughput Speed- 2 MBps.

Total Tables-229.

Big Size Tables-7

Database Size of Biggest table- 500 GB.

I wanted to migrate the data while the application is on, as currently the data migration of 1.5Tb is taking 100 hr.

Question-1 - I would be migrating the data each table wise, and some big tables have 1 billion+ records, the application would be on.

Will data migration while application is on impact the performance of the source oracle database ?

ADF only consumes read operation of each table while the data migration on oracle prod database?

Does ADF reads approx. 2000+internally at a time while reading big tables with 1+ million records as shown on ADF copy activity UI?

Details:

Copy Activity- It is using just select statement from respective tables while migration.

I would be migrating one table at a time.

Big Tables- For 1 + million records tables , I observed ADF reads approx. 2000+ records at a time while performing the data migration.

Question-2 -Since the tables are getting updated while it is connected to the application.

Will ADF migrates the current data at a point while running migration pipeline, while application will perform its insert,update,delete on the oracle prod database without interruption ?

FYI- All insert , updates and delete happening in application while data migration , i will handle later on through upsert and delete detection adf pipeline I build?

Please answer the above as we are planning to perform the data migration in PROD database while the application is connected on to the prod database ?

0 comments

Answer accepted by question author

Vinodh247-1375 43,181 Volunteer Moderator

Hi,

Thanks for reaching out to Microsoft Q&A.

only read action will be performed in the source table hence there will be little or no impact on source database tables.

Will ADF migrates the current data at a point while running migration pipeline, while application will perform its insert, update, delete on the oracle prod database without interruption?

In case you haven't, please go through the below migration guide to validate and implement the best practices for moving data between oracle on-prem to azure sql.

https://learn.microsoft.com/en-us/azure/azure-sql/migration-guides/database/oracle-to-sql-database-guide?view=azuresql

Please Upvote and Accept as answer if the reply was helpful, this will be benefitting the other community members who go through the same issue.

Rahul 251 Reputation points

2023-07-12T17:44:13.3166667+00:00

Hi,

Please help with the below query.

Will ADF migrates the current data at a point while running migration pipeline, while application will perform its insert, update, delete on the oracle prod database without interruption?

I went through the below mentioned document , but somehow need more details regarding the any updates happening in database and handling it while the data migration.
Vinodh247-1375 43,181 Reputation points Volunteer Moderator

2023-07-13T05:01:36.2666667+00:00

Will ADF migrates the current data at a point while running migration pipeline,

The copy will pick the data from source right until the point of pipeline trigger initiation.

while application will perform its insert, update, delete on the oracle prod database without interruption?

The data insert/update/delete that are happening once the pipeline trigger has started can be copied only in the next iteration, will not be included in the current pipeline trigger that has already started and running.

There will be no interruption to any of the operations (insert,update,delete) that you will be doing in the source.

1 additional answer

Your answer

Rahul 251 Reputation points

2023-07-12T17:44:13.3166667+00:00

Hi,

Please help with the below query.

Will ADF migrates the current data at a point while running migration pipeline, while application will perform its insert, update, delete on the oracle prod database without interruption?

I went through the below mentioned document , but somehow need more details regarding the any updates happening in database and handling it while the data migration.
Vinodh247-1375 43,181 Reputation points Volunteer Moderator

2023-07-13T05:01:36.2666667+00:00

Will ADF migrates the current data at a point while running migration pipeline,

The copy will pick the data from source right until the point of pipeline trigger initiation.

while application will perform its insert, update, delete on the oracle prod database without interruption?

The data insert/update/delete that are happening once the pipeline trigger has started can be copied only in the next iteration, will not be included in the current pipeline trigger that has already started and running.

There will be no interruption to any of the operations (insert,update,delete) that you will be doing in the source.

Answer 1

Rahul 251

Hi,

Thanks.

I went through the specified link but need more details regarding below please.

Will ADF migrates the current data at a point while running migration pipeline, while application will perform its insert, update, delete on the oracle prod database without interruption?

Vinodh247-1375 43,181 Reputation points Volunteer Moderator

2023-07-13T16:03:31.44+00:00

@Kashish : I have responded in a comment above, I have copied my input again here. Pls go through and let me know if you still have any questions.

Will ADF migrates the current data at a point while running migration pipeline,

The copy will pick the data from source right until the point of pipeline trigger initiation.

while application will perform its insert, update, delete on the oracle prod database without interruption?

The data insert/update/delete that are happening once the pipeline trigger has started can be copied only in the next iteration, will not be included in the current pipeline trigger that has already started and running.

There will be no interruption to any of the operations (insert,update,delete) that you will be doing in the source.
Vinodh247-1375 43,181 Reputation points Volunteer Moderator

2023-07-15T09:02:13.29+00:00

Just checking in to see if the above answer helped. Please do consider clicking 'Accept Answer' and 'Up-Vote' for the same as accepted answers help other users in community as well.
Vinodh247-1375 43,181 Reputation points Volunteer Moderator

2023-07-19T09:52:43.07+00:00

Kashish: Pls confirm if we are still connected. If you did found any resolution, please share the same here which will help the community.
Rahul 251 Reputation points

2023-07-23T18:03:06.05+00:00

@Vinodh247-1375 Thanks above solution works.

Share via

Data Migration through ADF while the application is running

1 additional answer

Your answer