copy activity

Question

hello,

I use the copy activity to ingest a table from azure sql server 500 million rows into storage cloud...

This takes a long time and after 12 hours it times out.

Any suggestions how I resolve timing out?

Thanks

Answer 1

Hi @arkiboys

1. As mentioned in comment, you can increase the timeout of the copy acitivty which is by default 12 hours
2. you can try to increase parallel copies to increase number of threads that query the source in parallel. I recommend you going through the below page which discusses use cases where source table has physical partitions and otherwise and to know more on parallel copy.

https://learn.microsoft.com/en-us/azure/data-factory/connector-azure-sql-database?tabs=data-factory#parallel-copy-from-sql-database

hope it helps. Please let us know if you have any further question on it.

Answer 2

Hi @arkiboys ,

<In addtion to Suba's answer>

On the source end, please see if you can add an index to optimize your source dataset. and if you are using SQL query, try using filters(using where) to split the data into smaller chunks.

(or) see if you can able to increase the Azure SQL DTUs to improve the performance.

(or) see if you can able to increase the data integration units size to a different number to improve the performance.

The allowed DIUs to empower a copy activity run is between 2 and 256. If not specified or you choose "Auto" on the UI, the service dynamically applies the optimal DIU setting based on your source-sink pair and data pattern.

Please see this document for more details