copy activity

arkiboys 9,646 Reputation points
2023-10-13T16:29:26.3333333+00:00

hello,

I use the copy activity to ingest a table from azure sql server 500 million rows into storage cloud...

This takes a long time and after 12 hours it times out.

Any suggestions how I resolve timing out?

Thanks

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,708 questions
{count} votes

Accepted answer
  1. Suba Balaji 11,186 Reputation points
    2023-10-14T14:56:28.84+00:00

    Hi @arkiboys

    1. As mentioned in comment, you can increase the timeout of the copy acitivty which is by default 12 hours
    2. you can try to increase parallel copies to increase number of threads that query the source in parallel. I recommend you going through the below page which discusses use cases where source table has physical partitions and otherwise and to know more on parallel copy.

    https://learn.microsoft.com/en-us/azure/data-factory/connector-azure-sql-database?tabs=data-factory#parallel-copy-from-sql-database

    hope it helps. Please let us know if you have any further question on it.

    1 person found this answer helpful.

1 additional answer

Sort by: Most helpful
  1. BhargavaGunnam-MSFT 27,081 Reputation points Microsoft Employee
    2023-10-17T20:01:04.8066667+00:00

    Hi @arkiboys ,

    <In addtion to Suba's answer>

    On the source end, please see if you can add an index to optimize your source dataset. and if you are using SQL query, try using filters(using where) to split the data into smaller chunks.

    (or) see if you can able to increase the Azure SQL DTUs to improve the performance.

    (or) see if you can able to increase the data integration units size to a different number to improve the performance.

    The allowed DIUs to empower a copy activity run is between 2 and 256. If not specified or you choose "Auto" on the UI, the service dynamically applies the optimal DIU setting based on your source-sink pair and data pattern.

    Please see this document for more details

    User's image

    0 comments No comments