COPY ACTIVITY - ADF

Question

Hi,

We are using ADF, to backup tables from aws oracle to blob and from there to azure synapse. For PoC, we took 5 tables and we used lookup, for each and inside for each used copy activity.

Among 5 tables, All the tables got completed within 10 mins. One table alone taking more time. The total no.of rows present in that table is 4565244 in that for 981000 rows, it is taking nearly 5 hours

Can anyone please suggest to speed up the copy process in adf

Answer 1

Hello @Anonymous

Are the rest tables having similar amount of data?

If you are using a Self-Hosted Integration Runtime then there are some factors which can slow the Copy performance, Viz.: Network Bandwidth at IR VM, number of Nodes configured in SHIR, RAM configuration for SHIR

When using self-hosted IR, you can take either of the following approaches:
a) Manually scale up the machine.
b) Scale out to multiple machines (up to 4 nodes), and a single copy activity will partition its file set across all nodes.

Answer 2

Thanks @Pratik Somaiya