How to improve data copy activity from Salesforce to ADLS?

Question

Trying to fetch Salesforce Attachment Bytearray data and copy into a Text delimited dataset in ADLS. It takes 40-45 mins to upload 1000 files in ADLS. How to improve the performance?

Answer 1

Hello @Mannur, Deepak, Thanks for posting this useful topic on this forum,

1. Increase the number of parallel copy activities:
2. Use a larger compute size for the integration runtime:
3. Use a staging area:
4. You can also use Azure Data Lake Storage Gen2 instead of ADLS to take advantage of its improved performance and scalability.
5. Compression can reduce the amount of data that needs to be transferred, while partitioning can help distribute the workload across multiple nodes.

**Copy is scalable at different levels:
**
Control flow can start multiple copy activities in parallel, for example using For Each loop.

A single copy activity can take advantage of scalable compute resources.

• When using Azure integration runtime (IR), you can specify up to 256 data integration units (DIUs) for each copy activity, in a serverless manner.
• When using self-hosted IR, you can take either of the following approaches:
• Manually scale up the machine.
• Scale out to multiple machines (up to 4 nodes), and a single copy activity will partition its file set across all nodes.

A single copy activity reads from and writes to the data store using multiple threads in parallel.

• 

Performance tuning steps:

Parallel copy

You can set the parallelCopies property to indicate the parallelism you want the copy activity to use. Think of this property as the maximum number of threads within the copy activity. The threads operate in parallel. The threads either read from your source, or write to your sink data stores. Learn more.

Troubleshoot copy activity performance:

I hope these suggestions help you improve the performance of uploading 1000 files to ADLS. Let me know if you have any further questions or concerns.

If the response is helpful, please click "Accept Answer" and upvote it. So that we can close this thread.