Azure Datafactory Out of memory error when read from Salesforce

Olga Valiuk 0

I use Datafactory Copy activity to copy data from Salesforce to ADLS, I am facing with Out of memory error. The file size is 129k of rows 800MB, then I set block size to 100 MB and Max rows per file to 100 000, but the error still exist. What can you suggest?
1.jpg
2.jpg

3.jpg
4.jpg

Harishga 3,505 Reputation points Microsoft Vendor

2024-04-05T10:29:12.7333333+00:00
Hi @Olga Valiuk

Welcome to Microsoft Q&A platform and thanks for posting your question here.

The error message you are receiving states that there is an issue with the Java Virtual Machine memory allocation when copying data from Salesforce to ADLS using Data Factory Copy activity. The error message specifically mentions a "ParquetJavaInvocationException" error, which is caused by insufficient memory allocation for a task in a Java environment.

To resolve this issue, you can try adjusting the JVM settings to allocate more memory or optimize your data processing code to use memory more efficiently. Here are some suggestions:

Increase the Memory Allocation: This suggestion involves adjusting the JVM settings to allocate more memory to the Data Factory pipeline or Self-Hosted Integration Runtime. The JVM is the environment in which Java programs run, and it has a default memory allocation that may not be sufficient for large data processing tasks. By setting the "-Xmx" flag in the JVM options, you can allocate more memory to the JVM.

Optimize Data Processing Code: I recommend optimizing your data processing code to use memory more efficiently. This can be done by processing smaller amounts of data at a time, using more efficient ways to store data, or improving the way your code works.

Use Parallel Copy:
Parallel Copy is a technique used to optimize data processing code and resolve Out of Memory errors when copying large amounts of data. It splits the data into smaller chunks and copies them in parallel, reducing the amount of memory required for the operation.

Use a Different Integration Runtime:
This suggestion involves using a different Integration Runtime that has more memory or is optimized for big data workloads.

https://learn.microsoft.com/en-us/archive/msdn-technet-forums/7621a34d-c822-4cfc-9dae-91ab5e5b92be
https://learn.microsoft.com/en-us/azure/data-factory/copy-activity-performance

I hope this information helps you. Let me know if you have any further questions or concerns.
Harishga 3,505 Reputation points Microsoft Vendor

2024-04-08T02:39:17.2433333+00:00

@Olga Valiuk
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Olga Valiuk 0 Reputation points

2024-04-12T16:17:30.88+00:00

@Harishga Thank you for the response!

I tried parallel copy, but it didn't help.

Sure, I can manage data in smaller batches, but sometimes I need to process a large amount of data.

I also see that the duration does not exceed 3: 30 min, not sure where I can configure it. Because Timeout for copy activity is 7 hours.
Harishga 3,505 Reputation points Microsoft Vendor

2024-04-15T11:57:06.04+00:00

Hi @Olga Valiuk
If you need to process a large amount of data, you will consider using a different Integration Runtime that is designed to handle big data workloads. This can help avoid Out of Memory errors and ensure that your data processing tasks run smoothly.