Copy activity failing with SystemErrorOutofMemory

Question

Hi, I have a pipeline which is executing copy activity, copying data from SQL server and storing to a blob container. It is failing with error -

ErrorCode=SystemErrorOutOfMemory,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=A task failed with out of memory.,Source=Microsoft.DataTransfer.TransferTask,''Type=System.OutOfMemoryException,Message=Exception of type 'System.OutOfMemoryException' was thrown.,Source=Microsoft.DataTransfer.ClientLibrary,'

Pipeline Run ID - 41f235b8-000e-4709-8aaa-57f9fdc6fbe7

Activity ID - 6c7e63b1-df04-4524-bdf2-a692d95abc8a

Answer 1

Based on this old thread, the product team suggests using Mapping Data Flow instead of Copy Activity to overcome this limitation. Data Flow allows you to move thousands of large Excel files to another data store efficiently. It supports streaming reads with minimal CPU and memory usage. Internal performance tests show that Data Flow can transfer a 1GB Excel file within 5 minutes using Azure Integration Runtime with 32 cores.

If you have specific feedback regarding the limitations of Copy Activity for copying large Excel files, please log your feedback on the Azure feedback forum here: Azure Feedback Forum.

Additionally, you can view and upvote an existing feedback request on this issue here: Existing Feedback. Your input will help prioritize this request.

Answer 2

Hello Mayank Gupta,

My understaning is you are using self-hosted IR while meets "OutOfMemoryException" issue.

Please check the resource usage and concurrent-activity-execution on the IR node.

If there is no enough memory left to support this run, the error message is by design

Mitigation:

You will need to adjust the trigger time and interval of activity runs to prevent too many activities from executing on the same IR node simultaneously.

I hope this help.