Azure Data Factory copy activity encounters out of memory error

Question

I am trying to copy data from SQL Server to a blob container using Azure Data Factory. However, the copy activity is failing with the following error:

ErrorCode=SystemErrorOutOfMemory,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=A task failed with out of memory.,Source=Microsoft.DataTransfer.TransferTask,''Type=System.OutOfMemoryException,Message=Exception of type 'System.OutOfMemoryException' was thrown.,Source=Microsoft.DataTransfer.ClientLibrary,'

Can someone please provide any guidance or assistance to resolve this issue? Thank you.

Answer 1

Hello Michael M,

It seems like you are getting "OutOfMemoryException" issue while using self-hosted IR

Please check the resource usage and concurrent-activity-execution on the IR node.

If there is not enough memory left to support this run, the error message is by design

Mitigation:

You will need to adjust the trigger time and interval of activity runs to prevent too many activities from executing on the same IR node simultaneously.

I hope this help. Please let me know if you have any further questions.