ErrorCode=SystemErrorOutOfMemory getting while processing file

Question

ErrorCode=SystemErrorOutOfMemory getting while processing file

Madasi, Srinivas 21

Error details
Error code1000
Troubleshooting guide
Failure typeSystem error
DetailsFailure happened on 'Sink' side. ErrorCode=SystemErrorOutOfMemory,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=A task failed with out of memory.,Source=Microsoft.DataTransfer.TransferTask,''Type=System.OutOfMemoryException,Message=Array dimensions exceeded supported range.,Source=System.Core,'

I'm using 2 copy datas one is to insert the CSV(blob) into SQL and the other is to export from SQL to blob storage.

file size is aprox 200 MB and record count is 400 K.

run id: fd597d6b-da0a-4ec9-8ef6-7c00d0705388

KranthiPakala-MSFT 46,827 Reputation points Moderator

2021-01-21T02:36:27.787+00:00

Hi @Madasi, Srinivas ,

Welcome to Microsoft Q&A forum and thanks for your query.

Could you please confirm which copy activity is throwing this error? Is it Copy from CSV to SQL or Export from SQL to Blob? And are you experiencing this error message consistently or intermittent?

Also please do share the failed Activity runID as well as the failed pipeline runID for further analysis.

Looking forward to your response.

Thanks.
KranthiPakala-MSFT 46,827 Reputation points Moderator

2021-01-21T23:30:01.717+00:00

Hi @Madasi, Srinivas ,

Following up to see if you have got a chance to see my previous comment? If so could you please share few additional details as requested to better assist on this issue.

Thanks
Madasi, Srinivas 21 Reputation points

2021-01-22T10:16:12.087+00:00

Hi Kranthi, Thanks for your reply.

This happening consistently in the copy data activity where I'm exporting to Blob. Up to some 200K records, I don't see any issues, when I increase it to 400K started seeing this issues.

Failed pipeline run id: 4582bdb1-f589-4cec-8772-eae6cdc1b5af
Failed Activity run id: 837409a9-22dc-44d2-94a7-e5a7a5f6f1a7

Regards,
Srinivas
KranthiPakala-MSFT 46,827 Reputation points Moderator

2021-01-26T00:18:14.807+00:00

Hi @Madasi, Srinivas , sorry for the delay in my response and thanks for your response, additional details. We will look into the runID's provided and will get back as soon as we have an update.

We appreciate your patience.

Thank you
KranthiPakala-MSFT 46,827 Reputation points Moderator

2021-01-26T01:54:20.23+00:00

Hi @Madasi, Srinivas ,

Could you please confirm if you have set DIU under copy settings to Auto or any particular number? If you have set it to a number, could you please try increasing the DIU or changing it to Auto and see if that helps? As per the error message your issue is related to CPU, memory, and network resource allocation.

Note: Please note that you will be charged # of used DIUs * copy duration * unit price/DIU-hour. See the current prices here

Let us know how it goes.
KranthiPakala-MSFT 46,827 Reputation points Moderator

2021-01-26T22:57:41.727+00:00

Hi @Madasi, Srinivas ,

Following up to see if you have got a chance to try the above suggestion. If so, please let us know how it goes.

Thanks
Madasi, Srinivas 21 Reputation points

2021-01-27T07:53:46.08+00:00

Hi Kranthi, Thanks for the suggestion and follow up.

Earlier it was mentioned "Auto", I changed it to "32" which is max number showing up for me in the drop down and tested it, still failing with same error.

Regards,
Srinivas
KranthiPakala-MSFT 46,827 Reputation points Moderator

2021-01-27T19:11:26.137+00:00
Hi @Madasi, Srinivas ,

Thanks for the update. I am reaching out to internal team to get additional inputs on this issue. In the meantime could you please check if you have set a value for Max concurrent connections? If yes, then please remove the setting of Max concurrent connections to make the below 3 properties as default and let us know how it goes.

DIUs

ParallelCopies

Max concurrent connections

And also please tell us what is the file format you are trying to write to your blob?

Thanks
robo3459 1 Reputation point

2021-01-28T00:07:33.03+00:00

Hi, same issue here when copying about 1GB of data using the Salesforce source to Azure blob storage sink JSON file. Copy works fine for smaller amounts of data from Salesforce. Default settings on the pipeline (see screenshot). Same error with DIU's set to 32. Details imply the 1 GB was pulled from Salesforce ok. Please advise.

Run ID: b303acc2-561d-4b60-aa9f-b118fddcbde7

Failure happened on 'Sink' side. ErrorCode=SystemErrorOutOfMemory,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=A task failed with out of memory.,Source=Microsoft.DataTransfer.TransferTask,''Type=System.OutOfMemoryException,Message=Exception of type 'System.OutOfMemoryException' was thrown.,Source=mscorlib,'
KranthiPakala-MSFT 46,827 Reputation points Moderator

2021-01-28T19:47:32.973+00:00
Hi @Madasi, Srinivas ,

After having conversation with internal team. Below are findings and workaround to overcome this issue.

One possible root cause could be you might have a very big row, runtime will try to allocate very big array to store the row string while serialize the row to CSV, that will hit the limit (https://learn.microsoft.com/en-us/dotnet/framework/configure-apps/file-schema/runtime/gcallowverylargeobjects-element).

To work around this issue, it is recommended to run the copy activity via SHIR:

In the SHIR installation folder 'Microsoft Integration Runtime{version}\Shared\diawp.exe.config', find a file with name 'diawp.exe.config'

<gcAllowVeryLargeObjects enabled="true" /> as a child XML node of runtime XML node

Restart the self-hosted IR service

Run the copy activity via the SHIR by change the linked service of source or sink to the SHIR
KranthiPakala-MSFT 46,827 Reputation points Moderator

2021-01-28T19:56:17.04+00:00

Continuation to above comment:

@Madasi, Srinivas , @robo3459 - Could you please try the above workaround provided and let us know if that helps to resolve your issue.

@robo3459 - If the workaround didn't resolve your issue, I would recommend to open a separate new thread as your requirement seems different. It also help us to track this issue separate and also helps other community members to pitch in if they have experienced similar situation.

Please let us know how it goes with SHIR. We look forward to your confirmation.

Thank you
Madasi, Srinivas 21 Reputation points

2021-01-29T11:54:15.047+00:00

Hi Kranthi,

Thanks for this workaround, I will try and let you know if this is working.

Regards,
Srinivas
KranthiPakala-MSFT 46,827 Reputation points Moderator

2021-01-29T19:26:43.213+00:00

Thank you @Madasi, Srinivas . Looking forward to your confirmation.
Madasi, Srinivas 21 Reputation points

2021-02-01T05:58:43.15+00:00

Hi Kranthi,

Due to some admin related restriction we are not able to proceed with this SHIR installation at the moment, we keep it n HOLD for sometime. I will update this thread once the restriction removed.

Thanks for your time and help!

Regards,
Srinivas
KranthiPakala-MSFT 46,827 Reputation points Moderator

2021-02-02T00:09:48.103+00:00

Sure, @Madasi, Srinivas . Please keep this thread updated as soon as you have your findings.

Take care!
Cathy Lee 26 Reputation points

2021-08-04T14:08:17.16+00:00

@KranthiPakala-MSFT we are having the same issue and I'm wondering if the modification of the config file applies to updating Azure IR's or just on prem IR's?
Edwin Bundi 0 Reputation points

2024-09-30T09:49:32.04+00:00

ErrorCode=SystemErrorOutOfMemory,''Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=A task failed with out of memory.,Source=Microsoft.DataTransfer.TransferTask,''''Type=System.OutOfMemoryException,Message=Exception of type ''System.OutOfMemoryException'' was thrown.,Source=Microsoft.DataTransfer.DataContracts,'' i got this whoile migrating to cloud from on prem