ADF error when unzipping file

John Aherne 141

I have seen this issue pop up here before, but no solutions.

We are downloading a zip file from SFTP, to ADLS gen 2 using a copy activity.
Then another copy activity unzips the file to a separate location.
This has been working for all files until yesterday. The zip file in question has one really large file (just over 25gb) and two smaller files - 812mb, and a few kb. The zip file itself is 1.8gb.

We are copying using a binary dataset for source and sink, with ZipDeflate.
The IR is autoresolve East US.

The two smaller files are extracted and look to be ok, but we get an error on the third file.

The error we are getting is:
Code: 9011
Failure happened on 'Source' side. ErrorCode=UserErrorFailedFileOperation,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Upload file failed at path path/xyz.txt.,Source=Microsoft.DataTransfer.Common,''Type=System.ArgumentOutOfRangeException,Message=Number was less than the array's lower bound in the first dimension.
Parameter name: srcIndex,Source=mscorlib,'

We have manually downloaded the zip file from Azure storage and unzipped it successfully on our desktop. We rezipped the files and uploaded to storage but got the same error again.

Is there an upper limit we are hitting with the capabilities of the copy activity?

Thanks!

ShaikMaheer-MSFT 38,441 Reputation points Microsoft Employee

2022-11-11T13:18:48.257+00:00

Hi @John Aherne ,

Thank you for posting query in Microsoft Q&A Platform.

Compression type of zip file usually will be root cause of this kind of error.

Could you please check below link where similar issue discussed and see which compression type is for your zip files and check its supported or not?
https://learn.microsoft.com/en-us/answers/questions/178276/pipeline-failed-to-extract-zip-file.html

Hope this helps. Please let me know how it goes. Thank you.
John Aherne 141 Reputation points

2022-11-11T14:36:07.7+00:00

The compression is regular zip.
As I said, it extracts the two smaller files, but fails on the large one.
We were able to unzip the files successfully on a windows desktop with only the built-in windows zip.
We also rezipped the files and uploaded them, only to get the same error.

Later we zipped only the large file and tried to extract it but got the same error.
Nitin Kumar 0 Reputation points

2023-07-08T06:52:40.96+00:00

Any updates on the above question?
John Aherne 141 Reputation points

2023-07-11T00:35:13.9466667+00:00

We ended up just unzipping the file manually and uploading that to be processed. Didn't find a workaround, but also didn't run into the issue since.
Nitin Kumar 0 Reputation points

2023-07-11T04:49:25.3466667+00:00

Yes, there seems to be a limitation in ADF even I couldn't find a way to unzip large files in copy stage of ADF. I had to use a python script (in ADF pipeline) to unzip the file (after copying the zip file to Gen2 through copy stage) for automation.

Share via

ADF error when unzipping file

Your answer