ADF fails to unzip files

Amit Srivastava 6

Hi,
I have Zip files on a SFTP location. The zip files contain single CSV. The zip files are approx 80 MB in size. I need to ingest that through ADF , unzip and save the csv files on ADLS. I am using the Zip deflate option in my source dataset. Compression type on sink is set as None.

However I get the error

"errorCode": "2200", "message": "Failure happened on 'Sink' side. ErrorCode=UserErrorUnzipInvalidFile,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The file 'XXXXXX.zip' is not a valid Zip file with Deflate compression method.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.IO.InvalidDataException,Message=End of Central Directory record could not be found.,Source=Microsoft.DataTransfer.ClientLibrary,'", "failureType": "UserError", "target": "XXXXXX", "details": []

When I use a smaller version of the zip files~5KB (after reducing the no of rows in the CSV), the same settings work absolutely fine.

Assuming that may be there are some zipping issues in my source zip files, I also tried with another zip file of about 220 MB which I got from somewhere else. Again got the same error.

Anyone can help me with this ?

Thanks,
Amit

KranthiPakala-MSFT 46,512 Reputation points Microsoft Employee

2020-09-16T19:59:25.077+00:00
Hi @Amit Srivastava ,

Welcome to Microsoft Q&A community forum and thanks for reaching out.

To better assist on your issue, could you please share below details for further investigation?

Failed pipeline run ID?

What is your data factory region?

What is the IR used? If SHIR, what is the version?

Please do share the pipeline run ID which ran successfully with small file size?

Looking forward to your response.

Thank you
Amit Srivastava 6 Reputation points

2020-09-16T20:46:06.04+00:00

Failed pipeline: 824c0b6c-49ee-41c8-af17-825d3fddea84
Successful pipeline: e8378b70-e63e-40b7-b3eb-460ce0e3a004
Data Factory Region: US West 2
IR: AutoResolve

Thanks,
Amit
KranthiPakala-MSFT 46,512 Reputation points Microsoft Employee

2020-09-16T21:53:08.753+00:00

Thanks for the details @Amit Srivastava . I have escalated this to internal and will get back to you as soon as I have an update from the team.

Thank you for your patience.
KranthiPakala-MSFT 46,512 Reputation points Microsoft Employee

2020-09-19T00:03:54.213+00:00

Hi @Amit Srivastava ,

Just wanted to let you know that I'm still waiting for a response form internal team. Will update this thread once I have a response from the team.

We appreciate your patience.
Amit Srivastava 6 Reputation points

2020-09-21T05:44:35.047+00:00

Pleae let me know if there is any update.
Regards,
Amit
KranthiPakala-MSFT 46,512 Reputation points Microsoft Employee

2020-09-21T20:29:10.747+00:00

Hi @Amit Srivastava ,

Thank you for your patience. Just heard back from internal team. Since the failed activity and succeeded activity are copying different files, it's hard to tell if this is a file format/compression issue or SFTP connector issue. Hence this needs a deeper analysis. For deeper investigation and immediate assistance, If you have a support plan you may file a support ticket, else could you please send an email to AzCommunity[at]Microsoft[dot]com with the below details, so that we can enable a one-time-free support for you to work closely on this issue.

Email subject: <Attn - Kranthi : Microsoft Q&A Thread title>
Thread URL: <Microsoft Q&A Thread>
Subscription ID: <your subscription id>

Please let me know here once the email is sent

Thank you
KranthiPakala-MSFT 46,512 Reputation points Microsoft Employee

2020-09-22T19:22:42.387+00:00

Hi @Amit Srivastava ,

Following up to see if you have got a chance to file a SR, if you don't have a support plan, please do email the details as requested in my previous response.

Let me know here once the email is sent.

Thank you
Amit Srivastava 6 Reputation points

2020-09-23T10:59:07.007+00:00

Hi,

I have sent an e-mail as required.

Thanks,
Amit
KranthiPakala-MSFT 46,512 Reputation points Microsoft Employee

2020-09-24T06:44:34.59+00:00

Thanks for sharing the details Amit. Please file a support request as per the instruction provided in the email and do share the SR# once you file a support request so that I can track this case internally.

Thank you

1 answer

MartinJaffer-MSFT 26,091 Reputation points

2020-10-26T21:47:27.467+00:00

Reviewing the support case, it seems the file was corrupted or format changed as it moved thru the SFTP.

For anyone reading this and seeking help, here are recommended steps to help locate the point-of-failure.

Manually download and unzip / decompress the file , at each place it travels through.
In this case, it would be download from blob and unzip, download from SFTP and unzip. If either fails, then you know the file is not in the correct format at that location. If both succeed, yet Data Factory fails, then it may be an issue with Data Factory. Please let us know if the last one is the case.
Please sign in to rate this answer.
Mahesh Kumar 106 Reputation points

2021-04-13T09:20:52.407+00:00

Hi @MartinJaffer-MSFT ,

I am facing a similar issue where a file which is around 21 MB has failed while unzipping. The rest all files ranging 130 KB to 2.7 MB have got unzipped perfectly. This 21 MB file, when I download to my local machine, I am able unzip it without any issues.

Attaching screenshots for your reference...

The screenshot "Error_13Apr" shows the error message.

The screenshot "File List_13Apr" shows the list of the files (which are less than 21 MB) that have succeeded and the file (which is around 21 MB) that has failed.

Please advise.
Thanks.

Regards,
Mahesh

Dr. Alexei Vasilevich 0 Reputation points

2023-02-01T08:35:10.3633333+00:00

Hi, I just had the same issue, and I'm able to download the file from blob storage and unzip it.

Alexander Littau 0 Reputation points

2023-05-17T12:36:10.7466667+00:00

Hi, I have the same issue :(.

I copy a 5GiB+ file from a sftp site to ADL gen2. I have tried to decompress files when copying from sftp to ADL and got the same error. I then tried to copy the file to ADL without decompression, and then made another ADL to ADL copy with decompress (Zipdeflate). Same error.

I can download the file from ADL to a VM and decompress without errors.

Please advice....
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

ADF fails to unzip files

1 answer

Your answer