question

AmitSrivastava-4935 avatar image
0 Votes"
AmitSrivastava-4935 asked MaheshKumar-1184 edited

ADF fails to unzip files

Hi,
I have Zip files on a SFTP location. The zip files contain single CSV. The zip files are approx 80 MB in size. I need to ingest that through ADF , unzip and save the csv files on ADLS. I am using the Zip deflate option in my source dataset. Compression type on sink is set as None.

However I get the error

"errorCode": "2200", "message": "Failure happened on 'Sink' side. ErrorCode=UserErrorUnzipInvalidFile,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The file 'XXXXXX.zip' is not a valid Zip file with Deflate compression method.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.IO.InvalidDataException,Message=End of Central Directory record could not be found.,Source=Microsoft.DataTransfer.ClientLibrary,'", "failureType": "UserError", "target": "XXXXXX", "details": []

When I use a smaller version of the zip files~5KB (after reducing the no of rows in the CSV), the same settings work absolutely fine.

Assuming that may be there are some zipping issues in my source zip files, I also tried with another zip file of about 220 MB which I got from somewhere else. Again got the same error.

Anyone can help me with this ?

Thanks,
Amit

azure-data-factory
· 9
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @AmitSrivastava-4935,

Welcome to Microsoft Q&A community forum and thanks for reaching out.

To better assist on your issue, could you please share below details for further investigation?

  • Failed pipeline run ID?

  • What is your data factory region?

  • What is the IR used? If SHIR, what is the version?

  • Please do share the pipeline run ID which ran successfully with small file size?

Looking forward to your response.

Thank you



0 Votes 0 ·

Failed pipeline: 824c0b6c-49ee-41c8-af17-825d3fddea84
Successful pipeline: e8378b70-e63e-40b7-b3eb-460ce0e3a004
Data Factory Region: US West 2
IR: AutoResolve

Thanks,
Amit

0 Votes 0 ·

Thanks for the details @AmitSrivastava-4935. I have escalated this to internal and will get back to you as soon as I have an update from the team.

Thank you for your patience.

0 Votes 0 ·

Hi @AmitSrivastava-4935,

Just wanted to let you know that I'm still waiting for a response form internal team. Will update this thread once I have a response from the team.

We appreciate your patience.

0 Votes 0 ·
Show more comments

1 Answer

MartinJaffer-MSFT avatar image
0 Votes"
MartinJaffer-MSFT answered MaheshKumar-1184 edited

Reviewing the support case, it seems the file was corrupted or format changed as it moved thru the SFTP.

For anyone reading this and seeking help, here are recommended steps to help locate the point-of-failure.


Manually download and unzip / decompress the file , at each place it travels through.
In this case, it would be download from blob and unzip, download from SFTP and unzip. If either fails, then you know the file is not in the correct format at that location. If both succeed, yet Data Factory fails, then it may be an issue with Data Factory. Please let us know if the last one is the case.

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @MartinJaffer-MSFT,

I am facing a similar issue where a file which is around 21 MB has failed while unzipping. The rest all files ranging 130 KB to 2.7 MB have got unzipped perfectly. This 21 MB file, when I download to my local machine, I am able unzip it without any issues.

Attaching screenshots for your reference...

The screenshot "Error_13Apr" shows the error message.


The screenshot "File List_13Apr" shows the list of the files (which are less than 21 MB) that have succeeded and the file (which is around 21 MB) that has failed.


Please advise.
Thanks.

Regards,
Mahesh


[1]: /answers/storage/attachments/87315-error-13apr.png

[2]: /answers/storage/attachments/87307-file-list-13apr.png

0 Votes 0 ·
error-13apr.png (17.1 KiB)
file-list-13apr.png (156.1 KiB)