What compression type should we choose to unzip .gz files using Copy activity on ADF?

Question

I'm trying to use ADF Copy Activity to unzip .gz files I have on my ADLS Gen2, which compression type should we choose from the drop-down list below? All of them failed to work for me.

Answer 1

Hi

Thanks for reaching out to Microsoft Q&A.

Your source files are zipped from windows or non-windows machine? ADF has limited support when it comes to unzipping in the copy activity. I had a similar issue in the past where I raised a case with support. This is the reply I got from them...

• User’s zip file was compressed by the algorithm of “Zipdeflate(.zip)” or “deflate64”, while our internal zip library only supports “deflate”.
• If the zip file is compressed by windows system, and the overall file size exceeds a certain number (maybe 2 GB I don’t remember clearly), windows will use deflate64 by default, resulting in not supported by ADF. On the other hand, if the file size is smaller, or use some 3rd party zip tools which supports specifying the compress algorithm, it will be “deflate”.

More detail: Actually, both public docs  https://docs.microsoft.com/en-us/azure/data-factory/format-binary  and  https://docs.microsoft.com/en-us/azure/data-factory/format-delimited-text  clearly say we do not support

I came through this issue by using the following workaround which worked for me.

• Create a copy activity to binary load the data to a staged blob storage without any compression.
• Once the files are moved to staging storage account, run another copy activity following the former one, to move it to blob to final blob/ADLS copy with compression type.

I suggest you to give this a try & let me know.

Please Upvote and Accept as answer if the reply was helpful, this will be helpful to other community members.