Azure Data Factory - Change .zip to .gz on the fly using ADF pipeline

Question

Hi Guys,

We have a requirement to load .zip files from amazon s3 to DWH. As a first step we want to convert these into .gz files so that it could be picked by polybase to be loaded in DWH.

Source dataset is set to compression type = ZipDeflate
Sink dataset is set to compression type = gzip

On debugging the pipeline, we are getting the following error.

{ "errorCode": "2200", "message": "Failure happened on 'Sink' side. ErrorCode=UserErrorUnzipInvalidFile,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The
file 'file1.zip' is not a valid Zip file with Deflate compression method.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.IO.InvalidDataException,Message=End of Central Directory record could not be found.,Source=Microsoft.DataTransfer.ClientLibrary,'",
"failureType": "UserError", "target": "S3toBlob", "details": [] }

We have tried some other compression types as well, for no luck.

Anyone tried this before ?

Regards

Lokesh

Answer 1

Exact similar question is posted here in correct forum. Please follow up there -

https://social.msdn.microsoft.com/Forums/en-US/84b4cc7e-0ff8-42f4-b3af-d7f35eecc393/convert-zip-file-to-gz-on-the-fly?forum=AzureDataFactory