Decompress a tar.Z file in Azure Data Factory

Question

Decompress a tar.Z file in Azure Data Factory

Shiv Kumar 40

I am using ADF and I have a source file available on ADLS locations and file extension is '.tar.Z', How i can decompress. Z file in Azure Data Factory.

Sally Dabbah 5 Reputation points Microsoft Employee

2023-02-16T14:01:29.5233333+00:00

you can unzip it using copy activity like so:
in source: compression type: select 'ZipDeflate'
in sink: copy behavior -> 'flatten hierarchy'
Shiv Kumar 40 Reputation points

2023-03-01T09:54:36.8133333+00:00

@Sally Dabbah , I tried but getting a error.

"errorCode": "2200", "message": "ErrorCode=UserErrorUnzipInvalidFile,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The file 'abc.tar.Z' is not a valid Zip file with Deflate compression method.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.IO.InvalidDataException,Message=End of Central Directory record could not be found.,Source=Microsoft.DataTransfer.ClientLibrary,'", "failureType": "UserError", "target": "cpy_file_raw_to_unzippedArchive_adls", "details": [] }
Shiv Kumar 40 Reputation points

2023-03-01T09:54:48.75+00:00

@Sally Dabbah , I tried but get a error.
"errorCode": "2200", "message": "ErrorCode=UserErrorUnzipInvalidFile,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The file 'abc.tar.Z' is not a valid Zip file with Deflate compression method.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.IO.InvalidDataException,Message=End of Central Directory record could not be found.,Source=Microsoft.DataTransfer.ClientLibrary,'", "failureType": "UserError", "target": "cpy_file_raw_to_unzippedArchive_adls", "details": [] }

1 answer

Your answer

Sally Dabbah 5 Reputation points Microsoft Employee

2023-02-16T14:01:29.5233333+00:00

you can unzip it using copy activity like so:
in source: compression type: select 'ZipDeflate'
in sink: copy behavior -> 'flatten hierarchy'
Shiv Kumar 40 Reputation points

2023-03-01T09:54:36.8133333+00:00

@Sally Dabbah , I tried but getting a error.

"errorCode": "2200", "message": "ErrorCode=UserErrorUnzipInvalidFile,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The file 'abc.tar.Z' is not a valid Zip file with Deflate compression method.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.IO.InvalidDataException,Message=End of Central Directory record could not be found.,Source=Microsoft.DataTransfer.ClientLibrary,'", "failureType": "UserError", "target": "cpy_file_raw_to_unzippedArchive_adls", "details": [] }
Shiv Kumar 40 Reputation points

2023-03-01T09:54:48.75+00:00

@Sally Dabbah , I tried but get a error.
"errorCode": "2200", "message": "ErrorCode=UserErrorUnzipInvalidFile,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The file 'abc.tar.Z' is not a valid Zip file with Deflate compression method.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.IO.InvalidDataException,Message=End of Central Directory record could not be found.,Source=Microsoft.DataTransfer.ClientLibrary,'", "failureType": "UserError", "target": "cpy_file_raw_to_unzippedArchive_adls", "details": [] }

Answer 1

Sally Dabbah 5 Microsoft Employee

you can decompress it using Copy activity:
in source dataset : choose your compression type
User's image

in sink -> choose a copy behavior -> Flatten hierarchy.
and you can write the output to a staging/ temp file and use it the way you like.
User's image

Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-02-17T03:17:30.8666667+00:00

Thank you @Sally Dabbah

Hello @Shiv Kumar NE/SNG/SMBC,

Just checking in to see if the below answer by @Sally Dabbah helped.

If this answers your query, click Accept Answer and Yes to was this answer helpful. And if you have any further queries, do let us know.
Shiv Kumar 40 Reputation points

2023-03-01T10:21:51.0866667+00:00

@Sally Dabbah , I am doing same thing but still getting an issue.

ERROR:
"errorCode": "2200", "message": "ErrorCode=UserErrorUnzipInvalidFile,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The file 'abc.tar.Z' is not a valid Zip file with Deflate compression method.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.IO.InvalidDataException,Message=End of Central Directory record could not be found.,Source=Microsoft.DataTransfer.ClientLibrary,'", "failureType": "UserError", "target": "cpy_file_raw_to_unzippedArchive_adls", "details": [] }

**
Below is ADF parse JSON, where you can see the patter and ZipDeflateRead Settings that I have passed.**
{ "source": { "type": "BinarySource", "storeSettings": { "type": "AzureBlobFSReadSettings", "recursive": false, "deleteFilesAfterCompletion": false }, "formatSettings": { "type": "BinaryReadSettings", "compressionProperties": { "type": "ZipDeflateReadSettings" } } }, "sink": { "type": "BinarySink", "storeSettings": { "type": "AzureBlobFSWriteSettings", "copyBehavior": "FlattenHierarchy" } }, "enableStaging": false }
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-03-01T21:27:47.1366667+00:00

Hello @Shiv Kumar NE/SNG/SMBC,

Can you please send us your sample '.tar.Z' file(please remove any sensitive data), so that I will try to reproduce the issue and get back to you with more details?

Looking forward to your reply.
Shiv Kumar 40 Reputation points

2023-03-02T03:33:51.4666667+00:00

Just for reference, I have added a support ticket also with number '2303020030000269'. and

Here you can find sample file.
https://drive.google.com/file/d/1UMZ9kdRpiqdAEkjIaPSgybZG7NPL2x-V/view?usp=share_link
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-03-02T22:32:03.9166667+00:00

Hello @Shiv Kumar NE/SNG/SMBC,

Thanks for the sample file. I tried to uncompress the .tar.z file using ADF, but no luck.

I don't see this compression type in ADF. Let me check with my internal team and get back to you whether Decompress a tar.Z file is supported in ADF or not.

In the meantime, you can use Azure functions as a workaround.

Please see the below documents for your reference.

https://github.com/Azure/Azure-DataFactory/tree/main/SamplesV2/UntarAzureFilesWithAzureFunction

https://learn.microsoft.com/en-us/azure/data-factory/control-flow-azure-function-activity T
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-03-07T21:07:47.7933333+00:00

Hello @Shiv Kumar NE/SNG/SMBC,

I am checking to see if you got a chance to look into the workaround.

Share via

Decompress a tar.Z file in Azure Data Factory

1 answer

Your answer