External file compression

Question

Hi ,

My input csv file is coming from in a zip format as like abcfile.csv.zip.

After unzipping the input zip file to CSV flatfile , I am able to get the data into the external file in Azure DW.

But I could not able to get the data without unzipping the file. I want to get the data into the external file from zipped file only

below is the external file format stmt I am using

CREATE EXTERNAL FILE FORMAT [CompressZip] WITH (FORMAT_TYPE = DELIMITEDTEXT, FORMAT_OPTIONS (FIELD_TERMINATOR = N'|', DATE_FORMAT = N'yyyy-MM-dd', USE_TYPE_DEFAULT = False), DATA_COMPRESSION = N'org.apache.hadoop.io.compress.DefaultCodec')

if i use this external file format i am getting the error like below,

Location: '/TestData/TestTFolder/abcfile.csv.zip' Error: UTF8 decode failed

Can you please suggest is my create external file format stmt is right ?

Kind Regards,
Gowri Shankar

Answer 1

@Gowri Shankar
No, you cannot decompress .zip files.
Please note that .zip is not a file compression format but a compressed archive format. As such it could contain more than one file. Thus you cannot support .ZIP as a compression format. Instead you use .gz (which is a compression format).

If you want to create an external table over a ZIP archive, you either need to unpack the archive (and recompress the files individually with any of the supported compression formats) or you have to create your own data source adapter that will know how to handle ZIP archives and the files contained in the archives.

Thanks
Saurabh

----------

Please do not forget to "Accept the answer" wherever the information provided helps you to help others in the community.