Lookup specific file inside zip file

Question

I have a zip file that contains 3 csv file.

Extracts_2023-Mar-15.zip

• booking.csv
• client.csv
• vendor.csv

structure folder :

region / singapore / extracts / Extracts_2023-Mar-15.zip

Previously there was only 1 file in zip file, but now added 2 files in 1 zip file.

This is the pipeline that I made before.

Source dataset linked to amazon s3 bucket with file path : region /

File path in lookup use wildcard paths :
@concat(pipeline().parameters.select_country, '/extracts') / @concat('Extracts_', variables('current_date'), '.zip')

How can I use lookup to be able to retrieve data from a specific file inside a zip file?

I just want to use lookup for client.csv file only in extract.zip file, is it possible?

or do I have to create a new pipeline just to extract the zip files and move them to each file's folder to be readable by the pipeline above?

which one is possible and the lightest and also the fastest process?

Answer 1

Hi @Galih ,

Thanks for using Microsoft Q&A forum and posting your query.

My understanding if the requirement is that you have a Zip file in Amazon S3 which contains multiple .csv files and you would like to exact a specific file out of the 3 files. Please correct if I misunderstood your requirement.

ADF doesn't have a functionality to filter down to specific file while unzipping multiple files from a single Zip file. The best way is to exact/unzip all 3 files to an intermediate location in Azure (may be azure blob or ADLS Gen2) and you then may use lookup and point to that specific intermediate location and specific file and do the regular process as you did earlier in your old pipeline and then at the end if you would like to get rid of those files from the intermediate location, then you can utilize Delete activity in ADF to delete all files in that intermediate folder. That way you can just lookup your specific file after unzipping and clear the folder or you can keep and do other process as per your specific need.

But this seems like a good feature request for ADF product hence I would recommend you to please log your feedback in IDEAS forum here: https://feedback.azure.com/d365community/forum/1219ec2d-6c26-ec11-b6e6-000d3a4f032c

All the feedback shared in this forum are actively reviewed and monitored by respective product owners.

Please do share the feature request link once the feedback is logged so that we can take it forward to respective product owners and have them review it.

Hope this helps. Let us know if you have further questions

---

Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.