i'm not able to do write the zip file into adls using adf

Question

I have rest API, first using web activity i have sent request to get the url, we got the url that url got response . here response means zip file, here we are getting json format, here my question is how to take that response as a zip file and load into blob and need to unzip. can any one please give suggestion.

Answer 1

You can use a Web Activity to call your REST API endpoint and get the URL of the zip file.

(with the REST API as the source and your ADLS as the sink) then use a Custom Activity (which allows you to run custom code) to unzip the downloaded file in ADLS.

Step 1: Web Activity

1. Add a Web Activity to your pipeline.
2. Configure it to call your REST API endpoint.
3. Capture the URL of the zip file from the response.

Step 2: Copy Activity

1. Add a Copy Activity to your pipeline.
2. In the Source tab, configure it to use the URL obtained from the Web Activity.
   • Use the dynamic content feature to pass the URL from the Web Activity.
3. In the Sink tab, configure it to write to ADLS.

Step 3: Custom Activity to Unzip the File

1. Add a Custom Activity to your pipeline. This requires setting up an Azure Batch Account and creating a pool of compute nodes.
2. Write a script (If you are familiar with Python script) to unzip the file.

import zipfile

import os


zip_file_path = "<path_to_the_zip_file>"

extract_path = "<path_to_extract_directory>"


os.makedirs(extract_path, exist_ok=True)


with zipfile.ZipFile(zip_file_path, 'r') as zip_ref:

    zip_ref.extractall(extract_path)

3. Upload the script to a location accessible by the Custom Activity (e.g., a storage account).
4. Configure the Custom Activity to run the script.