ADF extract .zip file from Web call's output response

Question

So I currently have 2 pipelines (the first one is finished):

First pipeline: an API call to request a bearer token from the external source server (this requires a certificate and basic authentication), then a set variable activity to extract only the bearer token from the response, and lastly an execute pipeline to invoke the other pipeline (to request the dataset) the execute pipeline sets the variable (bearer token) as a parameter for the second pipeline to use.

second pipeline: this is where i'm struggling, i made an WEB-API call to request the data and it works as intended, however the WEB-API response is a .zip file containing a .txt file. the response i get is as followed:

How can i extract the .zip file or .txt file from the web response ? i've tried using the activity ''Copy Data" but unfortunately without success as it requests a dataset as source and not a response. is there any way to use copy data with a response as a source?

Answer 1

Hello @Steff ,
Thanks for the question and using MS Q&A platform.

I understand you want to fetch (compressed) data with web activity, and then decompress and write it.

I tried to do something similar -- maybe a year ago? -- with Stack Overflow api. I fetched with web activity, then used a second web activity to write to blob, then used copy activity to decompress. The copy failed, unable to decompress. It couldn't get the magic number. My theory goes like this:

The web activity input/output appears to be stored as string interim data type (assumption, not verified). This somehow might be corrupting the binary nature of the data. The corruption made the decompress/unzip fail.

So, in short, web activity is not a good idea when the result is not text-based. I do have alternative suggestion:

• Have you tried using a HTTP > Binary dataset as source for your copy activity, instead of using a web activity?

Please do let me if you have any queries.

Thanks
Martin

---

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
  • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators