How to import dynamically generated csv file as a source in ADF

Question

In data factory what source setup can be used to get this dynamically generated csv file as a source? I can't seem to get a valid file from this public url which auto creates a file for download (and where we are not providing a filename). I have tried various options but the CSV file I output to blob storage is not correctly created. An example of the csv file is the one created by this URL:

https://api.coronavirus.data.gov.uk/v1/data?filters=areaType=nation;areaName=england&structure={"date":"date","name":"areaName","code":"areaCode","cases":{"daily":"newCasesByPublishDate","cumulative":"cumCasesByPublishDate"},"deaths":{"daily":"newDeathsByDeathDate","cumulative":"cumDeathsByDeathDate"}}format=csv

Answer 1

Hello @123ay4h321 and welcome to MIcrosoft Q&A. Thank you for this question. The example URL helped greatly. Let me share the steps I took to solve your issue.

I visited the link you shared, with ym browser. It resulted in a download, which I was able to read in a text editor.
So I then tried using a web activity in ADF on the same URL. The output content was not readable, as shown below. I recalled that attached content is often compressed. So I checked the output again. See the highlighted below.

The response headers included "Content-Encoding" : "gzip" .

To solve one issue at a time, I tried copying, using HTTP > Binary dataset for source, and BLOB > Binary for sink. When I set the
source dataset compression to gzip , and the sink dataset compression to none, I was able to read the output in my storage.

TLDR: Match the source dataset compression to that in the headers.

If this answer solved your issue, please mark as accepted. Otherwise let me know what I can do to help better.

Answer 2

Many thanks @MartinJaffer-MSFT this solves the issue perfectly, Much appreciated!