I have a REST API call to download a CSV file in Postman. (see screenshot).
I would like to build this request in Azure Data Factory to automatically store this CSV in Azure Data Lake Storage.
In the pipeline, I first execute a few HTTP requests to authenticate myself and prepare for the download.
Then I want to use a Copy Activity to save the CSV to the Azure Data Lake Storage via the REST API call.
The REST API call does not need any additional headers or values in the body.
After a successful request in Postman, I get the following response headers.
Here I can see that the CSV was compressed with gzip.
Therefore, I have configured the following as the source of the Copy Activity:
However, with this configuration I get the following error message:
ErrorCode=InvalidDataFormat,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The source data has an invalid format. Cannot decompress the source data. Source file name: '../../../xxxx'.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=ICSharpCode.SharpZipLib.GZip.GZipException,Message=Error GZIP header, first magic byte doesn't match,Source=ICSharpCode.SharpZipLib,'
What could be the reason for this? Where is the difference?