Put Blob REST API Content-Length error

Question

I'm trying to write the XML response obtained from a REST API GET request to a file in Azure Blob Storage, using Azure Data Factory.

I'd prefer to use the Copy activity to achieve this, but it doesn't support XML sinks. I've therefore been forced into using separate Web activities, to get the response and then write it to Blob storage.

I have 3 activities in my pipeline:

1. A Web activity, to issue my REST API GET request
2. A Set Variable activity, to assign the response obtained from step 1 to the variable called "Body"
3. A Web activity, to issue a REST API PUT request (using the "Body" variable from step 2)

This approach works fine for small responses but when I try doing the same for larger responses (around 1MByte), I get the following error:

Error calling the endpoint 'https://mystorageaccount.blob.core.windows.net'. Response status code: 'NA - Unknown'. More details: Exception message: 'NA - Unknown [ClientSideException] Bytes to be written to the stream exceed the Content-Length bytes size specified.'. Request didn't reach the server from the client. This could happen because of an underlying issue such as network connectivity, a DNS failure, a server certificate validation or a timeout

What am I doing wrong, or is there a better way of achieving my goal (but still using Azure Data Factory)?

Answer 1

TestPipeline.txt

For anyone wanting to reproduce the problem, the above test pipeline (I've changed the extension changed from ".json" to ".txt") contains 3 activities:

1. A Web activity to issue a REST API GET request, to get makes of cars (using a publicly available anonymous endpoint)
2. A Set Variable activity to extract the XML response from step 1 (into a variable called "Body")
3. A Web activity to issue a REST API PUT request, to write the XML response to Blob storage (using the variable called "Body" from step 2)

For privacy/security reasons, I've amended the URL in the Web activity in step 3 (to use the storage account "mystorageaccount", and the container "mycontainer").

Answer 2

Given how long the Put Blob REST API has been available, I can't believe I'm the first person to encounter this issue. Anyway, after much investigation, this turns out to be a documentation issue.

The documentation for the Put Blob REST API wrongly states that the "Content-Length" request header is required. This isn't true. To be more precise, it isn't true when issuing the POST request via an ADF Web activity. I don't have capacity to investigate whether the header is mandatory in other environments/toolsets.

If "Content-Length" is not omitted, then the POST request fails if the blob is larger than approximately 1MByte. I don't have capacity to investigate at what precise blob size the requests start failing.

If "Content-Length" is omitted, then the POST request succeeds for all blob sizes.