Why does the Azure Data Factory Web Activity return a compressed gzip response when there is an API error but returns a normal (uncompressed) JSON response when there is no error?

Question

When I make a Postman POST call into a system, with a request that is designed to fail, I get back the appropriate error message. The only request header I add is Content-Type: application/json.

{
    "error": {
        "message": "Unable to complete operation for a project of status Cancelled",
        "title": null,
        "msgKey": "exception.attask",
        "attributes": [
            ""
        ],
        "code": 0
    }
}

When I make the exact same request using the Azure Data Factory Web Activity I get back the following:

{
    "Response": "\u001F�\b\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0014̱\u000E�0\f��W�<��3k�Ν�\u000E&\u001C�6đm�\"Ļ7�n��;\u0018fj�\u001F��]fpϏ\"C\u0006�Rҵf\u0004H+Lb�B�\u001A\tU�7R�N�!�9ݤ$䌑;�%r�ʖsǫ�w��7����D�Z؆-�\u0016p�̯����y=�\u001F\u0000\u0000\u0000��\u0003\u0000)�\u0000\u0000\u0000"
}

Requests that don't fail return expected (uncompressed) JSON results in both environments. Is this a bug or am I missing something?

Answer 1

@Peter Kierstead

Thanks for reaching out to Microsoft Q&A.

The behavior you’re experiencing with Azure Data Factory (ADF) returning a compressed gzip response for API errors, while providing a normal JSON response for successful requests, can be attributed to how the API is configured to handle error responses.

1. API Design: Some APIs are designed to return compressed responses for error messages to save bandwidth, especially if the error messages are lengthy or contain detailed stack traces. This might not be consistent across all endpoints or methods.
2. Content-Encoding Header: When the API encounters an error, it might automatically apply gzip compression to the response. If the Content-Encoding header is set to gzip, ADF may not handle it correctly, leading to the compressed output you’re seeing.
3. Error Handling in ADF: ADF might not be configured to decompress gzip responses when an error occurs. This could be a limitation or a specific behavior of the Web Activity in ADF.
4. Modify Request Headers: You might try adding an Accept-Encoding: identity header to your ADF request. This could instruct the server not to compress the response, allowing you to receive the error message in a readable format.
5. Post-Processing: If you continue to receive gzip responses, consider implementing a post-processing step in your pipeline to handle decompression. You could use Azure Functions or another service to decompress the response before further processing.

Hope this helps. Do let us know if you any further queries.

Answer 2

@phemanth

Thank you for your response. I tried putting this into a comment but even thought the Comment button accepted the text it still does not show up here... so, putting it in the Answer section...

I've already tried your suggestions, and more, with no resolution. My pipeline is already at its runtime limit (Web Activity calls are in a loop) as per customer response requirements so adding additional calls to Azure Functions or Batch is not something I can do. This problem occurs even with other non-related APIs. There are complaints on Google about this same behavior going back a few years. How can this not be considered a bug in Web Activity response processing?