How do I call an API that redirects on the backend with a Copy Activity in Azure Data Factory without an authentication error?

Question

I'm running into issues in Azure Data Factory (ADF) with an API I am working with. I am unfortunately not at liberty to share the API URL or much data on the call I am making, but I can say that I have been testing this API call in Postman and through other tools, so I know I am using the right authentication method, URL, endpoint, etc.

When I try to call the API in ADF, I get the following error.

Only one auth mechanism allowed; only the X-Amz-Algorithm query parameter, Signature query string parameter or the Authorization header should be specified

Based on what research I've done so far, it appears that the API I am calling is referencing Amazon Web Services (AWS) through a second API on the backend to fetch data. When I call the API in Postman and through other tools, it drops the initial authentication I use for the first API when it calls the second API for AWS. However, when I use ADF, it is keeping the authentication on, so when it calls the AWS API it is still using my authentication for the first API.

I haven't seen a working solution for this problem, so I was wondering if anyone has run into this problem and knows a workaround?

Any and all help is appreciated.

Answer 1

Hello Westerfield, Jared T,

Thank you for providing additional details.

After further checking with my internal team, it has been confirmed that ADF doesn't clean up the authentication header on redirect requests, causing the issue. This behavior is intentional and by design.

We have created a product backlog item for addressing this issue in a future release.

Please feel free to reach out if you have any further questions or concerns.