How can I prevent Azure Data Factory (Web Activity) from handling Redirects?

Question

We have a requirement to perform a series of callouts to get the report and we have built the pipeline and used the web activity to make the callouts.

In that process, we’re getting the below error while making a get request using the web activity.

<Error><Code>InvalidArgument</Code><Message>Only one auth mechanism allowed; only the X-Amz-Algorithm query parameter, Signature query string parameter or the Authorization header should be specified</Message>

While making a request to the API I need to pass in an Authorization header in my request and then the service will generate the report and respond with a 302 status code which has a Location response header. That Location header contains the link I need to go to in order to bring back the actual report data.

However, in ADF, using a Web Activity, I send my GET request to the API, the API responds, and ADF seems to automatically be following the 302 redirect. When it follows the redirect url, the new url returns an error message because ADF is passing in the original Authorization header and the redirect url responds with "only one Authorization method is allowed". It turns out the redirect url from the Location header comes with an authorization token as a url parameter (X-Amz-Signature).

What I want to do is stop ADF from following the Redirect url so I can capture the original response headers and go to the Location url in a seperate Web Activity.

Is there any way I can tell ADF not to follow Redirects and just give me the original response (vs giving me the response from the second url)?

Answer 1

Hi @MartinJaffer-MSFT ,

Yes, you're right. The second call is using a different authorization scheme. The call is going to a AWS s3 bucket to get the stored data. The first call is successful but the redirection url is missing in the response as the activity failed. This is the response from the first call.

{
"Response": "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>InvalidArgument</Code><Message>Only one auth mechanism allowed; only the X-Amz-Algorithm query parameter, Signature query string parameter or the Authorization header should be specified</Message><ArgumentName>Authorization</ArgumentName><ArgumentValue>Bearer aaaaa</ArgumentValue><RequestId>TDFBYZJ73XW97T3S</RequestId><HostId>PS2j4O9Q=</HostId></Error>",
"ADFWebActivityResponseHeaders": {
"x-amz-request-id": "TDFBYZJaaaaa",
"x-amz-id-2": "fftyyy=",
"Transfer-Encoding": "chunked",
"Connection": "close",
"Date": "Fri, 06 Dec 2022 20:17:42 GMT",
"Server": "AmazonS3",
"Content-Type": "application/xml"
},
"effectiveIntegrationRuntime": "AutoResolveIntegrationRuntime (East US 2)",
"executionDuration": 0,
"durationInQueue": {
"integrationRuntimeQueue": 0
},
"billingReference": {
"activityType": "ExternalActivity",
"billableDuration": [
{
"meterType": "AzureIR",
"duration": 0.016666666666666666,
"unit": "Hours"
}
]
}
}

Answer 2

same here, I'm calling report APIs from Talkdesk service, it redirects to AWS but Azure Data Factory copies the autentication headers to the redirect call, giving the same error.

To be more precise, I call

https://api.talkdeskapp.com/data/reports/explore_calls/jobs/[job number, obtained from a previous call]

with oauth2 auhorization, the answer is (cleaned of private data):

Rest call failed with client error, status code 400 BadRequest, please check your activity settings.
Request URL: https://td-infra-prd-us-east-1-s3-hadesreports.s3.amazonaws.com/hades/reports/[job id]?response-content-type=application%2Fjson&X-Amz-Algorithm=[algorithm info]X-Amz-Date=20230207T081007Z&X-Amz-SignedHeaders=host&X-Amz-Expires=3600&X-Amz-Credential=[some credential info]&X-Amz-Signature=[signature ID].
Response: 
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>InvalidArgument</Code><Message>Only one auth mechanism allowed; only the X-Amz-Algorithm query parameter, Signature query string parameter or the Authorization header should be specified</Message><ArgumentName>Authorization</ArgumentName><ArgumentValue>Bearer [bearer string from the first call]</ArgumentValue><RequestId>[request ID]</RequestId><HostId>[host ID]</HostId></Error>
Activity ID: [Activity ID]

The problem is that, when a rest api is redirected, Azure Data Factory copies the header data into the new redirect call, causing the issue. I could extract the request url from the error, but if I use a web component it gives the error without the request url, this error is from a copy data component connected to a rest api linked service

Answer 3

I am having this same issue right now. Were either of you able to find a solution?

Answer 4

I've solved this by creating an azure function that do the two calls, this way I've managed all the authorizations in C#, but this is quite a workaround

Answer 5

I had the same issue. Try the Disable async pattern option.