ADF Web activity can't connect to API When API Key has special character

Question

ADF Web activity can't connect to API When API Key has special character

Nick Baker 1

I am trying to use a Web activity in an ADF pipeline to issue a GET to an API. The API authentication is through an Authorization header with an API key provided to be by the API developer. The API Key is of the form username:password (note the colon)

When I try and authenticate against the API with my correct API key containing that colon character, I get the following error (sensitive data redacted):

"Error calling the endpoint 'https://api.XXXXX.com/XXX'. Response status code: ''. More details:Exception message: 'The format of value 'USERNAME:PASSWORD' is invalid.'. No response from the endpoint. Possible causes: network connectivity, DNS failure, server certificate validation or timeout."

When I try with an incorrect API key with no colon, I get a valid response from the API:

"{"payload":{"error_code":401,"message":"Not authorized."},"status":401,"requestTime":"2021-06-04T12:14:33","responseTime":"2021-06-04T12:14:33","statusText":"Unauthorized","url":"XXXX","method":"GET"}"

There is nothing wrong with my original API call - I have tried this exact setup in Postman and it works perfectly. I can only think that the way ADF is sending that header string differs in some way to how Postman sends it when there is a special character.

Any help would be appreciated!

MartinJaffer-MSFT 26,236 Reputation points

2021-06-04T21:03:53.707+00:00
Hello @Nick Baker and welcome to Microsoft Q&A. Thank you for bringing this to our attention.

I agree with your assessment. There are a number of possibilities.

escape the : as %3A (@encodeUriComponent('USERNAME:PASSWORD'))

escape the : as \: in the JSON code

there could be a deeper problem in Data Factory (needs testing)

The error sounds to me like somewhere in the Data Factory side, it is having trouble parsing the : and reading as "Header":"Value":"Value". The results of trying with each escape method should help give greater insight into the cause. I expect the resulting error(s) will be different.

Please let me know what you find.

Thank you.
Nick Baker 1 Reputation point

2021-06-07T11:11:14.477+00:00

Thanks @MartinJaffer-MSFT - I've tried both options you've suggested:

Using @encodeUriComponent() results in the same 'Unauthorised' response from the API. I assume the replacement of the : with %3A results in an incorrect string being submitted.

Adding the escape character directly into the JSON code seems to result in an error in ADF of 'invalid JSON syntax' - screenshot below in case I've misunderstood your suggestion.
Nick Baker 1 Reputation point

2021-06-07T11:14:53.757+00:00

Also just an aside - I don't have this issue when I use this exact same API as a linked service\dataset in a copy data activity. In fact I can return the data I want in this web activity and store it - that is my current workaround, but seems clunky when I only want to do a simple call.
MartinJaffer-MSFT 26,236 Reputation points

2021-07-14T17:06:25.95+00:00

If you found your own solution, please share here with the community.

Your answer

MartinJaffer-MSFT 26,236 Reputation points

2021-06-04T21:03:53.707+00:00

Hello @Nick Baker and welcome to Microsoft Q&A. Thank you for bringing this to our attention.

I agree with your assessment. There are a number of possibilities.

escape the : as %3A (@encodeUriComponent('USERNAME:PASSWORD'))

escape the : as \: in the JSON code

there could be a deeper problem in Data Factory (needs testing)

The error sounds to me like somewhere in the Data Factory side, it is having trouble parsing the : and reading as "Header":"Value":"Value". The results of trying with each escape method should help give greater insight into the cause. I expect the resulting error(s) will be different.

Please let me know what you find.

Thank you.
Nick Baker 1 Reputation point

2021-06-07T11:11:14.477+00:00

Thanks @MartinJaffer-MSFT - I've tried both options you've suggested:

Using @encodeUriComponent() results in the same 'Unauthorised' response from the API. I assume the replacement of the : with %3A results in an incorrect string being submitted.

Adding the escape character directly into the JSON code seems to result in an error in ADF of 'invalid JSON syntax' - screenshot below in case I've misunderstood your suggestion.
Nick Baker 1 Reputation point

2021-06-07T11:14:53.757+00:00

Also just an aside - I don't have this issue when I use this exact same API as a linked service\dataset in a copy data activity. In fact I can return the data I want in this web activity and store it - that is my current workaround, but seems clunky when I only want to do a simple call.
MartinJaffer-MSFT 26,236 Reputation points

2021-07-14T17:06:25.95+00:00

If you found your own solution, please share here with the community.

Share via

ADF Web activity can't connect to API When API Key has special character

Your answer