ADF - Error at sink side - Rest API- Key Content-Type conflicts in AuthHeaders/AdditionalHeaders

Question

ADF - Error at sink side - Rest API- Key Content-Type conflicts in AuthHeaders/AdditionalHeaders

Anurag Yadav 1

Hello,

I am getting error as below when I am trying to use the copy activity which as source as Json file from container and sink as Rest API.

I have defined auth token in linked service . Content Type header is defined in additional headers in Sink tab of copy activity. Where we need to define this actually ? In Linked service I can understand I can define API key for auth but Content Type has to reside in Sink tab.

Error :

Sink :

Sink Dataset :

Linked Service:

@AnnuKumari-MSFT or anyone else , please help.

Anurag Yadav 1 Reputation point

2022-09-27T04:44:37.783+00:00

@AnnuKumari-MSFT - can you please help into this.
AnnuKumari-MSFT 34,556 Reputation points Microsoft Employee Moderator

2022-09-27T06:27:47.53+00:00

Hi @Anurag Yadav ,

Thankyou for using Microsoft Q&A platform and thanks for posting your question here.

As I understand your issue, you are trying to copy data from Json file in ADLS to REST API using POST method. However, it is throwing the above mentioned error . Please let me know if my understanding is incorrect.

For providing better help, I would require few additional details from your side. Is the linked service connection successful?
Looking at the error message, It seems there is a conflict between additional headers that is being passed to the API. Probably, the API is auto generating the header by itself and there is no need to explicitly mention the content-Type header. Could you please check removing the header and see what happens.

Also, the schema present in the source JSON and the sink API schema needs to be mapped to customize the payload. As needed, you can use the copy activity schema mapping to reshape the source data to conform to the expected payload by the REST API.

Additionally, you can check with your source team , if there is any documentation related to the target API so as to get information like what are the required parameters to pass in the header or body section if needed.

Please let me know how it goes. Thankyou.
Anurag Yadav 1 Reputation point

2022-09-27T06:45:06.923+00:00

Hi @AnnuKumari-MSFT
Please find the respose :

Is the linked service connection successful? - Yes , using web activity I can POST the data successfully , just Copy activity is not working.

Could you please check removing the header and see what happens. -->I tried all the ways removing header from additional header and specifying the auth header in linked service but no luck. It doesn't work.

if there is any documentation related to the target API so as to get information like what are the required parameters to pass in the header or body section if needed. --> I have the documentation and the payload is exactly same as they require. It is posting successfully using web activity where we define content type header.

the schema present in the source JSON and the sink API schema needs to be mapped to customize the payload. As needed, you can use the copy activity schema mapping to reshape the source data to conform to the expected payload by the REST API. --> I tried with mapping and without mapping but no luck.
AnnuKumari-MSFT 34,556 Reputation points Microsoft Employee Moderator

2022-09-27T08:13:51.88+00:00

Hi @Anurag Yadav ,

What I can notice in the linked service Auth headers is, you have selected X-API-Token, usually, in ADF and synapse , we have the option as 'X-API-Key' , could you please delete the auth header from linked service and try adding it in copy activity itself like this:

If this doesn't work, I feel copy activity and web activity default payload is different. Probably, copy activity is changing the payload during the runtime , which we can't debug. If web activity is working fine for you, you can try a lookup activity pointing to the .json file and get the content of the json file and pass the output of the lookup as the body in web activity.
Anurag Yadav 1 Reputation point

2022-09-27T11:04:15.97+00:00

Hi @AnnuKumari-MSFT

X-API-Token is defined in target vendor API so we need to pass the same as header and it is working fine with web activity.

However I have tried the way as suggested by you still no luck.

You are right copy activity is changing the payload during the runtime and adding [] square brackets in the payload.

I had this idea in my mind with lookup activity but the target API accepts 1000 records at max in one call and we have almost 60K records to be pushed which I feel can't be achieved with this approach as it is tedious to break the payload in 1000s batch which is a basic feature in copy activity with Batch Size Property.
Please share your view.
AnnuKumari-MSFT 34,556 Reputation points Microsoft Employee Moderator

2022-10-03T07:03:21.63+00:00

Hi @Anurag Yadav ,

Apologies for delay in response. I would recommend you to create a support ticket with Microsoft so that our product team can directly engage with you and provide solution after deeper investigation. Here is how you can do that: Create an Azure support request

If you have a support plan you may file a support ticket to get to the root of this , else could you please send an email to azcommunity@microsoft.com with the below details, so that we can create a one-time-free support ticket for you to work closely on this matter. Thanks.
AnnuKumari-MSFT 34,556 Reputation points Microsoft Employee Moderator

2022-10-07T08:41:19.85+00:00

Hi @Anurag Yadav ,
Just following up to see if your issue got resolved? It would be much appreciated if you could share with us the mitigation steps performed to resolve the same. Thanks in advance.
Michael Wong 0 Reputation points

2023-05-25T04:23:37.41+00:00

Hi Guys,

I'm having the same issues, problem is not the source file or API Key, when you user Json POST to read a file, it automatically add the square bracket. There is no way of removing them, have tried using all available encoding as well.

4 answers

Your answer

Anurag Yadav 1 Reputation point

2022-09-27T04:44:37.783+00:00

@AnnuKumari-MSFT - can you please help into this.
AnnuKumari-MSFT 34,556 Reputation points Microsoft Employee Moderator

2022-09-27T06:27:47.53+00:00

Hi @Anurag Yadav ,

Thankyou for using Microsoft Q&A platform and thanks for posting your question here.

As I understand your issue, you are trying to copy data from Json file in ADLS to REST API using POST method. However, it is throwing the above mentioned error . Please let me know if my understanding is incorrect.

For providing better help, I would require few additional details from your side. Is the linked service connection successful?
Looking at the error message, It seems there is a conflict between additional headers that is being passed to the API. Probably, the API is auto generating the header by itself and there is no need to explicitly mention the content-Type header. Could you please check removing the header and see what happens.

Also, the schema present in the source JSON and the sink API schema needs to be mapped to customize the payload. As needed, you can use the copy activity schema mapping to reshape the source data to conform to the expected payload by the REST API.

Additionally, you can check with your source team , if there is any documentation related to the target API so as to get information like what are the required parameters to pass in the header or body section if needed.

Please let me know how it goes. Thankyou.
Anurag Yadav 1 Reputation point

2022-09-27T06:45:06.923+00:00

Hi @AnnuKumari-MSFT
Please find the respose :

Is the linked service connection successful? - Yes , using web activity I can POST the data successfully , just Copy activity is not working.

Could you please check removing the header and see what happens. -->I tried all the ways removing header from additional header and specifying the auth header in linked service but no luck. It doesn't work.

if there is any documentation related to the target API so as to get information like what are the required parameters to pass in the header or body section if needed. --> I have the documentation and the payload is exactly same as they require. It is posting successfully using web activity where we define content type header.

the schema present in the source JSON and the sink API schema needs to be mapped to customize the payload. As needed, you can use the copy activity schema mapping to reshape the source data to conform to the expected payload by the REST API. --> I tried with mapping and without mapping but no luck.
AnnuKumari-MSFT 34,556 Reputation points Microsoft Employee Moderator

2022-09-27T08:13:51.88+00:00

Hi @Anurag Yadav ,

What I can notice in the linked service Auth headers is, you have selected X-API-Token, usually, in ADF and synapse , we have the option as 'X-API-Key' , could you please delete the auth header from linked service and try adding it in copy activity itself like this:

If this doesn't work, I feel copy activity and web activity default payload is different. Probably, copy activity is changing the payload during the runtime , which we can't debug. If web activity is working fine for you, you can try a lookup activity pointing to the .json file and get the content of the json file and pass the output of the lookup as the body in web activity.
Anurag Yadav 1 Reputation point

2022-09-27T11:04:15.97+00:00

Hi @AnnuKumari-MSFT

X-API-Token is defined in target vendor API so we need to pass the same as header and it is working fine with web activity.

However I have tried the way as suggested by you still no luck.

You are right copy activity is changing the payload during the runtime and adding [] square brackets in the payload.

I had this idea in my mind with lookup activity but the target API accepts 1000 records at max in one call and we have almost 60K records to be pushed which I feel can't be achieved with this approach as it is tedious to break the payload in 1000s batch which is a basic feature in copy activity with Batch Size Property.
Please share your view.
AnnuKumari-MSFT 34,556 Reputation points Microsoft Employee Moderator

2022-10-03T07:03:21.63+00:00

Hi @Anurag Yadav ,

Apologies for delay in response. I would recommend you to create a support ticket with Microsoft so that our product team can directly engage with you and provide solution after deeper investigation. Here is how you can do that: Create an Azure support request

If you have a support plan you may file a support ticket to get to the root of this , else could you please send an email to azcommunity@microsoft.com with the below details, so that we can create a one-time-free support ticket for you to work closely on this matter. Thanks.
AnnuKumari-MSFT 34,556 Reputation points Microsoft Employee Moderator

2022-10-07T08:41:19.85+00:00

Hi @Anurag Yadav ,
Just following up to see if your issue got resolved? It would be much appreciated if you could share with us the mitigation steps performed to resolve the same. Thanks in advance.
Michael Wong 0 Reputation points

2023-05-25T04:23:37.41+00:00

Hi Guys,

I'm having the same issues, problem is not the source file or API Key, when you user Json POST to read a file, it automatically add the square bracket. There is no way of removing them, have tried using all available encoding as well.

Answer 1

Nadimpalli, R V Satya Avinash Varma 0

Hi All, I have same issue and did anyone managed to solve this issue ? If yes could you please share with us the mitigation steps performed to resolve the same. Thanks in advance.

Answer 2

Michael Wong 0

I manage to resolve my issues, instead of using Copy, I used Web activity instead.

I used lookup activity to read data followed by ForEach activity to get around the 1000 line limit.

Answer 3

Arundhati Sen 151

Hi I am having the same issue. Has someone solved this using copy activity and not web activity?

Answer 4

Kodanda Ramudu 0

Did the issue resolve ?. we are also facing same issue.

Share via

ADF - Error at sink side - Rest API- Key Content-Type conflicts in AuthHeaders/AdditionalHeaders

4 answers

Your answer