azure data factory - REST API in web activity vs copy-data activty

Question

azure data factory - REST API in web activity vs copy-data activty

Leroy Mason 1

I'm using Azure DAta Factory V2 in Australia East to write a pipeline that need to call a REST API hosted in the on-premise data centre.

If a Web activity is used to invoke the API the pipeline throws the error below. The same REST API in a Copy-data activity (creating a JSON file) and it works fine.

The pipeline JSON support file is attached.

Why is the Web activity failing but the Copy-data activity pointing at the same REST API working? The pipeline's integration runtime (IR) has line of sight to the API endpoint and both activities are using the same IR.

Error message using Web activity:
Operation on target Web1 failed: Error calling the endpoint 'https://999.999.999.999:16900/api/Patient'. Response status code: ''. More details:Exception message: 'An error occurred while sending the request.'. No response from the endpoint. Possible causes: network connectivity, DNS failure, server certificate validation or timeout.

HarithaMaddi-MSFT 10,146 Reputation points

2020-08-06T11:54:55.33+00:00

Hi @Leroy Mason ,

Welcome to Microsoft Q&A Platform.

As per the Web Activity documentation, the activity will timeout at 1 minute with an error if it does not receive a response from the endpoint. Can you please check and confirm if this is an issue here.
Leroy Mason 1 Reputation point

2020-08-06T20:56:45.323+00:00

Thank you for writing.

I had extended the timeout previously and upon your suggestion I tried again. I set the timeout to 1:00:00 (1 hour), published and ran in debug mode. The same error as in the article above occurred after duration 00:00:13 (13 ms).

My current thinking is to write an Azure function in C# to implement REST API calls as there are quite a few I need to invoke.

regards
Leroy
HarithaMaddi-MSFT 10,146 Reputation points

2020-08-10T12:00:01.513+00:00

Hi @Leroy Mason ,

Thank you for sharing the details. Yes, Azure function is native component of Azure that can be leveraged for this type of requirement. However, I reached out to internal team to understand the root cause of the issue from Web Activity. To investigate further by internal team, can you please share pipeline run id that failed with the above issue. I will share more updates once I hear back from them.
HarithaMaddi-MSFT 10,146 Reputation points

2020-08-14T10:52:08.253+00:00

Hi @Leroy Mason ,

We have not received a response from you. Are you still facing the issue? If you found a solution, would you please share it here with the community? Otherwise, please share the pipeline run id and we will continue to engage with you on the issue.
HarithaMaddi-MSFT 10,146 Reputation points

2020-08-17T04:29:03.26+00:00

Hi @Leroy Mason ,

Since we have still not heard back from you, following up to check the details. Are you still facing the issue? If you found a solution, would you please share it here with the community? Otherwise, please share pipeline run id as mentioned and we will continue to engage with you on the issue.
Balram Singh 6 Reputation points

2021-04-22T17:14:45.327+00:00

Hi LeroyMason - Can you please provide the steps to pass the SAS token in the copyData activity for the REST API (sink). I am getting error as "binary source should always have a binary destination". It will be good if you can share the Json code for your copyData Activity.
Leroy Mason 1 Reputation point

2021-04-22T22:16:07.287+00:00

I think you've created the sink dataset incorrectly, i.e. it needs to be not a binary sink.

JSON for copyData attached.
90447-jsonpatientquery.txt
Leroy Mason 1 Reputation point

2021-04-22T22:17:30.827+00:00

90514-jsonpatientquery.txt
data set

1 answer

Your answer

HarithaMaddi-MSFT 10,146 Reputation points

2020-08-06T11:54:55.33+00:00

Hi @Leroy Mason ,

Welcome to Microsoft Q&A Platform.

As per the Web Activity documentation, the activity will timeout at 1 minute with an error if it does not receive a response from the endpoint. Can you please check and confirm if this is an issue here.
Leroy Mason 1 Reputation point

2020-08-06T20:56:45.323+00:00

Thank you for writing.

I had extended the timeout previously and upon your suggestion I tried again. I set the timeout to 1:00:00 (1 hour), published and ran in debug mode. The same error as in the article above occurred after duration 00:00:13 (13 ms).

My current thinking is to write an Azure function in C# to implement REST API calls as there are quite a few I need to invoke.

regards
Leroy
HarithaMaddi-MSFT 10,146 Reputation points

2020-08-10T12:00:01.513+00:00

Hi @Leroy Mason ,

Thank you for sharing the details. Yes, Azure function is native component of Azure that can be leveraged for this type of requirement. However, I reached out to internal team to understand the root cause of the issue from Web Activity. To investigate further by internal team, can you please share pipeline run id that failed with the above issue. I will share more updates once I hear back from them.
HarithaMaddi-MSFT 10,146 Reputation points

2020-08-14T10:52:08.253+00:00

Hi @Leroy Mason ,

We have not received a response from you. Are you still facing the issue? If you found a solution, would you please share it here with the community? Otherwise, please share the pipeline run id and we will continue to engage with you on the issue.
HarithaMaddi-MSFT 10,146 Reputation points

2020-08-17T04:29:03.26+00:00

Hi @Leroy Mason ,

Since we have still not heard back from you, following up to check the details. Are you still facing the issue? If you found a solution, would you please share it here with the community? Otherwise, please share pipeline run id as mentioned and we will continue to engage with you on the issue.
Balram Singh 6 Reputation points

2021-04-22T17:14:45.327+00:00

Hi LeroyMason - Can you please provide the steps to pass the SAS token in the copyData activity for the REST API (sink). I am getting error as "binary source should always have a binary destination". It will be good if you can share the Json code for your copyData Activity.
Leroy Mason 1 Reputation point

2021-04-22T22:16:07.287+00:00

I think you've created the sink dataset incorrectly, i.e. it needs to be not a binary sink.

JSON for copyData attached.
90447-jsonpatientquery.txt
Leroy Mason 1 Reputation point

2021-04-22T22:17:30.827+00:00

90514-jsonpatientquery.txt
data set

Answer 1

I had a similar task to extract an XML document from a REST API request made from WEB activity in ADF via an on-prem gateway as the URL was private.

I had an opposite result from what @Leroy Mason had, ie, a legit response from WEB activity but failed message from Copy-data activity made from a REST API call.

This is the legit JSON response with embedded XML made by WEB activity:

{
"Response": "<response status=\"success\"><result>\n <msg>\n <line>Report job enqueued with jobid 2681</line>\n </msg>\n <job>2681</job>\n </result></response>",
"ADFWebActivityResponseHeaders": {
"Connection": "keep-alive",
"X-FRAME-OPTIONS": "SAMEORIGIN",
"X-XSS-Protection": "1; mode=block",
"X-Content-Type-Options": "nosniff",
"Content-Security-Policy": "default-src 'self'; script-src 'self' 'unsafe-eval' 'unsafe-inline'; style-src 'self' 'unsafe-inline'; img-src 'self' data:;",
"Strict-Transport-Security": "max-age=31536000",
"Pragma": "no-cache",
"Cache-Control": "no-store, must-revalidate, no-cache, post-check=0, pre-check=0",
"Date": "Sun, 31 Jan 2021 22:36:44 GMT",
"Set-Cookie": "PHPSESSID=b5cb39846b6fa1d09614c1e855f60125; path=/; secure; HttpOnly",
"Content-Length": "154",
"Allow": "GET;HEAD;POST;PUT;DELETE;OPTIONS",
"Content-Type": "application/xml; charset=UTF-8",
"Expires": "Thu, 19 Nov 1981 08:52:00 GMT"
},
"effectiveIntegrationRuntime": "OnPremRuntime",
"executionDuration": 2,
"durationInQueue": {
"integrationRuntimeQueue": 4
},
"billingReference": {
"activityType": "ExternalActivity",
"billableDuration": [
{
"meterType": "SelfhostedIR",
"duration": 0.016666666666666666,
"unit": "Hours"
}
]
}
}

And this is the failed message made by Copy-data activity REST API:

Failure happened on 'Source' side. ErrorCode=JsonInvalidDataFormat,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Error occurred when deserializing source JSON file ''. Check if the data is in valid JSON object format.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=Newtonsoft.Json.JsonReaderException,Message=Unexpected character encountered while parsing value: <. Path '', line 0, position 0.,Source=Newtonsoft.Json,'

The output from WEB activity has passed a JSON validator successfully. I was puzzled why it failed from Copy-data activity REST API request?

I know it may not have answered @Leroy Mason 's question directly. Please refer to the attached WEB activity configuration to see if it could shed a tad light.

Sukanya Wattal 1 Reputation point

2022-09-08T09:15:24.57+00:00

Hi @AMJ can you please share how you resolved your copy data activity error as I am getting the same error. Web activity is working for the same API call.

Error code: 21155
Details: Error occurred when deserializing source JSON file ''. Check if the data is in valid JSON object format.
Unexpected character encountered while parsing value: <. Path '', line 0, position 0.
Luiz Santos, Stênia 1 Reputation point

2022-11-30T14:48:10.45+00:00

Hi @AMJ and @Sukanya Wattal Did you get an answer? I'm getting the same error. Thanks.

Share via

azure data factory - REST API in web activity vs copy-data activty

1 answer

Your answer