azure data factory - REST API in web activity vs copy-data activty

Leroy Mason 1 Reputation point
2020-08-05T22:48:19.02+00:00

I'm using Azure DAta Factory V2 in Australia East to write a pipeline that need to call a REST API hosted in the on-premise data centre.

If a Web activity is used to invoke the API the pipeline throws the error below. The same REST API in a Copy-data activity (creating a JSON file) and it works fine.

The pipeline JSON support file is attached.

Why is the Web activity failing but the Copy-data activity pointing at the same REST API working? The pipeline's integration runtime (IR) has line of sight to the API endpoint and both activities are using the same IR.

Error message using Web activity:
Operation on target Web1 failed: Error calling the endpoint 'https://999.999.999.999:16900/api/Patient'. Response status code: ''. More details:Exception message: 'An error occurred while sending the request.'. No response from the endpoint. Possible causes: network connectivity, DNS failure, server certif15971-snag-a2a5c39.pngicate validation or timeout.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,652 questions
{count} votes

1 answer

Sort by: Most helpful
  1. AMJ 6 Reputation points
    2021-02-01T01:06:53.637+00:00

    I had a similar task to extract an XML document from a REST API request made from WEB activity in ADF via an on-prem gateway as the URL was private.

    I had an opposite result from what @Leroy Mason had, ie, a legit response from WEB activity but failed message from Copy-data activity made from a REST API call.

    This is the legit JSON response with embedded XML made by WEB activity:

    {
    "Response": "<response status=\"success\"><result>\n <msg>\n <line>Report job enqueued with jobid 2681</line>\n </msg>\n <job>2681</job>\n </result></response>",
    "ADFWebActivityResponseHeaders": {
    "Connection": "keep-alive",
    "X-FRAME-OPTIONS": "SAMEORIGIN",
    "X-XSS-Protection": "1; mode=block",
    "X-Content-Type-Options": "nosniff",
    "Content-Security-Policy": "default-src 'self'; script-src 'self' 'unsafe-eval' 'unsafe-inline'; style-src 'self' 'unsafe-inline'; img-src 'self' data:;",
    "Strict-Transport-Security": "max-age=31536000",
    "Pragma": "no-cache",
    "Cache-Control": "no-store, must-revalidate, no-cache, post-check=0, pre-check=0",
    "Date": "Sun, 31 Jan 2021 22:36:44 GMT",
    "Set-Cookie": "PHPSESSID=b5cb39846b6fa1d09614c1e855f60125; path=/; secure; HttpOnly",
    "Content-Length": "154",
    "Allow": "GET;HEAD;POST;PUT;DELETE;OPTIONS",
    "Content-Type": "application/xml; charset=UTF-8",
    "Expires": "Thu, 19 Nov 1981 08:52:00 GMT"
    },
    "effectiveIntegrationRuntime": "OnPremRuntime",
    "executionDuration": 2,
    "durationInQueue": {
    "integrationRuntimeQueue": 4
    },
    "billingReference": {
    "activityType": "ExternalActivity",
    "billableDuration": [
    {
    "meterType": "SelfhostedIR",
    "duration": 0.016666666666666666,
    "unit": "Hours"
    }
    ]
    }
    }

    And this is the failed message made by Copy-data activity REST API:

    Failure happened on 'Source' side. ErrorCode=JsonInvalidDataFormat,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Error occurred when deserializing source JSON file ''. Check if the data is in valid JSON object format.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=Newtonsoft.Json.JsonReaderException,Message=Unexpected character encountered while parsing value: <. Path '', line 0, position 0.,Source=Newtonsoft.Json,'

    The output from WEB activity has passed a JSON validator successfully. I was puzzled why it failed from Copy-data activity REST API request?

    I know it may not have answered @Leroy Mason 's question directly. Please refer to the attached WEB activity configuration to see if it could shed a tad light.

    62245-03-web-activity-rest-api.png

    1 person found this answer helpful.

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.