Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
Azure Data Factory: Web Activity vs HTTP Acivity/Service
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(179.20000000000002, 31%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESL%3C/text%3E%3C/svg%3E)
Sam Lohbeck
0
Reputation points
Hi guys!
I am struggling with an issue in Azure Data Factory. Essentially, I can successfully get data from an API with the Web activity, but not with the HTTP (or REST) service.
I plug in a URL endpoint in the Web activity and POST some payload and it returns some JSON back perfectly. But, with the HTTP (or REST) service, I setup the same endpoint with the base URL in a Linked Service & a relative URL in a Source Dataset and I consistently get 403 Forbidden, or 400 Bad Request errors.
It leaves me to believe that there is some sort of authentication it is trying to do that I am missing. The company API I am accessing is internal and requires me to send a username & password in the POST itself, so there is no authentication step other than that.
I can provide more details as needed. If anybody has fought this before or has any ideas, I would appreciate it!
Thanks!
Works:
Azure Data Factory
{count} votes
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(54.400000000000006, 38%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EQ%3C/text%3E%3C/svg%3E)
QuantumCache 20,676 Reputation points Moderator2024-02-09T17:44:39.72+00:00 Hello @Sam Lohbeck Basically in such scenarios, we recommend to test with some other API endpoints and compare the beahvior! Ideally speaking there should not be any issue when used either WebActivity or HTTP Linked Service
Did you try to test the company API endpoint outside of ADF, Such as POSTMAN ? "I can successfully get data from an API with the Web activity, but not with the HTTP (or REST*) service."***
The HTTP activity requires a Linked Service and a Dataset, while the Web activity only requires a URL endpoint.
"I setup the same endpoint with the base URL in a Linked Service & a relative URL in a Source Dataset and I consistently get 403 Forbidden*, or** 400 Bad Request errors."*
Please do add screenshots of the Baseurl and relative URL when passed to the HTTP LS/DS.
How are you passing this relative URL in the Web-Activity?
Some Further readings:
The difference among this HTTP connector, the REST connector and the Web table connector are:
- REST connector specifically support copying data from RESTful APIs; Specifically, this generic REST connector supports:
- Copying data from a REST endpoint by using the GET or POST methods and copying data to a REST endpoint by using the POST, PUT or PATCH methods.
- Copying data by using one of the following authentications: Anonymous, Basic, Service Principal, OAuth2 Client Credential, System Assigned Managed Identity and User Assigned Managed Identity.
- Pagination in the REST APIs.
- For REST as source, copying the REST JSON response as-is or parse it by using schema mapping. Only response payload in JSON is supported.
To test a request for data retrieval before you configure the REST connector in Data Factory, learn about the API specification for header and body requirements. You can use tools like Postman or a web browser to validate.
- HTTP connector is generic to retrieve data from any HTTP endpoint, e.g. to download file. Before REST connector becomes available, you may happen to use the HTTP connector to copy data from RESTful APIs, which is supported but less functional comparing to REST connector. You can use this HTTP connector to:
- Retrieve data from an HTTP/S endpoint by using the HTTP GET or POST methods.
- Retrieve data by using one of the following authentications: Anonymous, Basic, Digest, Windows, or ClientCertificate.
- Copy the HTTP response as-is or parse it by using supported file formats and compression codecs.
To test an HTTP request for data retrieval before you configure the HTTP connector, learn about the API specification for header and body requirements. You can use tools like Postman or a web browser to validate.
- Web table connector extracts table content from an HTML webpage.
- REST connector specifically support copying data from RESTful APIs; Specifically, this generic REST connector supports:
-
QuantumCache 20,676 Reputation points Moderator
2024-02-15T18:33:08.8933333+00:00 Hello @Sam Lohbeck Just checking if we are still connected on this discussion? Please let us know if you need to add more info so that we better assist you!
-
QuantumCache 20,676 Reputation points Moderator
2024-02-21T04:10:58.8866667+00:00 Hello @Sam Lohbeck Please do let us know if you need any further help in this regard, just wanted to check if the issue got resolved?
Sign in to comment
Sign in to answer