Azure Data Factory truncating Base URL after .com

Question

I'm trying to create a request to an API and it keeps giving me the following error (See below). It seems the Base URL in the Linked Service is getting truncated after ".com" so when ADF combines the Base and Relative URLs it is calling the wrong endpoint.

Example
Base URL at Linked Service is https://cloud.a.b.com/c/d
Relative URL at Dataset is /e/f
Absolute URL should be https://cloud.a.b.com/c/d/e/f

Error
Failure happened on 'Source' side. ErrorCode=RestCallFailedWithClientError,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Rest call failed with client error, status code 404 NotFound, please check your activity settings.
Request URL: https://cloud.a.b.com/e/f.
Response: {"fault":{"faultstring":"Unable to identify proxy for host: api-x-cdn and url: /e/f","detail":{"errorcode":"messaging.adaptors.http.flow.ApplicationNotFound"}}},Source=Microsoft.DataTransfer.ClientLibrary,'

Answer 1

Anything on this? I'm scoping Azure as a solution for our Data Engineering team, and it's a red flag that this problem prevents us from using Azure to integrate our necessary APIs but it hasn't been fixed on Azure's end after 1.5 years.

Same scenario here as the other two individuals have expressed.

We have a base url of:

• https://address.com/v3.1

and relative url needed is:

• /products?page=1&items_per_page=20&attributes=all

When I put these in - I get an error in my pipeline copy activity - but clearly the 'v3/1' has been removed from the base url and it's just sending https://address.com/products?page=1&items_per_page=20&attributes=all

We need this to work, or wont be able to paginate. I can get it to work by hardcoding the whole baseurl and relative url into the linked services base url, but this will only work if we have 1 page worth of data, and some of these have dozens of pages of data and require pagination - thus treating base url and relative url seprately.

Answer 2

Hi! We had a discussion with Microsoft about this.

Azure Data Factory behaviour is RFC-compliant behaviour. RFC-standard defines that relative url's SHOULD NOT start with / -character. Using / -character will cause the path to be assumed as absolute path to the domain. Standard here: https://datatracker.ietf.org/doc/html/rfc1808#section-5

So, if a REST API says it is returning a relative URL but it starts with /, this is wrong behaviour and the API should be fixed to follow standards.

Now, the question becomes that whether Data Factory allows manipulation of values retrieved from the body of a response (when reading the nextPageUrl for example, can we manipulate that before the activity uses it for pagination), if a API is not following standard behaviour. This is not currently possible, this was confirmed with the product team.

So Steven in your situation, if possible, use relative url without the leading /-character. If you are reading it from the initial requests' body, so you have no control over the value that is attempted to used dynamically in the pagination logic, then you have to request a change from the team owning the API.