Azure Data Factory truncating Base URL after .com

Question

Azure Data Factory truncating Base URL after .com

sidefxs 11

I'm trying to create a request to an API and it keeps giving me the following error (See below). It seems the Base URL in the Linked Service is getting truncated after ".com" so when ADF combines the Base and Relative URLs it is calling the wrong endpoint.

Example
Base URL at Linked Service is https://cloud.a.b.com/c/d
Relative URL at Dataset is /e/f
Absolute URL should be https://cloud.a.b.com/c/d/e/f

Error
Failure happened on 'Source' side. ErrorCode=RestCallFailedWithClientError,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Rest call failed with client error, status code 404 NotFound, please check your activity settings.
Request URL: https://cloud.a.b.com/e/f.
Response: {"fault":{"faultstring":"Unable to identify proxy for host: api-x-cdn and url: /e/f","detail":{"errorcode":"messaging.adaptors.http.flow.ApplicationNotFound"}}},Source=Microsoft.DataTransfer.ClientLibrary,'

KranthiPakala-MSFT 46,647 Reputation points Microsoft Employee Moderator

2022-07-22T05:02:29.777+00:00
Hello @sidefxs ,

Thanks for the question and using MS Q&A platform.

As per my understanding, you are using REST connector and in BASE URL you have configured it as https://cloud.a.b.com/c/d and your Dataset settings, Relative URL as /e/f. But when you run the pipeline, it is failing with error and from the error you noticed that the request endpoint is formed incorrectly as https://cloud.a.b.com/e/f truncating c/d from the endpoint URL. Please correct If I misunderstood it.

Are you seeing the same error message when you try to Preview the data from the dataset level? Or were you able to see the data from dataset preview but it only throws error when you debug run or trigger tun the pipeline?

Recommendations to overcome the issue:

In order to overcome this issue, could you please try to configure your Linked service to just have the actual base URL as https://cloud.a.b.com and in the dataset configure the Relative URL as c/d/e/f and that should help resolve the issue.

Or configure your linked service base URL to have the complete endpoint URL (Combination of Base URL + Relative URL for eg: https://cloud.a.b.com/c/d/e/f).

Hope this helps. Do let us know how it goes.

Thank you
sidefxs 11 Reputation points

2022-07-22T13:50:02.643+00:00

You understood correctly but those solutions do not solve my problem, I can't use pagination properly if this isn't solved.

The problem arises when debugging/running the pipeline. BTW, this happens while using a Copy Activity.
KranthiPakala-MSFT 46,647 Reputation points Microsoft Employee Moderator

2022-08-02T22:57:13.02+00:00

Hello @sidefxs ,

Thanks for confirmation. This seems to be strange. When I tried to just use the base URL in Linked service and Relative URL in Dataset configuration, I was able to get the data, and everything is working as expected. But whereas in your case the URL is getting truncated which is strange.

As this requires a deeper investigation, I recommend filing a support ticket and do share the SR# with us here for tracking it internally. And if you don't have a support plan, please let us know here.

Thank you
Tomi Asp 10 Reputation points

2023-03-02T09:10:07.7766667+00:00

Any updates to this? This is getting really annoying that simple fixes cannot be made...

If a linked service's base url is like so https://address.com/public/, if I define a dataset's relative url with /relativePath, it truncates the base url's public section away. If I on the other hand define the relative without the "/" slash as only relativePath, then it appends the relative path to the end of /public/ correctly.

The same happens with pagination. If I have relative next url coming in the body of an payload, attempting to configure a $.links.next will truncate the base path if the payload's url has a "/" slash as well starting the relative url.

Could you please make a fix that stops trying to be smart with the slashes and just treat base urls and relative urls as concatenations to each other.

2 answers

Your answer

KranthiPakala-MSFT 46,647 Reputation points Microsoft Employee Moderator

2022-07-22T05:02:29.777+00:00

Hello @sidefxs ,

Thanks for the question and using MS Q&A platform.

As per my understanding, you are using REST connector and in BASE URL you have configured it as https://cloud.a.b.com/c/d and your Dataset settings, Relative URL as /e/f. But when you run the pipeline, it is failing with error and from the error you noticed that the request endpoint is formed incorrectly as https://cloud.a.b.com/e/f truncating c/d from the endpoint URL. Please correct If I misunderstood it.

Are you seeing the same error message when you try to Preview the data from the dataset level? Or were you able to see the data from dataset preview but it only throws error when you debug run or trigger tun the pipeline?

Recommendations to overcome the issue:

In order to overcome this issue, could you please try to configure your Linked service to just have the actual base URL as https://cloud.a.b.com and in the dataset configure the Relative URL as c/d/e/f and that should help resolve the issue.

Or configure your linked service base URL to have the complete endpoint URL (Combination of Base URL + Relative URL for eg: https://cloud.a.b.com/c/d/e/f).

Hope this helps. Do let us know how it goes.

Thank you
sidefxs 11 Reputation points

2022-07-22T13:50:02.643+00:00

You understood correctly but those solutions do not solve my problem, I can't use pagination properly if this isn't solved.

The problem arises when debugging/running the pipeline. BTW, this happens while using a Copy Activity.
KranthiPakala-MSFT 46,647 Reputation points Microsoft Employee Moderator

2022-08-02T22:57:13.02+00:00

Hello @sidefxs ,

Thanks for confirmation. This seems to be strange. When I tried to just use the base URL in Linked service and Relative URL in Dataset configuration, I was able to get the data, and everything is working as expected. But whereas in your case the URL is getting truncated which is strange.

As this requires a deeper investigation, I recommend filing a support ticket and do share the SR# with us here for tracking it internally. And if you don't have a support plan, please let us know here.

Thank you
Tomi Asp 10 Reputation points

2023-03-02T09:10:07.7766667+00:00

Any updates to this? This is getting really annoying that simple fixes cannot be made...

If a linked service's base url is like so https://address.com/public/, if I define a dataset's relative url with /relativePath, it truncates the base url's public section away. If I on the other hand define the relative without the "/" slash as only relativePath, then it appends the relative path to the end of /public/ correctly.

The same happens with pagination. If I have relative next url coming in the body of an payload, attempting to configure a $.links.next will truncate the base path if the payload's url has a "/" slash as well starting the relative url.

Could you please make a fix that stops trying to be smart with the slashes and just treat base urls and relative urls as concatenations to each other.

Answer 1

Hi! We had a discussion with Microsoft about this.

Azure Data Factory behaviour is RFC-compliant behaviour. RFC-standard defines that relative url's SHOULD NOT start with / -character. Using / -character will cause the path to be assumed as absolute path to the domain. Standard here: https://datatracker.ietf.org/doc/html/rfc1808#section-5

So, if a REST API says it is returning a relative URL but it starts with /, this is wrong behaviour and the API should be fixed to follow standards.

Now, the question becomes that whether Data Factory allows manipulation of values retrieved from the body of a response (when reading the nextPageUrl for example, can we manipulate that before the activity uses it for pagination), if a API is not following standard behaviour. This is not currently possible, this was confirmed with the product team.

So Steven in your situation, if possible, use relative url without the leading /-character. If you are reading it from the initial requests' body, so you have no control over the value that is attempted to used dynamically in the pagination logic, then you have to request a change from the team owning the API.

Hulshof E. (Eric) 0 Reputation points

2024-03-06T09:12:32.8466667+00:00

Hey Tomi,

Did you figure out a way to remove the / at the start? We are currently facing the same issue.

Answer 2

Anything on this? I'm scoping Azure as a solution for our Data Engineering team, and it's a red flag that this problem prevents us from using Azure to integrate our necessary APIs but it hasn't been fixed on Azure's end after 1.5 years.

Same scenario here as the other two individuals have expressed.

We have a base url of:

https://address.com/v3.1

and relative url needed is:

/products?page=1&items_per_page=20&attributes=all

When I put these in - I get an error in my pipeline copy activity - but clearly the 'v3/1' has been removed from the base url and it's just sending https://address.com/products?page=1&items_per_page=20&attributes=all

We need this to work, or wont be able to paginate. I can get it to work by hardcoding the whole baseurl and relative url into the linked services base url, but this will only work if we have 1 page worth of data, and some of these have dozens of pages of data and require pagination - thus treating base url and relative url seprately.

Hulshof E. (Eric) 0 Reputation points

2024-03-06T09:13:27.11+00:00

Hey Steven

Did you figure out a way to remove the / at the start? We are currently facing the same issue.
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.

Share via

Azure Data Factory truncating Base URL after .com

2 answers

Your answer