Dataflow not using the Base and Relative URLs in Dataset

Question

I've set Base and Relative URLs in REST Linked Service and Dataset. These are read correctly by the dataset, which connects successfully. However, Dataflows do not read the Base and Relative URLs correctly from Dataset / Linked Service (only reads the Base URL (https://api.stripe.com/v1), not Relative URL (customers)). Here's the error (should be GET: /v1/customers):

Failure to read most recent page request: DF-REST_001 - Error response from server:

Some({  "error": {"message": "Unrecognized request URL (GET: /v1). Please see https://stripe.com/docs or we can help at https://support.stripe.com/.","type": "invalid_request_error"  }}), Status code: 404. Please check your request url and body. (url:https://api.stripe.com/v1,request body: None, request method: GET)

As I said, the dataset connection is successful. Before I parameterize the dataset and dataflow (as per Accepted Answer in https://learn.microsoft.com/en-us/answers/questions/1083901/how-can-i-parameterize-rest-api-dataset-or-linked), is there an obvious and solution to the problem?

Answer 1

Hi @Chris Jones
Welcome to Microsoft Q&A platform.

Thank you for sharing the details of your issue. Based on your description, it seems the Dataflow in Azure Data Factory is only using the Base URL (https://api.stripe.com/v1) and ignoring the Relative URL (customers) from your dataset configuration. This results in an incomplete endpoint and the 404 error you’re seeing.

Here’s how you can address this:

1. Ensure that the Base URL in your linked service ends with a trailing slash (/). For example: Base URL: https://api.stripe.com/v1/ Relative URL: customers Without the trailing slash, the URLs may not combine properly, leading to incorrect API requests.
2. In the Dataflow source configuration, instead of relying on the dataset to combine the Base and Relative URLs, try specifying the full URL directly:

      http://api.stripe.com/v1/customers
   

   You can achieve this dynamically by parameterizing your dataset or pipeline to construct the full URL.
3. If you’re using parameters in your dataset, ensure the parameters are being correctly passed to the Dataflow. Use expressions like @concat(BaseURL, RelativeURL) to create the full URL if needed.
4. Enable debugging in the Dataflow to see how the URLs are being resolved during execution. This can help pinpoint where the configuration is failing.

I hope this helps! Let me know if you have any further questions.

If it was helpful, please click Upvote on this post to let us know.