Substring inside pagination in ADF

sam nick 286 Reputation points
2023-07-25T18:26:44.3033333+00:00

Hi,

I am trying to retrieve data from a rest source, but the pagination is in the header as a link but when i pass it in the parameters the additional fields are causing a failure. How can i

  1. use substring for the header
  2. capture another parameter from the header as i need to use that value for a loop

Screenshot 2023-07-25 142222

User's image

for 1, how can i pass only the string of the Link that stops at 301 and does not read/pass the "rel=next" value

For 2, how to capture the x-ratelimit_remaining as i need to stop the pipeline before it gets to 0, that parameters keeps on decreasing after every GET.

thank you.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,149 questions
{count} votes

1 answer

Sort by: Most helpful
  1. ShaikMaheer-MSFT 37,231 Reputation points Microsoft Employee
    2023-07-26T14:59:56.05+00:00

    Hi

    Thank you for posting query in Microsoft Q&A Platform.

    Pagination rules for headers are documented in below link and also various other scenarios as well for pagination. Kindly check below documentation and see if that helps.

    https://learn.microsoft.com/en-us/azure/data-factory/connector-rest?tabs=data-factory#example-3variables-in-headers

    Please let me know how it goes. Thank you.