Substring inside pagination in ADF

Question

Hi,

I am trying to retrieve data from a rest source, but the pagination is in the header as a link but when i pass it in the parameters the additional fields are causing a failure. How can i

1. use substring for the header
2. capture another parameter from the header as i need to use that value for a loop

for 1, how can i pass only the string of the Link that stops at 301 and does not read/pass the "rel=next" value

For 2, how to capture the x-ratelimit_remaining as i need to stop the pipeline before it gets to 0, that parameters keeps on decreasing after every GET.

thank you.

Answer 1

Hi

Thank you for posting query in Microsoft Q&A Platform.

Pagination rules for headers are documented in below link and also various other scenarios as well for pagination. Kindly check below documentation and see if that helps.

https://learn.microsoft.com/en-us/azure/data-factory/connector-rest?tabs=data-factory#example-3variables-in-headers

Please let me know how it goes. Thank you.