In your pipeline, you should parameterize the marker so that you can dynamically insert the next marker value into your API calls, so create a pipeline parameter to store the initial marker value.
Then configure the REST dataset to accept a marker as a query parameter. Your URL will look something like api.com/orders?next=500&marker=@{dataset().marker}
.
You may need also to use an Until activity to handle the pagination loop. The Until activity will continue to make API calls until there is no next marker returned by the API.
Inside the Until activity, use a Web activity to call the API and store the response. Then, use a Set Variable activity to extract the marker from the response JSON and set it as the new value for the pipeline parameter.
Inside the Until activity, you also need to add a condition to check if the marker is present in the response. If not, the loop should end.
Finally, chain the Web activity and Set Variable activity inside the Until activity, ensuring that the Set Variable activity is contingent on the Web activity's success.
How to use markers from API response body as pagination query parameter in Azure Data Factory
A REST API that I am using a as source dataset in ADF uses markers in the query parameter for pagination purposes.
It works similar as an absoluteURL because in the JSON response of the API call, it provides a key named "marker" which contains the marker which can be used in the query parameter of the next API call.
The difference however is that this marker is numeric and not indeed a URL.
It does not work like a page count either (it is a large numeric value that I don't have a range for).
I am looking for a way to implement this form of pagination in ADF but none of the options seem to work.
API sample request: api.com/orders?next=500&marker=00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000169439628988400000000000000000506
The key in the response:
I have tried most combinations but this is the way that I would expect it to work:
This returns an error of "Invalid PaginationRule" however.
I have tried it with adding the marker value as a parameter to the REST dataset as well as using {marker} in the relative url.
Would love to hear if there is an option of implementing this form of pagination
-
Amira Bedhiafi 28,381 Reputation points
2024-01-16T16:20:32.8266667+00:00