Azure Data Factory pagination issue

Question

I'm struggling with a pagination issue in a Copy Data activity fetching some data from a REST API.

The API gives me a response with the total row count of all pages, the start rownumber of this page, and the rowcount on this page.

I can correctly fetch all data from some endpoints using the RANGE in a QueryParameter with settings start = 0 & offset = 500. But on some endpoints this is not working, because the number of rows on the page is not always the same. Apparently, some pages are too large to show 500 rows and the rowcount is randomly lower, eg. 248 or 137 for some pages. So I need the offset parameter in the RANGE to be dynamic, being the start + count from the response body.

But I can not get this to work in ADF.

I hope someone can help me with this.

Best regards

Answer 1

Hello @Joost Romijn,

Did you try any expression to control the dynamic values for the start and Offset?

Or Did you try to use 1) WebActivity, to fetch the Start & Total, Count values into variables 2) Until Activity, which will do Web Activity & Copy Activity