Azure Data Factory REST Pagination only including first row of every page

Liam Bluett 11 Reputation points
2022-11-25T04:23:24.307+00:00

I have approximately 1890 records. My API provider only lets me grab 100 per page. As a result, I do the following:

264121-image.png

Where $.meta.total = 1890.

So this should go through this and give me all 1890 records. However, it only gives me 19. These 19 records are the top row for every page, i.e.

www.api/contacts?limit=100&offset=0
Returns rows 0-99. In my output, I am receiving just row 0.

www.api/contacts?limit=100&offset=100
Returns rows 100-199. In my output, I am receiving just row 100.

This goes on 1890/100 = 19 times. However, I am wanting all of the data, not just the heads.

EDIT:

I have found a work-around, but it is quite slow. To do this I set my limit=1 (to lower the amount of data being transmitted) and then proceed to increment my offset by 1, as seen here:
264098-image.png

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,515 questions
0 comments No comments
{count} votes

2 answers

Sort by: Most helpful
  1. Liam Bluett 11 Reputation points
    2022-11-25T06:13:58.847+00:00
    1 person found this answer helpful.

  2. MartinJaffer-MSFT 26,021 Reputation points
    2022-11-29T18:35:21.667+00:00

    Resolution was to enable collection reference in mapping part of Copy Activity

    0 comments No comments