Azure Data Factory REST Pagination only including first row of every page

Liam Bluett 11 Reputation points

I have approximately 1890 records. My API provider only lets me grab 100 per page. As a result, I do the following:


Where $ = 1890.

So this should go through this and give me all 1890 records. However, it only gives me 19. These 19 records are the top row for every page, i.e.

Returns rows 0-99. In my output, I am receiving just row 0.

Returns rows 100-199. In my output, I am receiving just row 100.

This goes on 1890/100 = 19 times. However, I am wanting all of the data, not just the heads.


I have found a work-around, but it is quite slow. To do this I set my limit=1 (to lower the amount of data being transmitted) and then proceed to increment my offset by 1, as seen here:

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,073 questions
0 comments No comments
{count} votes

2 answers

Sort by: Most helpful
  1. Liam Bluett 11 Reputation points
    1 person found this answer helpful.

  2. MartinJaffer-MSFT 26,056 Reputation points

    Resolution was to enable collection reference in mapping part of Copy Activity

    0 comments No comments