solved, azure bug:
https://stackoverflow.com/questions/64049193/while-copying-data-from-api-using-azure-data-factory-to-azure-sql-im-only-gett
Azure Data Factory REST Pagination only including first row of every page
I have approximately 1890 records. My API provider only lets me grab 100 per page. As a result, I do the following:
Where $.meta.total = 1890.
So this should go through this and give me all 1890 records. However, it only gives me 19. These 19 records are the top row for every page, i.e.
www.api/contacts?limit=100&offset=0
Returns rows 0-99. In my output, I am receiving just row 0.
www.api/contacts?limit=100&offset=100
Returns rows 100-199. In my output, I am receiving just row 100.
This goes on 1890/100 = 19 times. However, I am wanting all of the data, not just the heads.
EDIT:
I have found a work-around, but it is quite slow. To do this I set my limit=1 (to lower the amount of data being transmitted) and then proceed to increment my offset by 1, as seen here:
2 answers
Sort by: Most helpful
-
-
MartinJaffer-MSFT 26,021 Reputation points
2022-11-29T18:35:21.667+00:00 Resolution was to enable collection reference in mapping part of Copy Activity