REST api with Offset & limit Data fetching problem

Question

REST api with Offset & limit Data fetching problem

Shreyash Choudhary 126

Hi, i need to fetch full data from api i have api url with offset and limit in link itself (in order as 1st offset then limit), i need to fetch nearly 1 lakh (assumeing need to fetch all data what api is sending no limit ,means dont have any max no) records and store it into ADLS also need to increase offset ,i am not able to think much ,required suggestions , api return only data no header

all data need to be stored in adls in one file

please help

1 answer

Your answer

Answer 1

ShaikMaheer-MSFT 38,546 Microsoft Employee Moderator

Hi @Shreyash Choudhary ,

Thank you for posting query in Microsoft Q&A Platform.

Pagination settings in copy activity will help you here.

Kindly go through below official link of pagination settings, there variety of use cases covered. You use case also fit in one of them.
Pagination support in Copy activity

If it suits, you can consider watching below video as well, there I explained using ForEach activity for fetching data in pages.
Copy data from REST API which sends response in Pages using Azure data factory

Hope this helps. Please let me know if any further queries.

----------

Please consider hitting Accept Answer button. Accepted answers help community as well.

Shreyash Choudhary 126 Reputation points

2022-12-19T05:54:27.96+00:00

Wat i observed is api which we are using to fetch data only gives data ,no info regarding next page in but api url has offset and limit in this case how to make pipeline which can fetch data till api end, also we don't know when api will end sending data in this case which pagination rules need to use exactly.

i tried with the above links, i am able to fetch data but not able to end means its is running infinitely!!
ShaikMaheer-MSFT 38,546 Reputation points Microsoft Employee Moderator

2022-12-22T09:38:25.877+00:00

Since API is not providing info regarding next page or URL, you can consider having some offset and limit values stored in some table or file or array variable and loop that array and inside loop make API call dynamically by taking offset and limit values dynamically.

In below video similar approach is explained. Kindly check and follow same according to your scenario.
Copy data from REST API which sends response in Pages using Azure data factory

Hope this helps.

------------

Please consider hitting Accept Answer button. Accepted Answers help community as well.

Share via

REST api with Offset & limit Data fetching problem

1 answer

Your answer