Cant get all the data from rest api to Sql using adf

Question

HI

I am trying to load data from Rest ApI into Azure Sql db using ADF. Its only showing 100 rows in sql table.

Preview data Snippet

"metadata": {
      "next_page": "/feed/inspections?archived=false&completed=true&web_report_link=private&limit=100&modified_after=2018-01-15T15%3A10%3A04.868Z",
      "remaining_records": 345368
    },

How should I setup the pagination to get entire data

regards,

Naren

Answer 1

Hi @Sharma, Narinder ,

Welcome to Microsoft Q&A forum and thanks for reaching out here.

By looking at the above sample response body, noticed that the response contains a metadata object with a next_page field that provides the URL for the next page of data, hence you can set the Pagination rules option to "AbsoluteUrl": "metadata.next_page"

Below is how your pagination rule configuration should look like:

Note: Assuming that "metadata.next*page" contains the complete (well-formed) URL of the next page data. In case if your next__page field only contains relative URL of the next page data then please share complete response body? If that's the case also please confirm if your response body contains base URL , if yes, provide a sample.

In addition, please confirm if your API is a public or private? If public could you please point me to that documentation of that API so that I can provide more accurate information about the implementation. In general if your response body contains only the reltive URL of the next page data instead of the fill URL then you will have to use dynamic expression to create a well formed Absolute URL using your response body.
*
Hope this helps. Let me know if you have further questions

---

Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.