how collect more than 1000 records useing Post method with REST API on ADF

Kathe, Siddharth S (TCS) 1 Reputation point
2022-12-09T12:02:32.327+00:00

I am trying to kusto query data from ADF using Copy Data Activity. for source I am using Rest API connector, which is collecting only 1000 records as page size limit. when I query data using web activity with $skiptoken it is showing next 1000 records. how to consolidate all the records in one csv file.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,582 questions
0 comments No comments
{count} votes

1 answer

Sort by: Most helpful
  1. ShaikMaheer-MSFT 38,546 Reputation points Microsoft Employee Moderator
    2022-12-12T09:20:22.56+00:00

    Hi @Kathe, Siddharth S (TCS) ,

    Thank you for posting query in Microsoft Q&A Platform.

    If I understand correctly, your API supports pagination, and you would like to load all pages data to Sink storage in to one csv file. Please correct me if I am wrong.

    Copy activity, REST connector as source will support pagination, you can leverage paginations techniques to query all page's data from API and load to Sink storage.

    Below are few useful resources to help with pagination in Copy activity.
    How t p Perform Pagination in ADF using REST API - Here it explained about tweaking Sink settings to get all responses in to single file.
    Pagination support in Azure data factory
    Copy data from REST API which sends response in Pages using Azure data factory

    For some reason if you are getting each page data into each file separately then try to use another copy activity on top of folder in which all files available and use wild card option to take all files and under sink use dataset for sink storage to load in to single file.

    269633-image.png

    Hope this helps. Please let me know if any further queries.


    Please consider hitting Accept Answer button. Accepted Answers help community as well.


Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.