Rest API recursive get Request in Azure synapse pipeline

Bhoopathiraju Deepthi 66 Reputation points
2021-07-29T10:13:30.467+00:00

I need to copy the data from a rest API end point using azure synapse. By default the response limit is set to 1000. i need to make a recursive get request using one of the response header value in the request until all the response is copied.Any suggestions how to do that please?

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,456 questions
{count} votes

Accepted answer
  1. MartinJaffer-MSFT 26,036 Reputation points
    2021-08-02T20:46:15.677+00:00

    To calculate how many pages, and generate an array to iterate over:

    Variables:
    119936-image.png

    In your case, totalrecords would get replaced by the count from the web activity you showed me.
    @{activity('web1').output.response.count}
    The perpage would be 500, as that is the default in documentation. Or whatever you choose it to be.

    Make page array:
    119981-image.png

    @Rover (0,div(int(variables('totalrecords')),int(variables('perpage'))))
    number of pages = totalrecords / perpage. This makes that many elements in an array.

    Iterate over pagearray to keep track of how many times:
    119955-image.png

    119991-image.png


1 additional answer

Sort by: Most helpful
  1. Ziani_Az 21 Reputation points
    2021-09-09T07:09:04.107+00:00

    Hi @ShaikMaheer-MSFT ,
    How create and update statistics in Azure Synapse Analytics,
    Thank you

    0 comments No comments