Issue in Reading Paginated API using ADF

Rathesh Varadarajan 1 Reputation point
2022-06-14T12:39:24.987+00:00

I am trying to read the data from a paginated API using ADF copy activity. While accessing the API at first time, only the base URL is used(without any query params). When API returns the data, it will have a key called cursor with some token. This has to be passed as query string to get the next page. How to implement this using ADF?. How to append "?cursor=" in the base URL and then add cursor value from API response

First Page : {{Base URL}} #returns a JSON with a key called cursor
From Second Page : {{Base URL}}?cursor = {{value of cursor key in JSON Response}}

I have tried using Query Parameters inside the Copy Activity of pagination. But it is not working

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,206 questions
{count} votes

1 answer

Sort by: Most helpful
  1. MartinJaffer-MSFT 26,221 Reputation points
    2022-06-23T22:02:11.167+00:00

    You do not need to include the "? in the QueryParameter name. it should look more like cursor and the body value should be the path to the cursor. See examples below @Rathesh Varadarajan

    {  
        "cursor":"abc123",  
        "date":"2022-06-23",  
        "data":[1,2,3,4]  
    }  
      
    $.cursor  
      
    -----  
      
    {  
        "metadata": {  
          "cursor":"abc123",  
          "date":"2022-06-23"  
        },  
        "data":[1,2,3,4]  
    }  
      
    $.metadata.cursor  
    
    0 comments No comments

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.