How to retrieve the continuationToken from Rest API while using Azure Data Factory?

HEMANATH GOUD KANDULA 41 Reputation points

I'm trying to implement Azure Data Factory's Copy Activity to copy data from an API to Azure Blob Storage. I have set up the source & sinks correctly so that when I trigger the pipeline it pulls and loads the first load of data but I am struggling with pagination.

When I trigger the pipeline It loads the first page correctly and afterward, it doesn't return the next continuation token for fetching the data from API. If I use it until or for-each activity the pipeline copies the data of the same continuation token endless until timeout.

When I run the Rest API call in Postman it returns the data and also the next continuation token as well. The continuation Token will be like 0000xxxx-00000-xxx00-000000xx000000 and the next continuation token is like 0000xxyy-00000-xxx00-000000yy000000.

My goal is to retrieve the data from Rest API using a continuation token and retrieve the next continuation token so that I can retrieve the next page data until the continuation token is null and store it in Azure Blob Storage with Azure Data Factory Pipeline.

I am able to retrieve the access token from the rest API and only the first page data.

Is there any way to solve this issue please let me know?

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
7,173 questions
{count} votes