How to retrieve the continuationToken from Rest API while using Azure Data Factory?

Question

How to retrieve the continuationToken from Rest API while using Azure Data Factory?

HEMANATH GOUD KANDULA 41

I'm trying to implement Azure Data Factory's Copy Activity to copy data from an API to Azure Blob Storage. I have set up the source & sinks correctly so that when I trigger the pipeline it pulls and loads the first load of data but I am struggling with pagination.

When I trigger the pipeline It loads the first page correctly and afterward, it doesn't return the next continuation token for fetching the data from API. If I use it until or for-each activity the pipeline copies the data of the same continuation token endless until timeout.

When I run the Rest API call in Postman it returns the data and also the next continuation token as well. The continuation Token will be like 0000xxxx-00000-xxx00-000000xx000000 and the next continuation token is like 0000xxyy-00000-xxx00-000000yy000000.

My goal is to retrieve the data from Rest API using a continuation token and retrieve the next continuation token so that I can retrieve the next page data until the continuation token is null and store it in Azure Blob Storage with Azure Data Factory Pipeline.

I am able to retrieve the access token from the rest API and only the first page data.

Is there any way to solve this issue please let me know?

KranthiPakala-MSFT 39,007 Reputation points Microsoft Employee

2022-05-03T04:45:19.807+00:00

Hello @HEMANATH GOUD KANDULA ,

Thanks for the question and using MS Q&A platform.

Was the continuation token part of the request body or should be passed through the Query Parameters / headers? Appreciate if you could share a sample Rest API response structure so that we can try to reproduce on our end.

But here is a sample where a similar requirement has been discussed a detailed solution is provided by one of our engineers. I would recommend having a look at it and see if it helps.

Related MS Q&A thread: How to use pagination in Azure Data factory with REST API

Hope this helps.

Thanks
HEMANATH GOUD KANDULA 41 Reputation points

2022-05-03T05:15:13.427+00:00

This is the example of the ContinuationToken which will be used in the pipeline.
It just returns the initial continuation token 00000002-0000-0000-6400-000000000000 and it doesn't return the next token.

It has become a difficult task for me.

v1/legalentities/100214/employees?continuationToken=00000002-0000-0000-6400-000000000000
KranthiPakala-MSFT 39,007 Reputation points Microsoft Employee

2022-05-17T20:00:12.643+00:00

Hi @HEMANATH GOUD KANDULA ,

I'm a bit confused when you say that the continuation is token is returned when you tested from POSTMAN but not from ADF copy activity. Are you expecting the continuation token to be returned in the body like in the picture below?

Appreciate if you could share the sample Response (please mask any sensitive information) that you have received from the POSTMAN call.

Thank you
KranthiPakala-MSFT 39,007 Reputation points Microsoft Employee

2022-05-18T22:38:10.397+00:00

Hi @HEMANATH GOUD KANDULA ,

Following up to see if you have got a chance to see my previous comment if so, could you please clarify the points highlights?

Thank you

KranthiPakala-MSFT 39,007 Reputation points Microsoft Employee

2022-05-03T04:45:19.807+00:00

Hello @HEMANATH GOUD KANDULA ,

Thanks for the question and using MS Q&A platform.

Was the continuation token part of the request body or should be passed through the Query Parameters / headers? Appreciate if you could share a sample Rest API response structure so that we can try to reproduce on our end.

But here is a sample where a similar requirement has been discussed a detailed solution is provided by one of our engineers. I would recommend having a look at it and see if it helps.

Related MS Q&A thread: How to use pagination in Azure Data factory with REST API

Hope this helps.

Thanks
HEMANATH GOUD KANDULA 41 Reputation points

2022-05-03T05:15:13.427+00:00

This is the example of the ContinuationToken which will be used in the pipeline.
It just returns the initial continuation token 00000002-0000-0000-6400-000000000000 and it doesn't return the next token.

It has become a difficult task for me.

v1/legalentities/100214/employees?continuationToken=00000002-0000-0000-6400-000000000000
KranthiPakala-MSFT 39,007 Reputation points Microsoft Employee

2022-05-17T20:00:12.643+00:00

Hi @HEMANATH GOUD KANDULA ,

I'm a bit confused when you say that the continuation is token is returned when you tested from POSTMAN but not from ADF copy activity. Are you expecting the continuation token to be returned in the body like in the picture below?

Appreciate if you could share the sample Response (please mask any sensitive information) that you have received from the POSTMAN call.

Thank you
KranthiPakala-MSFT 39,007 Reputation points Microsoft Employee

2022-05-18T22:38:10.397+00:00

Hi @HEMANATH GOUD KANDULA ,

Following up to see if you have got a chance to see my previous comment if so, could you please clarify the points highlights?

Thank you