How to use continuation token received from Pipeline Runs - Query By Factory API in ADF copy activity pagination

Question

We are trying to use the continuation token received by Pipeline Runs - Query By Factory API in pagination rule of copy activity in ADF but it does not work as the continuation token must be passed as part of Body and not as query parameter. How to achieve copying all pages data without implementing a loop?

Answer 1

Based on an old thread :

Currently the Endpoint PipeLine runs - Query By Factory supports pagination through the Request Body.

The continuation token is taken as part of the request body and not through the Query Parameters / headers.

That is the reason in the above scenario - the data (first page) kept repeating - as the Continuation token was not being honored as it was passed as headers.

Unfortunately, at this point of time - pagination for this endpoint (Pipeline Runs - Query By Factory) cannot be achieved via headers/queryparameters/abosolute uri - which are the only supported form of pagination for the copy activity.

Having said that - one workaround I can think of :

Step 1 : Initialize a variable ContinuationToken as blank**
Step 2 :** Perform Copy activity with the ContinuationToken value (with merge behavior)**
Step 3 :** Perform Web activity
Step 4 : Set the variable ContinuationToken with continuationtoken obtained from the Web activity.
Step 5 : Repeat steps 2-4 using the Until activity - until the continuationtoken is not returned.

The end goal is to perform the copy activity in an iterative way with a different Request body for getting next pages of the data until ContinuationToken is returned blank. The above is just for demonstration, you can optimize the logic as per your convenience.

Note :
When you are passing ContinuationToken in the response body. Ensure, you pass the LastUpdatedAfter and LastUpdatedBefore - these are mandatory parameters of the request body without which the data will be returned blank.

https://www.appsloveworld.com/azure-data-factory/8/an-example-of-continuationtoken-with-pipeline-runs-query-by-factory

Answer 2

Hi Sujal Mandal,

Thank you for posting query in Microsoft Q&A Platform.

Since continuation token is part of request body, at this point of time we cannot directly use pagination rules. We have to implement via looping only.

But its really good feature to consider. Thanks for sharing this. I internally forwarded this feedback to internal team. I would request you to as well log this as feedback item in below portal and up-vote it. Product team consider feedbacks for furture implementations.

https://feedback.azure.com/d365community/forum/1219ec2d-6c26-ec11-b6e6-000d3a4f032c?ref=blog.ippon.fr

Hope this helps. Please let me know if any further queries.

---

Please consider hitting Accept Answer button. Accepted answers help community as well. Thank you.