Is automatic pagination of ADF REST API work from ADF supported?

Question

I'm trying to retrieve activity_runs via the REST API from an ADF pipeline, but my attempts to make it automatically iterate over pages (without using loops) fails, since the continuationToken in the response's body has to be passed into the request's body. After searching for a bit I found this answer this link that claims that it was unsupported three years ago.

Is this still the case? And, if not, how can I configure the Copy-Activity to make it work?

Answer 1

As you mentioned, when using the ADF REST API to retrieve logs, metrics, or data like activity runs, the responses are paginated and typically include a continuationToken which must be manually included in subsequent API calls to retrieve additional pages of data.

To handle it, you generally need to implement a loop within your code to check for the presence of a continuationToken and then make additional API requests accordingly. Here’s a basic outline of how you might handle pagination in a scripting or programming environment:

1. Make the initial API request to start retrieving data.
2. Check the API response for a continuationToken.
3. If a continuationToken is present, use it to make the next API request.
4. Repeat steps 2 and 3 until there is no continuationToken in the response, indicating that all pages have been retrieved.

Since you mentioned trying to avoid manual looping, if you are using a platform or programming environment that supports asynchronous programming or workflows, you could potentially wrap this process into a more automated, iterative function that manages the pagination logic more elegantly. However, this would still technically involve looping, just abstracted away from the main flow of your code.

Try to watch these videos :

https://www.youtube.com/watch?v=EbURwy2a8dM

https://www.youtube.com/watch?v=n8e_exWMH5k