How to handle pagination with $skipToken in Azure Data Factory

Question

I'm curious how to handle pagination with $skipToken in Azure Data Factory

According to the documentation (Pagination support) it seems quite simple in case when we have @odata.nextLink to use as next URL (Microsoft Graph API), but it's more complicated when it comes with $skipToken, that we need to insert in next POST body (Azure Resource Graph):

POST https://management.azure.com/providers/Microsoft.ResourceGraph/resources?api-version=2021-03-01

{
    "query": "Resources",
    "options": {
      "$skipToken": "eyAibm8iOiAibHVjayIsICJidXQiOiAibmljZSIsICJ0cnkiOiAiISIgfQ=="
    }
}

I thought about Copy activity with Until loop, but I couldn't find a way to grab $skipToken from previous activity.

Is there any simple solution for that or any workaround?

Answer 1

You can't reference one Until iteration from the next, but could you store the skip token in a pipeline variable at the end of each iteration? The variable would always hold the latest skip token, ready for use in the next iteration.