REST Pagination in a Data Flow

ralphjjr 106 Reputation points
2022-01-18T17:51:45.967+00:00

I'm trying to get pagination working in a data flow. It works great in a copy activity.
These are my settings:

166106-screen-shot-2022-01-18-at-124614-pm.png

When I try the same thing in a Data Flow. It just doesn't work. I've tried a bunch of variations and can't seem to get it to actually paginate. It will only ever get the first page.

166132-screen-shot-2022-01-17-at-71929-pm.png

I've tried many variations of values. The short explanation is, it's a Query Parameter called "starting_after" and it should get the value of body.data[99].id
In the Data Flow I did already notice that arrays start at 1, and not at 0, so I have data[100].id - but it doesn't even paginate, so that doesnt seem to be the issue. It only ever gets one page. I even adjusted Request Interval, to see if perhaps that somehow would get it going.

What am I missing?

Here is an example of the response.

{  
    "object": "list",  
    "data": [  
        {  
            "id": "evt_1",  
            "object": "event",  
            "api_version": "2018-02-28",  
            "created": 1642505475,  
            "data": {  
                "object": {  
                    "id": "sub_1",  
                    "object": "subscription",  
                    "application_fee_percent": null  
                },  
                "previous_attributes": {  
                    "current_period_end": 1642435365,  
                    "current_period_start": 1637856965  
                }  
            },  
            "livemode": false,  
            "pending_webhooks": 0,  
            "request": {  
                "id": null,  
                "idempotency_key": null  
            },  
            "type": "customer.subscription.updated"  
        },  
        {  
            "id": "evt_2",  
            "object": "event",  
            "api_version": "2018-02-28",  
            "created": 1642532021,  
            "data": {  
                "object": {  
                    "id": "sub_2",  
                    "object": "subscription",  
                    "application_fee_percent": null  
                },  
                "previous_attributes": {  
                    "current_period_end": 1642331884,  
                    "current_period_start": 1652445484  
                }  
            },  
            "livemode": false,  
            "pending_webhooks": 0,  
            "request": {  
                "id": null,  
                "idempotency_key": null  
            },  
            "type": "customer.subscription.updated"  
        }  
    ],  
    "has_more": true,  
    "url": "/v1/events"  
}  
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,348 questions
{count} vote