ADF Pagination for Rest API

Question

Hi ,

I am struggling to setup pagination in ADF for Company house REST API . it loads only 100 records into blob out of 223 .

How to setup pagination for below output in ADF Copy Activity Source

Below is the REST API output:

"items_per_page": 25,
    "start_index": 0,
    "total_count": 223,
    "filing_history_status": "filing-history-available",
    "items": [
        {
            "category": "mortgage",
            "date": "2023-01-17",
            "description": "mortgage-charge-part-release-with-charge-number",
            "description_values": {
                "charge_number": "018040470013"
            },
            "links": {
                "self": "/company/01804047/filing-history/xxxxxxx",
                "document_metadata": "https://frontend-doc-api.company-information.service.gov.uk/document/1OLR87fVn9Wfe5c9DeXB2IAydUEe3_AWJZedEYksuzQ"
            },
            "subcategory": "release-cease",
            "type": "MR05",
            "pages": 2,
            "barcode": "XBVDR0I9",
            "transaction_id": "XXXXXXXXXXX"
        },
        {
            "action_date": "2022-09-19",
            "category": "confirmation-statement",
            "date": "2022-10-21",
            "description": "confirmation-statement-with-no-updates",
            "description_values": {
                "made_up_date": "2022-09-19"
            },
            "links": {
                "self": "/company/01804047/filing-history/MzM1NjEwMzYxMWFkaXF6a2N4",
                "document_metadata": "https://frontend-doc-api.company-information.service.gov.uk/document/rBjAHspO5p7ULoGUZShXFqx8rzE2qRzYpUENVm8pL24"
            },
            "type": "CS01",
            "pages": 3,
            "barcode": "XBF2LMF5",
            "transaction_id": "XXXXXXXXXXXXXXXXXXXXXXXX"
        },

Answer 1

Hi srikanth Battula,

Thank you for posting query in Microsoft Q&A Platform. Unfortunately, your question doesn't have full details of your response or how exactly your API can give you next page. But still let me try to help you here.

Based on your API response you can set pagination in Copy activity. If your Api contains a property in json that gives next page URL or next page query parameter value accordingly, you need to set that in pagination rules.

Below link from MS official documentation has all kinds of scenarios with examples covered for pagination. Kindly go through them. That will help you surely.

https://learn.microsoft.com/en-us/azure/data-factory/connector-rest?tabs=data-factory#pagination-rules-examples

OR

Also, you can achieve pagination if your URL can be replaced with some values using foreach activity. Please check below video to understand that scenario and see if that suits you.

Copy data from REST API which sends response in Pages using Azure data factory

Hope this helps. Please let me know if any further queries.

---

Please consider hitting Accept Answer button. Accepted answers help community as well. Thank you.