Share via

zendesk pagination rules and multiple pages using adls gen2

Ahsan 20 Reputation points
2023-10-18T05:59:39.1+00:00

I am trying to implement pagination on my copy activity. I have 11,000 rows of data and my sink dataset is adls gen2 with format type: parquet. Currently, my pagination rules are: absoluteurl, body: next_page with maxrequestnumber = 10, which will give me 1000 records. this is because I get an error on page 11, so I paginate using date after ten pages.

I have done this exact method writing to Microsoft SQL server and got all 11,000 records to show up, however when I view my data in synapse analytics, I can only see 1000. When I paginate through the next 10 pages, it just replaces the previous records inserted. Is there a way to see all the records or another way to paginate for synapse analytics?

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,696 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,199 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. KranthiPakala-MSFT 46,442 Reputation points Microsoft Employee
    2023-10-19T23:45:47.0833333+00:00

    @Ahsan Welcome to Microsoft Q&A forum and thanks for reaching out here.

    As share by @Amira Bedhiafi and as per my past research on Zendesk API, you can use mapping data flow to implementation pagination.

    User's image

    I hope my other post by Amira will help you to get started on your requirement. But if you ever notice any issues, please feel free to open a new thread with details information about the issue and we will be happy to assist you.

    Thank you

    Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.

    0 comments