zendesk rest api pagination rules and multiple pages in data factory

Ahsan 20 Reputation points
2023-09-25T20:00:52.75+00:00

Using pagination rules absoluteURL with values body and next_page, i can get upto 1000 records (10 pages), but i get this error: API error - Invalid search: Requested response size was greater than Search Response Limits on page 11. It maxes out at 10 pages. How do i get all records to show up?

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,639 questions
0 comments No comments
{count} votes

Accepted answer
  1. Amira Bedhiafi 33,631 Reputation points Volunteer Moderator
    2023-09-26T13:50:19.76+00:00

    Zendesk typically returns links for pagination in the response under next_page.

    The next_page attribute provides a direct link to the next page of results. If there's no next_page link, you're on the last page of results.

    Don't forget that Zendesk has a rate limit. You should ensure you're not hitting this rate limit, which can cause errors. Check the X-Rate-Limit header in the API response to understand where you stand.

    In your case I assume you'll have to make multiple "sets" of paginated requests and also instead of relying only on pagination, use filters, such as updated_at or created_at to break down your requests.

    1 person found this answer helpful.

1 additional answer

Sort by: Most helpful
  1. ShaikMaheer-MSFT 38,546 Reputation points Microsoft Employee Moderator
    2023-09-26T16:37:06.92+00:00

    Hi Ahsan Aziz,

    Thank you for posting query in Micrsoft Q&A Platform.

    As mentioned in Amira's response, kindly try direct next_page URL. Below my videos on pagination will help you, how to configure same in pagination. Kindly check them and implement similar.

    Copy activity - Pagination rules - When API response have URL for next page

    Copy activity - Pagination rules - Variables in QueryParameters in Azure Data Factory

    Hope this helps. Please let me know if any further queries.


    Please consider hitting Accept Answer button. Accepted answers help community as well.


Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.