zendesk rest api pagination rules and multiple pages in data factory

Question

zendesk rest api pagination rules and multiple pages in data factory

Ahsan 20

Using pagination rules absoluteURL with values body and next_page, i can get upto 1000 records (10 pages), but i get this error: API error - Invalid search: Requested response size was greater than Search Response Limits on page 11. It maxes out at 10 pages. How do i get all records to show up?

0 comments

Answer accepted by question author

Amira Bedhiafi 43,046 MVP Volunteer Moderator

Zendesk typically returns links for pagination in the response under next_page.

The next_page attribute provides a direct link to the next page of results. If there's no next_page link, you're on the last page of results.

Don't forget that Zendesk has a rate limit. You should ensure you're not hitting this rate limit, which can cause errors. Check the X-Rate-Limit header in the API response to understand where you stand.

In your case I assume you'll have to make multiple "sets" of paginated requests and also instead of relying only on pagination, use filters, such as updated_at or created_at to break down your requests.

Ahsan 20 Reputation points

2023-09-27T05:30:41.9333333+00:00

I want all records in my table in mssms. I have over 100,000 records total. relying on filters such as updated_at or created_at would not allow me to add all records into the table. Currently I am trying to use cursor based incremental export. I have the current url: api/v2/incremental/tickets/cursor.json?start_time=1690919546, and i see the results in preview data, and when i debug it is successful, but nothing is showing up in the table where the data is supposed to go (my sink).
Ahsan 20 Reputation points

2023-09-27T16:36:56.08+00:00

I was able to make it work by using created_at and order by asc multiple times

1 additional answer

Your answer

Ahsan 20 Reputation points

2023-09-27T05:30:41.9333333+00:00

I want all records in my table in mssms. I have over 100,000 records total. relying on filters such as updated_at or created_at would not allow me to add all records into the table. Currently I am trying to use cursor based incremental export. I have the current url: api/v2/incremental/tickets/cursor.json?start_time=1690919546, and i see the results in preview data, and when i debug it is successful, but nothing is showing up in the table where the data is supposed to go (my sink).
Ahsan 20 Reputation points

2023-09-27T16:36:56.08+00:00

I was able to make it work by using created_at and order by asc multiple times

Answer 1

ShaikMaheer-MSFT 38,631 Microsoft Employee Moderator

Hi Ahsan Aziz,

Thank you for posting query in Micrsoft Q&A Platform.

As mentioned in Amira's response, kindly try direct next_page URL. Below my videos on pagination will help you, how to configure same in pagination. Kindly check them and implement similar.

Copy activity - Pagination rules - When API response have URL for next page

Copy activity - Pagination rules - Variables in QueryParameters in Azure Data Factory

Hope this helps. Please let me know if any further queries.

Please consider hitting Accept Answer button. Accepted answers help community as well.

Ahsan 20 Reputation points

2023-09-26T20:20:08.92+00:00

Hello, I did watch your 2 videos, and it is what helped me get up to 1000 records, however it does not give the solution for more than that. If you could provide specific details on how that would be done, I would appreciate it. Currently I am trying to use cursor based incremental export. I have the current url: api/v2/incremental/tickets/cursor.json?start_time=1690919546, and i see the results in preview data, and when i debug it is successful, but nothing is showing up in the table where the data is supposed to go (my sink).
ShaikMaheer-MSFT 38,631 Reputation points Microsoft Employee Moderator

2023-09-27T09:17:42.0733333+00:00

Hi Ahsan Aziz,

Thank you details. For me, it looks like, as Amira's pointed out, Zendesk related rate limitation may be. Could you please check with admins of Zendesk to deep dive and see if that helps?

Please consider hitting Accept Answer button. Accepted answers help community as well. Thank you.