Zendesk Rest API Pagination Rules in Data Flow in Azure Data Factory

SB 0

I am trying to apply pagination in Data Flow in Azure Data Factory to import all data from zendesk api, but I keep getting only 100 records. I tried using different pagination rules, but it is not working for me. They have asked me to use offset pagination. I see Pagination rules like EndCondition, AbsoluteURL, Query, Header, MaxRequestNumber and None. Anyone experience using ADF with zendesk that can give me some pointers? Your help is appreciated. Thank you.

KranthiPakala-MSFT 46,422 Reputation points Microsoft Employee

2023-02-08T00:34:26.35+00:00

Hi SB,

Welcome to Microsoft Q&A forum and thanks for posting your query.

To better assist on your query, could you please confirm what specific Zendesk REST API Endpoint you are trying to call from ADF Data flows and sample of your response body where is shows the next page or pagination offset details (please mask any sensitive information)?
In addition, just curious to know if there any specific reason to go with REST API, when ADF mapping dataflow support Zendesk as an Inline dataset?
Ref doc: Transform data in Zendesk (Preview) using Azure Data Factory or Synapse Analytics

Thanks
SB 0 Reputation points

2023-02-08T01:04:03.6433333+00:00

Thanks KranthiPakala - For your response... I want to two pull all Organizations and ticket form data from the two apis that were updated/created say in the last 24 hours and also way to paginate through all the pages to be able to load all the records using Data Flows in ADF.

These are the two End Points with masked sensitive data....

https://MYDOMAIN.zendesk.com/api/v2/organizations?

https://MyDomain.zendesk.com/api/v2/search?query=ticket_form_id:8836077753499%20custom_field_8839999999539:i_ttt_at_an_yyy_kkkk_trn_level

Thanks a lot...
SB 0 Reputation points

2023-02-09T03:51:16.77+00:00

Thanks KranthiPakala - I was able to create source and destination mapping using data flows as well as flattening the complex data attributes. What I wanted to know is how can we use the above APIs to get only data updated/created in the last say 12 hours and also what pagination rules we can apply in Data Flows on the above APIs to be able to load all the data. Right now no matter what pagination rules I apply, I am only able to load 100 rows (1 page). Thanks a lot for your assistance.
SB 0 Reputation points

2023-02-09T03:51:40.1433333+00:00

I was planning to use Data Flows because I have multiple fields that have complex data structure (arrays) so need to flatted all of those.

1 answer

KranthiPakala-MSFT 46,422 Reputation points Microsoft Employee

2023-02-09T03:50:41.9566667+00:00

Hi SB,

Thanks for the details. You may try configuring your dataflow settings as below for paginating the API response.

Please note that I don't have Zendesk instance to test it but providing guidance as per this API pagination documentation: Using offset pagination

For your requirement you may use REST API using the REST connector as shown above or you can also use Zendesk connector as in inline dataset and you can transform data as needed and also you can filter the data using filter transformation.

Hope this info helps.

Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.
Please sign in to rate this answer.
KranthiPakala-MSFT 46,422 Reputation points Microsoft Employee

2023-02-14T08:15:32.2233333+00:00

Hi SB,

Just checking in to see if the above suggestion was helpful. If it answers your query, please do click “Accept Answer” and/or Up-Vote, as it might be beneficial to other community members reading this thread. And, if you have any further query do let us know.

Thanks
Sign in to comment