How can I apply pagination with Rest API in Azure data factory?

Question

I am using ADF to transfer data from azure app insights to azure data explorer.

I created a rest service and applied this get query app-insights-get, my issue now is that I can't apply pagination on that API response.

The expected number of rows is very large, how can I do that?

Thanks.

Answer 1

Hello @Mohamed Abobakr (IBS) and welcome to Microsoft Q&A

I took a look at the doc you linked. I didn't see any pagination mechanism in the documentation or the example. I had a colleague try a big query, and it returned 1000 rows all in one go.

I don't think this api does pagination. This also means you can use the HTTP connector instead of the REST connector.

What is the reason you wanted to paginate? did you want to break things up into smaller chunks? Or did you run into some limit and it did start paginating?

Answer 2

True that Data Flow doesn't do HTTP connector. @Mohamed Abobakr (IBS)

I'm thinking maybe break this down into smaller bits, if you don't mind. Can we get clever with the query and the timespan?

For example, with the timespan, can we do multiple queries, each getting a different span of time, like the same query, but 1 month ago, another copying doing same query but 2 months ago, then 3 and so on?

Or perhaps do multiple queries, each retrieving a piece of the data, like one that does all even numbered events, and another doing all odd numbered events? If they do have numbers.

Copying the output of these to blob or lake, we can then operate on them, and perhaps unify at the end.

As I see it, we have several tasks:

1. get ALL the data in ANY form
2. massage the data into a form you like
3. send it to final destination.

I'm poking around to see what transformations I can do now.