Azure Data Factory - Data Flow Pagination

Question

Hi, i´m using ADF to process rest api from microsoft defender api/machines to get list of all machines.

To handle the limit i´m using pagination in copy activity

I want to use same in data flow:

But here it´s not working and it only fetches till limit of 10k rows.

Thx in advance for any advice

André

Answer 1

Hello @Müller, André

It worked in the Pipeline Copy Activity and not working when using via Data Flow, right?

What is the request URL looks like? Something like below..?

HTTP GET https://api.securitycenter.microsoft.com/api/machines?$filter=healthStatus+ne+'Active'&$top=100

Did you see the Note mentioned in the documentation, may be we can check the syntax!
The pagination rules in mapping data flows is different from it in copy activity in the following aspects:

1. Range is not supported in mapping data flows.
2. [''] is not supported in mapping data flows. Instead, use {} to escape special character. For example, body.{@odata.nextLink}, whose JSON node @odata.nextLink contains special character . .

I tried the DataFlow with Pagination Rules: AbsoluteURL worked for me.

My REST API response has a property called @odata.context

__with Special characters, so i have followed as per the documentation and i was able to read the value and write to the output!
__
[''] is not supported in mapping data flows. Instead, use {} to escape special character.

The mapping is Good!