How to use Data Factory's pagination EndCondition on a copy data rest api source

Pedro Rodrigues 6 Reputation points
2021-11-02T09:09:59.443+00:00

Hey,

my requests have a next_page and a end of stream in the json output. The next page rule works as an AbsoluteURL pagination rule with value $.next_page .

However, when I try an EndCondition with value $.end_of_stream this does not work. I can't seem to figure out a way to get it to work.

This image shows what each request page looks like with the next_page and end_of_stream feature.

145782-apiendpoint.png

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,640 questions

4 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. svijay-MSFT 5,206 Reputation points Microsoft Employee
    2021-11-17T11:00:27.767+00:00

    Hello @Pedro Rodrigues ,

    Thanks for the question and using MS Q&A platform.

    The document has been updated here https://learn.microsoft.com/en-us/azure/data-factory/connector-rest?tabs=data-factory#pagination-support

    for your scenario,can you please try the below

    150060-image.png

    1 person found this answer helpful.
  2. André Lucas da Silva Pereira 1 Reputation point
    2021-11-08T14:52:42.7+00:00

    Hello, I'm also looking into this new feature. A couple of months ago, the EndCondition was not implemented, and Data Factory's docs are not updated with this info.

  3. Sumit Singh 1 Reputation point
    2021-11-17T04:09:23.44+00:00

    Hello, Any update on this? How to implement end condition

    0 comments
  4. Jenni K 1 Reputation point
    2021-12-02T13:28:15.367+00:00

    @svijay-MSFT I'm not the original requestor but for some reason I could not make a similar case work with your instruction above. Can you be more specific and show for example a piece of the working .json script your input creates?

    It would also be nice to have a similar/clear example on using QueryParameters here as well.