Pagination of REST API call with "Link" in Header and rel="next"

Kumar, Arun 296

Hello,

I have a hard time doing pagination in ADF dataflow activity. Tried many of the below, nothing worked. My Link in Header is like below -

User's image

Any idea how to work on it?

BhargavaGunnam-MSFT 25,976 Reputation points Microsoft Employee

2023-02-02T21:26:36.6166667+00:00

Hello @Kumar, Arun,

Welcome to the MS Q&A platform.

From your query, it looks like you are getting a response header under the property called "link"

If this is correct, you can use the below setting for your AbsoluteURL.

Please see this Microsoft documentation that explains the supported values for pagination rules.

I hope this helps. Please let us know if you have any further questions.
Kumar, Arun 296 Reputation points

2023-02-04T01:22:22.2+00:00
Hello @BhargavaGunnam-MSFT

I tried you suggested, its throwing below error

My baseURL is

https://api.enverus.com/v3/direct-access

and LINK in header is like

</casings?action=next&next_page=casingid+%3C+1010170680&pagesize=100000>; rel="next"

How can this be called?

Error below -

"message":"Job failed due to reason: Failure to read most recent page request: Illegal character in path at index 0: </casings?action=next&next_page=casingid+%3C+1010170680&pagesize=100000>; rel="next". Details:com.microsoft.dataflow.Issues: DF-Rest_015 - Failure to read most recent page request: Illegal character in path at index 0: </casings?action=next&next_page=casingid+%3C+1010170680&pagesize=100000>; rel="next"\n\tat com.microsoft.dataflow.Utils$.failure(Utils.scala:76)\n\tat org.apache.spark.sql.execution.datasources.rest.RestClient$$anonfun$readResourcesWithDynamicPaging$1.apply(RestClient.scala:61)\n\tat org.apache.spark.sql.execution.datasources.rest.RestClient$$anonfun$readResourcesWithDynamicPaging$1.apply(RestClient.scala:49)\n\tat scala.util.Try$.apply(Try.scala:192)\n\tat org.apache.spark.sql.execution.datasources.rest.RestClient.readResourcesWithDynamicPaging(RestClient.scala:49)\n\tat org.apache.spark.sql.execution.datasources.rest.RestClient.readResources(RestClient.scala:27)\n\tat org.apache.spark.sql.execution.datasources.rest.RestRDD.compute(RestRDD.scala:20)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:347)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:311)\n\tat org.apache.spark.rdd.MapPar","failureType":"UserError","target":"EnverusV3_df","errorCode":"DFExecutorUserError"}
MartinJaffer-MSFT 26,021 Reputation points

2023-02-08T05:02:26.22+00:00

@Kumar, Arun That looks like it might be the RFC 5988 pagination type. See https://learn.microsoft.com/en-us/azure/data-factory/connector-rest?tabs=data-factory#example-7the-rfc-5988-pagination-rule-is-supported-by-default
BhargavaGunnam-MSFT 25,976 Reputation points Microsoft Employee

2023-02-10T02:27:35.28+00:00

Thank you, @MartinJaffer-MSFT

Hello @Kumar, Arun,

I am checking to see if the above response is helpful here. Please let us know if you have any further questions.
Josh Porter 0 Reputation points

2023-02-22T22:50:49.63+00:00

I have the same issue where RFC 5988 is not working in ADF against my API.

1 answer

Kumar, Arun 296 Reputation points

2023-02-14T21:40:02.79+00:00

@MartinJaffer-MSFT - i tried RFC 5988 also, did not work and it threw error. I solved the issue by creating a pyspark notebook to do looping of the pages. Don't think Azure have a mechanism to support these kind of scenarios.
Please sign in to rate this answer.

1 person found this answer helpful.
BhargavaGunnam-MSFT 25,976 Reputation points Microsoft Employee

2023-02-15T18:13:12.5766667+00:00

@Kumar, Arun

Glad to know that you were able to resolve the issue.

And thank you for sharing the resolution, as it helps the community members look for answers to similar questions.

Ryan Ruenroeng 30 Reputation points

2023-07-14T01:57:53.6066667+00:00

It's not very encouraging to know that a community member needed to use custom code to do the pagination because ADF was able to accommodate their need.

I don't totally know how Arun solved his problem and I'm unable to solve my issue with pagination also.

Josh Porter 0 Reputation points

2023-07-14T15:18:37.0833333+00:00

I solved the problem by ditching Synapse/ADF and moving our firm to Databricks after a 3 month long support thread that went nowhere on this topic. Innovation in Synapse is going to die with the Fabric announcement.
Sign in to comment