Pagination of REST API call with "Link" in Header and rel="next"

Question

Pagination of REST API call with "Link" in Header and rel="next"

Kumar, Arun 341

Hello,

I have a hard time doing pagination in ADF dataflow activity. Tried many of the below, nothing worked. My Link in Header is like below -

User's image

Any idea how to work on it?

Bhargava-MSFT 31,361 Reputation points Microsoft Employee Moderator

2023-02-02T21:26:36.6166667+00:00

Hello @Kumar, Arun,

Welcome to the MS Q&A platform.

From your query, it looks like you are getting a response header under the property called "link"

If this is correct, you can use the below setting for your AbsoluteURL.

Please see this Microsoft documentation that explains the supported values for pagination rules.

I hope this helps. Please let us know if you have any further questions.
Kumar, Arun 341 Reputation points

2023-02-04T01:22:22.2+00:00
Hello @Bhargava-MSFT

I tried you suggested, its throwing below error

My baseURL is

https://api.enverus.com/v3/direct-access

and LINK in header is like

</casings?action=next&next_page=casingid+%3C+1010170680&pagesize=100000>; rel="next"

How can this be called?

Error below -

"message":"Job failed due to reason: Failure to read most recent page request: Illegal character in path at index 0: </casings?action=next&next_page=casingid+%3C+1010170680&pagesize=100000>; rel="next". Details:com.microsoft.dataflow.Issues: DF-Rest_015 - Failure to read most recent page request: Illegal character in path at index 0: </casings?action=next&next_page=casingid+%3C+1010170680&pagesize=100000>; rel="next"\n\tat com.microsoft.dataflow.Utils$.failure(Utils.scala:76)\n\tat org.apache.spark.sql.execution.datasources.rest.RestClient$$anonfun$readResourcesWithDynamicPaging$1.apply(RestClient.scala:61)\n\tat org.apache.spark.sql.execution.datasources.rest.RestClient$$anonfun$readResourcesWithDynamicPaging$1.apply(RestClient.scala:49)\n\tat scala.util.Try$.apply(Try.scala:192)\n\tat org.apache.spark.sql.execution.datasources.rest.RestClient.readResourcesWithDynamicPaging(RestClient.scala:49)\n\tat org.apache.spark.sql.execution.datasources.rest.RestClient.readResources(RestClient.scala:27)\n\tat org.apache.spark.sql.execution.datasources.rest.RestRDD.compute(RestRDD.scala:20)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:347)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:311)\n\tat org.apache.spark.rdd.MapPar","failureType":"UserError","target":"EnverusV3_df","errorCode":"DFExecutorUserError"}
MartinJaffer-MSFT 26,161 Reputation points

2023-02-08T05:02:26.22+00:00

@Kumar, Arun That looks like it might be the RFC 5988 pagination type. See https://learn.microsoft.com/en-us/azure/data-factory/connector-rest?tabs=data-factory#example-7the-rfc-5988-pagination-rule-is-supported-by-default
Bhargava-MSFT 31,361 Reputation points Microsoft Employee Moderator

2023-02-10T02:27:35.28+00:00

Thank you, @MartinJaffer-MSFT

Hello @Kumar, Arun,

I am checking to see if the above response is helpful here. Please let us know if you have any further questions.
Josh Porter 0 Reputation points

2023-02-22T22:50:49.63+00:00

I have the same issue where RFC 5988 is not working in ADF against my API.
Simon Cuche 0 Reputation points

2025-09-08T19:45:15.5066667+00:00

Same issue with fabrics pipeline copy activity. Any solution available? I prefere to use the copy activity without pyspark.

1 answer

Your answer

Bhargava-MSFT 31,361 Reputation points Microsoft Employee Moderator

2023-02-02T21:26:36.6166667+00:00

Hello @Kumar, Arun,

Welcome to the MS Q&A platform.

From your query, it looks like you are getting a response header under the property called "link"

If this is correct, you can use the below setting for your AbsoluteURL.

Please see this Microsoft documentation that explains the supported values for pagination rules.

I hope this helps. Please let us know if you have any further questions.
Kumar, Arun 341 Reputation points

2023-02-04T01:22:22.2+00:00

Hello @Bhargava-MSFT

I tried you suggested, its throwing below error

My baseURL is

https://api.enverus.com/v3/direct-access

and LINK in header is like

</casings?action=next&next_page=casingid+%3C+1010170680&pagesize=100000>; rel="next"

How can this be called?

Error below -

"message":"Job failed due to reason: Failure to read most recent page request: Illegal character in path at index 0: </casings?action=next&next_page=casingid+%3C+1010170680&pagesize=100000>; rel="next". Details:com.microsoft.dataflow.Issues: DF-Rest_015 - Failure to read most recent page request: Illegal character in path at index 0: </casings?action=next&next_page=casingid+%3C+1010170680&pagesize=100000>; rel="next"\n\tat com.microsoft.dataflow.Utils$.failure(Utils.scala:76)\n\tat org.apache.spark.sql.execution.datasources.rest.RestClient$$anonfun$readResourcesWithDynamicPaging$1.apply(RestClient.scala:61)\n\tat org.apache.spark.sql.execution.datasources.rest.RestClient$$anonfun$readResourcesWithDynamicPaging$1.apply(RestClient.scala:49)\n\tat scala.util.Try$.apply(Try.scala:192)\n\tat org.apache.spark.sql.execution.datasources.rest.RestClient.readResourcesWithDynamicPaging(RestClient.scala:49)\n\tat org.apache.spark.sql.execution.datasources.rest.RestClient.readResources(RestClient.scala:27)\n\tat org.apache.spark.sql.execution.datasources.rest.RestRDD.compute(RestRDD.scala:20)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:347)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:311)\n\tat org.apache.spark.rdd.MapPar","failureType":"UserError","target":"EnverusV3_df","errorCode":"DFExecutorUserError"}
MartinJaffer-MSFT 26,161 Reputation points

2023-02-08T05:02:26.22+00:00

@Kumar, Arun That looks like it might be the RFC 5988 pagination type. See https://learn.microsoft.com/en-us/azure/data-factory/connector-rest?tabs=data-factory#example-7the-rfc-5988-pagination-rule-is-supported-by-default
Bhargava-MSFT 31,361 Reputation points Microsoft Employee Moderator

2023-02-10T02:27:35.28+00:00

Thank you, @MartinJaffer-MSFT

Hello @Kumar, Arun,

I am checking to see if the above response is helpful here. Please let us know if you have any further questions.
Josh Porter 0 Reputation points

2023-02-22T22:50:49.63+00:00

I have the same issue where RFC 5988 is not working in ADF against my API.
Simon Cuche 0 Reputation points

2025-09-08T19:45:15.5066667+00:00

Same issue with fabrics pipeline copy activity. Any solution available? I prefere to use the copy activity without pyspark.

Answer 1

Kumar, Arun 341

@MartinJaffer-MSFT - i tried RFC 5988 also, did not work and it threw error. I solved the issue by creating a pyspark notebook to do looping of the pages. Don't think Azure have a mechanism to support these kind of scenarios.

Bhargava-MSFT 31,361 Reputation points Microsoft Employee Moderator

2023-02-15T18:13:12.5766667+00:00

@Kumar, Arun

Glad to know that you were able to resolve the issue.

And thank you for sharing the resolution, as it helps the community members look for answers to similar questions.
Ryan Ruenroeng 30 Reputation points

2023-07-14T01:57:53.6066667+00:00

It's not very encouraging to know that a community member needed to use custom code to do the pagination because ADF was able to accommodate their need.

I don't totally know how Arun solved his problem and I'm unable to solve my issue with pagination also.
Josh Porter 0 Reputation points

2023-07-14T15:18:37.0833333+00:00

I solved the problem by ditching Synapse/ADF and moving our firm to Databricks after a 3 month long support thread that went nowhere on this topic. Innovation in Synapse is going to die with the Fabric announcement.

Share via

Pagination of REST API call with "Link" in Header and rel="next"

1 answer

Your answer