Pagination doesn't work as expected on ADF Copy activity

Question

Hi,

I'm looking for some help on configuring the pagination settings to copy data from a REST endpoint on ADF.

The API response provides the links like below. Even if it is supported by default, we also tried "paginationRules": {"supportRFC5988": "true"} in pagination rules. Activity stops after 1st call.

Normally AbsoluteUrl: $links[2].href can be also used. But in the second call rel:next becomes the 3rd item in the list because of rel:prev link.

"metadata": {
		"number": 0,
		"size": 100,
		"totalResources": 77182,
		"totalPages": 772,
		"cursor": "somevalue123"
	},
"links": [{
			"href": "BaseURL/myTable?includeSame=True&page=0&size=100&sort=id,asc",
			"rel": "first"
		},{
			"href": "BaseURL/myTable?includeSame=True&page=0&size=100&sort=id,asc",
			"rel": "self"
		},{
			"href": "BaseURL/myTable?includeSame=True&page=1&size=100&sort=id,asc&cursor=somevalue123",
			"rel": "next"
		},{
			"href": "BaseURL/myTable?includeSame=True&page=772&size=100&sort=id,asc",
			"rel": "last"
		}
	]

We also tried AbsoluteUrl: @if(equals('$.metadata.number', 0), '$.links[2].href', '$.links[3].href')

However, the condition "if(equals('$.metadata.number',0)" is not evaluated as expected. 

Already from the first response (page 0) the third item is called (being the last page), moreover that is throwing an 400 Bad Request error at the end.

Could you please help to resolve the issue?

Answer 1

Hi Mustafa Murat Sever ,

Welcome to Microsoft Q&A platform and thanks for posting your question here.

I understand that you're trying to configure pagination settings in Azure Data Factory (ADF) to copy data from a REST endpoint that provides pagination links in the response. However, it's not working as expected.

Could you please explain this statement : "But in the second call rel:next becomes the 3rd item in the list because of rel:prev link"

You mentioned that the condition if(equals('$.metadata.number', 0) is not evaluated as expected. This might be because the evaluation is done before the pagination rules are applied.

To handle this, you can use a set variable activity before your Copy Data activity . In the Set Variable activity, you can evaluate the condition and set a variable based on the condition. Then, use the variable in your Copy Data activity.

For example: Create a variable (e.g., NextLink) in your pipeline. In the Set Variable activity, set the value of NextLink based on the condition. Then, in your Copy Data activity, use @variables('NextLink').NextLink as the "Relative URL" or "Request URI" for the REST endpoint.

For better understanding on your scenario, kindly share your pipeline json if possible or attach the screenshot of the activity configuration so far. Thankyou