Azure Synapse Rest + Copy Data Activity for Graphql

Question

I have a graphl Endpoint. I want to copy the data from this endpoint to e.g. delta tables.
My issue that i do not understand if it copy data can handle pagination as i need it.
https://learn.microsoft.com/en-us/azure/data-factory/connector-rest?tabs=data-factory#pagination-support

My graphql query looks like below. The issue I have is that the body content is dynamic and needs to change but it depends on the result of the former. My understanding is that pagination is only supported if url/ url paremeters or headers dynmically depend on last page but i do not see an option to set the body content based on last page result with some intitial default.

What works to some extend for me is using basic http acitivity + for/until loop and variables. However, then my issue is how to save the data to any sink.
As described here https://learn.microsoft.com/en-us/answers/questions/129351/azure-data-factory-web-activity-save-output there is a solution to use a second webactivity and post results to blod-rest api but that limits my sinks to blob storage or custom rest api is required like azure functions. Yet, I wonder if i can use copy activity somehow for my specific usecase.

query {

	data(
      first: 100 #page size
      after: $after      # starting with 0 and then always set to = pageInfo.endCursor
	  oderBy: ...some condition
  ) {

    pageInfo {
      endCursor
      hasNextPage # boolean indicating if there is more data

    }
}

  
   
    
  

   
      
      
 

Answer 1

Hi @Thomas K ,

Welcome to Microsoft Q&A forum and thanks for reaching out here.

Yes, your understanding is correct about the pagination functionality. If your requirement is to dynamically update the POST body based on the previous response, then copy activity may not be suitable. As you rightly pointed out, you may have to use 2 web activities inside a loop and write it to blob.

But this sounds like a new use case scenario which isn't supported in current pagination rule setting available in REST Connector. Hence would recommend submitting a feature request in ADF IDEAS forum here: https://feedback.azure.com/d365community/forum/1219ec2d-6c26-ec11-b6e6-000d3a4f032c

Here is an already logged feature request by other users: Pagination Conditions in REST Body . Feel free to up-vote and comment on it with your specific use case to help increase the priority of this feature request.

Hope this helps.

---

Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.