Implement the Rest API pagination rule with Twitter API

Question

Hi everyone,

I'm trying to get tweets data in a json file with Microsoft Azure Data Factory. I've created a copy data activity which takes the url for the get tweets api with the needed parameters (those parameters are included in the dataset definition with a parameter that represents the keyword I'm looking for) as the source, and a json file dataset as a sink.
This works pretty well, the file is created with the rest api response.
Now I want to handle the pagination rule for twitter as I will get more that the number of tweets contained in one response.
The twitter way to handle this is to have a "meta" element in the API response containing various information and particularly a "next_token" element containing the value to add to a parameter in the next get query that has to be submitted, up until the next_token element is absent from the response.

so : first get example :

tweets/search/recent?query=textToSearch&max_results=100'  

You receive in the response

 "meta": {  
    "next_token": "MyNextToken"  
  }  

So th 2nd get should be :

tweets/search/recent?query=textToSearch&max_results=100&pagination_token=MyNextToken  

I've checked the help for pagination rule but can't quite figured out what to do with that, i've seen examples with the complete url but not with a new parameter which is not included in the first get query. I'm guessing you'll have to create a variable wich will contain the new parameter but I don't know how to add the variable to the url.

I've tried this but it did not work, I've only gotten the first response in my json file and not the other pages :

Thanks in advance for any tips on how to implement that !

Answer 1

Hi @Mikael BURBAN ,

Thank you for posting query in Microsoft Q&A Platform.

As per my understanding, you are using an API with pagination on Query parameters and you are structed with how exactly to implement same. Please correct me if I am wrong.

If our query parameters define pagination then we should consider having place holders in linked service of your API. Could you please try below format linked service and pagination rules and see if that helps.

Click here to know more about pagination rules

If you are using above kind of implementation, make sure to not to have anything in relative URL of REST dataset.

Please let us know how it goes. Thank you.

Answer 2

@ShaikMaheer-MSFT
I've tried the solution you posted before and got a :
Response payload:{"errors":[{"parameters":{"pagination_token":["{MyNextToken}"]},"message":"The pagination_token query parameter value [{MyNextToken}] is not a valid token."}]

No more luck with my solution, I've managed to get the first set of results in a file, get the next_token value contained in the result file and then send another request with this parameter and a pagination rule but I've only manage to obtain the 2nd set of results.

So the only option I see now is a loop of get calls + look-ups for next_token values until it's empty and put the results in different json files. Too bad, this pagination option seemed to be promising.