How to copy data from rest api using oauth to azure storage as json using adf

Question

I have difficulties creating an ADF pipeline that pulls data from a REST API using oAuth and writes it to azure blob storage in JSON format. I am a python developer and relatively new to adf. I will explain in more detail below.

1. I need to obtain an access token by posting a request to the token endpoint. This step works fine.
2. I need to pull the data from the API using a GET request in a copy activity. Also I need to include pagination and I am kind of wondering how to put the next link as it contains a "." inside its key.

I get the following error, hope you can help me out. Because I am trying to put this workflow on ADF instead of a python script that does this job.

Answer 1

Hi @MartinJaffer-MSFT ,

Thanks, and how do I escape the "." after "@odata" in '@odata.nextLink' ?

Answer 2

Hi @MartinJaffer-MSFT ,

The pagination url looks as follows
'@odata.nextLink': 'base_url/odata/table?$skip=250'

So the workflow is as follows:

1. POST request to token url to obtain bearer token
2. Session.Headers['Authorization'] = Token from step 1
3. GET request to endpoint, append data to list, make call to
   @odata.nextLink': 'base_url/odata/table?$skip=250
   , append to list, and so on.

Would this be possible in data factory? I want to fetch the json from the api and copy it to a blob storage