Copy Activity(REST source) How to map a particular json key-value to a variable

Ramamoorthy, Kesavan 61

Hi,

I am using a http connector to pull the data from a rest api source(due to other issues in rest connector) and store the data in ADLS gen2. The source api response is a json file with few fields.

Under 'Mapping' tab in the copy activity dataset, the array 'data' is what we need to store in ADLS and there is no issues in mapping that array to Sink and that has been done already.

The question is, Is there any possibility to assign the values coming from the json output key
"[pagination][cursors][after]" that has been highlighted in the picture to a variable ?

Saurabh Sharma 23,791 Reputation points Microsoft Employee

2021-01-27T01:22:19.373+00:00

@Ramamoorthy, Kesavan Thanks for using Microsoft Q&A !!

No, you cannot assign it to a variable in schema mappings. Can you please provide any specific need for doing the same ?
Ramamoorthy, Kesavan 61 Reputation points

2021-01-27T08:49:01.647+00:00

Hi @Saurabh Sharma ,
Below is the loop that's used to iterate thru' different pages based on pagenumber parameter in api.

But the people from source team have changed the source api design and there is no more pagenumber parameter in the api and instead they have implemented cursor based approach to iterate thru' pages. During the first api call,response gives us cursorId - "[pagination][cursors][after]" (as shown in schema mapping screenshot earlier) which should be given as api parameter for the second api call. I was thinking like if we capture the [pagination][cursors][after] in some variable, we can use it in the dataset parameter in the same way pagenumber parameter is used today, as shown in another snip. Pls let me know possible approach that can be implementd in loop to iterate based on cursor id which is known only after 1st api call
Ramamoorthy, Kesavan 61 Reputation points

2021-01-27T08:52:30.053+00:00

Hi @Saurabh Sharma ,
Missed a screenshot in earlier reply. Pls find it below.
Saurabh Sharma 23,791 Reputation points Microsoft Employee

2021-01-28T04:53:11.35+00:00

@Ramamoorthy, Kesavan Thanks for sharing. I am checking internally into this and get back to you.
Saurabh Sharma 23,791 Reputation points Microsoft Employee

2021-02-01T23:56:42.08+00:00

@Ramamoorthy, Kesavan Can you try if could use lookup activity to get the first row of same file and fetch the "after" once the copy activity finishes writing ?
You may try to use a custom activity to separate the data and metadata, and make both available if that's possible. Please let me know if any of these can help you to proceed further.
Saurabh Sharma 23,791 Reputation points Microsoft Employee

2021-02-09T01:09:27.037+00:00

@Ramamoorthy, Kesavan Please let me know if you have any questions.
Ramamoorthy, Kesavan 61 Reputation points

2021-02-12T09:45:02.057+00:00

Hi @Saurabh Sharma ,

The lookup activity that you suggested worked well. Thanks a lot for that. I have another question mentioned below.

The picture below is the loop in my ADF pipeline wherein we have a copy activity that fetches data from an api using http connector.

Whenever the copy activity fails due to some reason in the api, the loop goes back and starts from the 1st step. For example, when loop starts with the first iteration and when copy fails, it goes back to the first step i.e. "Get Access Token2" and start running from there. Is this a normal behavior? Sorry I am fairly new to Azure.

Is there any way that we stop the loop as soon as the copy activity fails.? Basically it should not go back to the first step of the loop when the copy activity fails.
Ramamoorthy, Kesavan 61 Reputation points

2021-02-18T20:31:55.337+00:00

Hi @Saurabh Sharma

The lookup activity that you suggested worked well. Thanks a lot for that. I have another question mentioned below.

The picture below is the loop in my ADF pipeline wherein we have a copy activity that fetches data from an api using http connector.

67401-image.png

Whenever the copy activity fails due to some reason in the api, the loop goes back and starts from the 1st step. For example, when loop starts with the first iteration and when copy fails, it goes back to the first step i.e. "Get Access Token2" and start running from there. Is this a normal behavior? Sorry I am fairly new to Azure.

Is there any way that we stop the loop as soon as the copy activity fails.? Basically it should not go back to the first step of the loop when the copy activity fails.
Saurabh Sharma 23,791 Reputation points Microsoft Employee

2021-02-19T23:35:05.757+00:00

@Ramamoorthy, Kesavan sorry, I have missed it. I will check and get back to you if it is possible.
Ramamoorthy, Kesavan 61 Reputation points

2021-03-03T08:12:42.39+00:00

Hi @Saurabh Sharma ,

Did you get chance to look into this ?

Share via

Copy Activity(REST source) How to map a particular json key-value to a variable