Azure Data Factories - Renewing Access Tokens every 1 hour.

Question

A long-running process is executing in the pipeline.

For sake of our example, the for each step in this pipeline takes 6 hours.
It is running a for each step in a batch count of 20.

The pipeline looks something like this.

1) Get Access Token
2) Set the access toke variable.
3) Enter for each step. In the foreach step, there is a copy activity process. The source for this copy activity is a webapi call that requires this access token. The sink part of the copy activity is writing to a text file. The next step in the foreach is an update a database table for each file that is written.
4) Exit the pipeline.

One option we have explored is to move the get & set access token steps inside the foreach loop, but that would mean unnecessary calls to just acquire the token.

So I though for checking the time inside the foreach step and if it has passed 50 minutes since it entered it, get a new access token. This is causing that step to be fired many times. Again unnecessary calls to just get a token.

What is the optimum way to do this?

It is using an access token to access a webapi. The access token expires every 1 hour.

How can I set up this pipeline so that every 50 minutes it will check just get a new access_token?

Answer 1

Hello @Abhijit Shrikhande ,
Thanks for the question and using MS Q&A platform.

As I understand you want help making pipeline logic to renew tokens for your copy in an efficient way.

The base requirement you gave is to renew token every 50 minutes. We can do this separate from the ForEach copy loop, because everything points to the same variable you store token in.

Below I have a proposal diagram.

We can use an Until loop to wait 50 minutes, then fetch and store token (exact same way you already have). It will repeat until something tells it to stop. We need something for this. Create a boolean type variable, with default value of true.

This will be used to tell the until loop whether to repeat or stop. See diagram for implemtation in untl activity.

We need to change the value to false after the ForEach loop ends, no matter success or failure. Use a Completion dependency to link the ForEach to a Set Variable. The Set Variable sets to false the boolean variable.

If you are copying from a REST source, make sure the copy does not take more than 50 minutes. The copy only checks the token variable when it starts. I don't think the pagination checks for new values between pages.

Please do let me if you have any queries.

Thanks
Martin

---

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
  • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

Answer 2

Just had some issues following this solution:

1. Boolean values are specified in lower case (true not True)
2. The until loop executes once, and then checks the expression defined in the until activity. It will keep running the child activities until that expression evaluates as true, and then stops. So that's the opposite logic sense as to shown in the images above. The continueloop variable should really be named stoploop with a default value of false, and the last activity will be to set the stoploop variable to true

Took me a while to figure out why my until loop was only running once, so thought I would share in case anyone else comes across this scenario.

Answer 3

I have question almost like that, but in my case the copy data is not a "For each" loop.

the "Get Token" generates a token that expires in 1 hour, but the "Copy data" activity has a pagination rule, and has many pages, and that takes more than 1 hour, so after 1 hour the token expires and the copy data fails.

is that possible generate a new token for each pagination? or generate a token with more than 1 hour of duration? (PS: this token is to get data from azure, it generate a bearer access token)

Thanks a lot.

Answer 4

Hi,

What if our activity like copy activity got completed before 50 minutes and now we don't need to do anything else and just want to end the pipeline. Do we have any way to tell the wait that you dont need to wait anymore?

Thanks

Arsalan Mehmood