Access token for api call expires during pipeline execution.

Question

During execution of the pipeline to copy data from Anaplan to Azure, the access token expires.
Anaplan's access token expires 35 minutes after creation.
Because of the large number of pages in the table to be retrieved and the pipeline taking more than 35 minutes to complete, not all pages can be copied.
Is there any good solution?
With the current pipeline setup, it can only copy about 50 pages in 35 minutes.

Foreach setting
@range(0,activity('retrieve_status_of_request').output.viewReadRequest.availablePages)

Answer 1

Hello @Kakehi Shunya (筧 隼弥) ,
Thanks for the question and using MS Q&A platform.

As I understand, your problem is that the authentication token expires before the loop containing copy activity finishes. I do have a proposed solution. Image below.

We can make an authentication token fetching (until) loop, which will run until the copy loop finishes. In this we will be storing the authentication token in a variable. A variable has only one instance, so activities inside a loop and activities outside a loop both access the same variable. We will also need a variable to tell our token-fetching-loop to stop. Let's call this "loopbreak". A pipeline doesn't end as long as something keep running, so it is important we only start the token-fetching-loop when we are certain there will be a way to stop the loop. So, let us give the loopbreak variable a default value to true. Add an until-loop, and set it to run as long as loopbreak is false, and stop when loopbreak is true.

After the copy-loop, place a Set Variable to set loopbreak to true. Connect this using an on-complete dependency, so no matter whether the copy-loop succeeds or fails, we will tell the token-fetching-loop to stop.
From the activity just before the copy-loop, place a set variable to set loopbreak variable to false. Then connect this set variable to the token-fetching-loop.
Inside the token-fetching-loop, we place a wait activity for 34 minutes, followed by a web activity to get the new token, and set variable to store this token.

Inside the copy-loop modify the copy activity to use the token variable as necessary.

Since the token-fetching-loop gets new token every 34 minutes, the next copy activity will get the new token from the variable without worry. Also, depending upon cadence / rhythm, the token-fetching-loop might still be running when the copy-loop finishes, but it will stop.

Please do let me if you have any queries.

Thanks
Martin

---

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
  • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators