Access token for api call expires during pipeline execution.

Kakehi Shunya (筧 隼弥) 201 Reputation points
2022-10-14T08:39:47.377+00:00

During execution of the pipeline to copy data from Anaplan to Azure, the access token expires.
Anaplan's access token expires 35 minutes after creation.
Because of the large number of pages in the table to be retrieved and the pipeline taking more than 35 minutes to complete, not all pages can be copied.
Is there any good solution?
With the current pipeline setup, it can only copy about 50 pages in 35 minutes.

Foreach setting
@range(0,activity('retrieve_status_of_request').output.viewReadRequest.availablePages)

250402-image.png
250289-image.png
250392-image.png

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,041 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,974 questions
{count} votes

1 answer

Sort by: Most helpful
  1. MartinJaffer-MSFT 26,106 Reputation points
    2022-10-14T18:04:42.497+00:00

    Hello @Kakehi Shunya (筧 隼弥) ,
    Thanks for the question and using MS Q&A platform.

    As I understand, your problem is that the authentication token expires before the loop containing copy activity finishes. I do have a proposed solution. Image below.
    proposed diagram

    We can make an authentication token fetching (until) loop, which will run until the copy loop finishes. In this we will be storing the authentication token in a variable. A variable has only one instance, so activities inside a loop and activities outside a loop both access the same variable. We will also need a variable to tell our token-fetching-loop to stop. Let's call this "loopbreak". A pipeline doesn't end as long as something keep running, so it is important we only start the token-fetching-loop when we are certain there will be a way to stop the loop. So, let us give the loopbreak variable a default value to true. Add an until-loop, and set it to run as long as loopbreak is false, and stop when loopbreak is true.

    After the copy-loop, place a Set Variable to set loopbreak to true. Connect this using an on-complete dependency, so no matter whether the copy-loop succeeds or fails, we will tell the token-fetching-loop to stop.
    From the activity just before the copy-loop, place a set variable to set loopbreak variable to false. Then connect this set variable to the token-fetching-loop.
    Inside the token-fetching-loop, we place a wait activity for 34 minutes, followed by a web activity to get the new token, and set variable to store this token.

    Inside the copy-loop modify the copy activity to use the token variable as necessary.

    Since the token-fetching-loop gets new token every 34 minutes, the next copy activity will get the new token from the variable without worry. Also, depending upon cadence / rhythm, the token-fetching-loop might still be running when the copy-loop finishes, but it will stop.

    Please do let me if you have any queries.

    Thanks
    Martin


    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
      • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators
    1 person found this answer helpful.

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.