ADF Copy activity pagination how to pass variable value to Header of next pagination call

Question

I have a pipeline to copy data from a REST source to Azure Blob Storage. Here are the things with this REST source:

• Its authorization requires a token that is valid for 20 minutes
• Because the extraction usually takes more than 20 minutes, I need to refresh the token every 15 minutes, and I need the Copy activity to switch to this new token during its run.

I have a separate Web activity to refresh the token and set its output to a variable named access_token. My question is, how do I specify in the Copy activity so it will continuously pick up the latest value of the access_token in the Header of next pagination calls?

Here is my 1st attempt at setting up: I simply put the value in Additional Headers section. After testing, I can see that it did not switch the token. It only uses the token when the Copy activity is started. Below is my setup in Additional Headers and Pagination Rules.

My 2nd attempt at setting up: I put Headers in Pagination Rules, put a variable as suggested in the official docs, and set a dynamic value to include the variable like below.

This doesn't work, as when I trigger a debug run, ADF informs me Failure happened on 'Source' side. ErrorCode=RestInvalidPaginationRule,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Invalid PaginationRule, RuleKey='Headers.{token}', RuleValue='Bearer _myTokenValueHere_',Source=Microsoft.DataTransfer.ClientLibrary,'

I also tried to change {token} to Authorization. Same error like above. Any way I can specify in the Pagination Rules that I want to pass a Constant, and use dynamic expression with that?

Answer 1

I understood that your activity should call the API to get a new access token and store it in a pipeline variable access_token.

After the Web Activity, add a Set Variable activity to store the token obtained from the Web Activity into the access_token variable and use it to to set the Authorization header

Since you need to refresh the token every 15 minutes, consider breaking down your copy activity into smaller chunks that run for less than 15 minutes each.

Then set up the pagination rules as per the API documentation. This usually involves specifying how to fetch the next set of results. ADF currently doesn't support dynamic content for headers in the pagination rules section. This is a limitation as of my last update in April 2023.

As a workaround, you can break your copy activity into smaller batches that complete within the token's validity period. If the standard activities in ADF don't meet your requirements, consider using an Azure Function or a custom activity to handle the data extraction and pagination logic, including token refresh.

Links to check :

https://learn.microsoft.com/en-us/answers/questions/687737/azure-data-factory-rest-api-pagination-rules

https://stackoverflow.com/questions/69934468/rest-dataset-for-copy-activity-source-give-me-error-invalid-paginationrule