API Pagination - "page" via expression

Question

API Pagination - "page" via expression

Ryan Abbey 1,186

We have an API that, rather than returning a "next page" link, gives us the current offset and current row count, like below

"count": 20,
"offset": 0,
"total": 702,

So to put in my next request, I need count + offset (and stop when exceeding total), is there any way to pass that in to the Copy Activity "pagination" options of an API call?

My alternative is going via a Web activity and looping... however, the Web Activity doesn't seem to support API key authentication (except when hard coded), have I missed something there?

svijay-MSFT 5,256 Reputation points Microsoft Employee Moderator

2021-11-03T10:45:30.207+00:00

Hello @Ryan Abbey ,

Thanks for the question and using MS Q&A platform.

Unfortunately, the pagination feature doesn't support the manipulation of values that will meet the requirement.

Coming back to your workaround you had mentioned - Using Web Activity. That is the Workaround I could also think that could meet your requirement. Can you please help me understand better why this may not be meeting your requirement ?

From my understanding you can pass dynamic content in the header section of the Web Activity.
Ryan Abbey 1,186 Reputation points

2021-11-03T20:08:52.947+00:00

Thanks for the response... I'm specifically looking to be able to use key vault in order to keep the key secure

1 answer

Your answer

svijay-MSFT 5,256 Reputation points Microsoft Employee Moderator

2021-11-03T10:45:30.207+00:00

Hello @Ryan Abbey ,

Thanks for the question and using MS Q&A platform.

Unfortunately, the pagination feature doesn't support the manipulation of values that will meet the requirement.

Coming back to your workaround you had mentioned - Using Web Activity. That is the Workaround I could also think that could meet your requirement. Can you please help me understand better why this may not be meeting your requirement ?

From my understanding you can pass dynamic content in the header section of the Web Activity.
Ryan Abbey 1,186 Reputation points

2021-11-03T20:08:52.947+00:00

Thanks for the response... I'm specifically looking to be able to use key vault in order to keep the key secure

Answer 1

svijay-MSFT 5,256 Microsoft Employee Moderator

Hello @Ryan Abbey ,

Thanks for getting back to me.

So, I could - have a Web Activity to get the keys from the Key Vault and Set in the Variable.

WEB Activity URL : https://<your-keyvalut-name>.vault.azure.net/secrets/<your-secret-name>

You could access the output of the web Activity using : @activity('Web1').output.value & Store in a variable.

You can reference this variable as the API KEY for the subsequent Web Activity (mentioned above workaround)

svijay-MSFT 5,256 Reputation points Microsoft Employee Moderator

2021-11-08T10:21:28.29+00:00

Hello @Ryan Abbey ,

Following up to see if the above suggestion was helpful. And, if you have any further query do let us know.
Ryan Abbey 1,186 Reputation points

2021-11-08T20:50:50.81+00:00

This is what we did in the end
svijay-MSFT 5,256 Reputation points Microsoft Employee Moderator

2021-11-15T13:38:40.777+00:00

Hello @Ryan Abbey ,

Please consider Accepting Answer and Up-Vote if the information has helped. And, if you have any further query do let us know.

Share via

API Pagination - "page" via expression

1 answer

Your answer