Pagination way to handel bulk copy in Logic app

Question

Hi ,

I am using Logic app to copy my blobs from one storage to other. the source storage has blobs which is large approx 1000000 blobs. Now I have the job to run this logic app to copy all the blobs. I would like to know how to configure the pagination in List blobs activity in Logic apps to achieve my requirement.

Thanks,

Answer 1

While logic apps are great to build this workflow, if this is a one-time copy, you could simply use azcopy to move files between storage accounts with a single command.

And if after the first-time bulk copy, you would want to have a workflow to copy newer files into the other storage account as they are created, you could have an Event Grid Logic App which would get triggered once a new blob is created.

That being said, while the pagination feature does help get all the blobs at once, it would be better to fetch the maximum blobs (5000 in this case) and trigger another logic app with the blob metadata returned. The second logic could do the processing, while your first logic app has a loop to fetch the next 5000 blobs to send for processing.

This way, your logic app doesn't wait for all the blobs metadata to be returned in a single call and instead copies files while fetching metadata for the rest of the blobs.

Also, the second logic app could be configured to trigger multiple runs from the array of blob metadata that the action returns.