How to copy an entire azure storage account to another without downloading every blob

Question

My company often does copy downs between different environments, such as copying databases from production down to test. We are attempting to do this as well with azure blob storage accounts, but running into major roadblocks. Everything we have found so far leads us to use azcopy as a way to copy all blobs from one account to another, using either sync or copy. Sync works well because we can have it mirror the source, but the problem with this approach is it does not scale. It downloads every file and uploads to the target account which we want to avoid. Our source account has over 50 million files and we need a way to copy all of them quickly. At the rate we are seeing it will take half a day or longer to copy using azcopy. My hope is that there is some tools that does the move between accounts without the roundtrip, or takes the entire account and clones it somehow. Something akin to a backup and restore of a SQL database. I have not found anything like that so far in my search. Does anyone have any suggestions on how this could be done faster at this scale?

Answer 1

@Jason Rodman AzCopy uses server to server APIs when copying data between storage accounts and should not use the local bandwidth of your computer. It is currently the fastest way to migrate data between accounts. To increase performance, you can try decreasing the logging level, separating the copy into multiple operations, and turn off length checking. See Optimize for large numbers of small files for more information.

Can you try making these changes to see if there is an improvement? If you are still facing issues or need further suggestions, please let me know,

-------------------------------

Please don’t forget to "Accept the answer" and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.

Answer 2

There's a CLI command to copy from account to account:

az storage copy

Docs have example:

    Copy an entire account data from blob account to another blob account, and you can also specify
    the storage account and container information of source and destination as above.
        az storage copy -s https://[srcaccount].blob.core.windows.net -d
        https://[destaccount].blob.core.windows.net --recursive