Hi Keerthana, D,
Welcome to Microsoft Q&A Forum, thank you for posting your query here!
Copying bulk data between environments in Azure Blob Storage can be efficiently managed using several methods
1. AzCopy
AzCopy is a command-line utility designed for copying data to and from Azure Blob Storage. It supports copying blobs, directories, and containers between storage accounts. Here’s a basic example of how to use AzCopy:
azcopy copy 'https://<source-storage-account>.blob.core.windows.net/<container>/<blob-path>?<SAS-token>' 'https://<destination-storage-account>.blob.core.windows.net/<container>/<blob-path>?<SAS-token>'
This command uses server-to-server APIs, ensuring data iscopied directly between storage servers
For more information:
https://https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-blobs-copy
2. Azure Data Factory
Azure Data Factory (ADF) is a powerful tool for orchestrating data movement and transformation. You can create pipelines to automate the copying of data from one blob storage to another. Here’s a high-level overview of the steps:
- Create a Data Factory: Set up a new data factory in the Azure portal.
- Create Linked Services: Define linked services for your source and destination storage accounts.
- Create Datasets: Define datasets for the source and destination blobs.
- Create Pipelines: Build pipelines to perform the copy operation. You can use the Copy Data tool in ADF to simplify this process.
For more information:
https://learn.microsoft.com/en-us/azure/data-factory/tutorial-bulk-copy-portal
3.Azure CLI: You can use Azure CLI commands to automate the copying of blobs. The az storage blob copy start command can be used to initiate the copy process. For more information:
These methods should help you efficiently manage bulk data transfers between your DEV and PROD environments.
If you have any other questions or are still running into more issues, let me know in the "comments" and I would be happy to help you
Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.