How to migrate 71 million files from AWS S3 to Azure Blob storage.

Question

I am using AzCopy to copy 71 million Tiny .sqs marker files (35-45 bytes each) from AWS S3 to Azure Blob storage. Here total size of s3 is around 50 GB but the files count is 71 Million. I started with azcopy but it took 2hrs to process 2.5 % of files.

So tell me the best approch.

Answer 1

Hello @Sabjar

With 71 million objects totaling only ~50 GB, the challenge isn't the amount of data. It's the extremely high object count. Each object requires a separate listing, metadata retrieval, and upload operation, so you'll likely be limited by transaction overhead rather than network bandwidth.

Here are some recommendations:

1. Tune AzCopy for high concurrency

• Increase the concurrency value if your VM has sufficient CPU and network resources: azcopy cp "https://<s3-source>" "https://<blob-destination>" --recursive=true
• Set the AZCOPY_CONCURRENCY_VALUE environment variable (for example, 128 or higher) and monitor CPU, memory, and network utilization.

2. Run multiple AzCopy jobs in parallel

Instead of copying the entire bucket with one job, partition the workload by prefixes (folders or key prefixes) and run multiple AzCopy processes concurrently. This often provides much better throughput when dealing with millions of small objects.

3. Expect transaction overhead

Files that are only 35–45 bytes each incur almost the same request overhead as much larger files. In this scenario, the limiting factor is the number of storage operations (List/Get/Put), not the 50 GB data volume.

4. Consider Azure Data Factory for orchestration

If this is a one-time migration, Azure Data Factory can orchestrate and monitor the copy process, although the underlying limitation of copying millions of small files still applies.

5. Evaluate whether all marker files are needed individually

If the application permits, combining the marker files before migration or redesigning the storage format can dramatically reduce migration time and future storage transaction costs.

A few questions that would help identify the best approach:

• Approximately how many top-level prefixes (folders) exist in the S3 bucket?
• What compute environment is running AzCopy (VM size, CPU cores, network bandwidth)?
• Are all 71 million files in a single prefix, or are they already partitioned?
• Is this a one-time migration, or will ongoing synchronization be required?

Please "Accept the Answer" if this information helped you. This will help us and others in the community as well.