Large Scale Data Uplaods To Blob Storage - Recommended Solutions

Question

I am currently working on a requirement that involves enabling external clients to upload large data files, typically exceeding 20GB and potentially reaching up to 50GB, to Azure Blob Storage. To ensure security and efficiency, the current solution involves Azure SFTP uploads directly to storage.

However, considering the inherent constraints of internet bandwidth and latency, I am reaching out to gather insights and explore additional solutions that could enhance the secure client upload process to Azure Blob Storage.

If anyone has suggestions, experiences, or alternative solutions that warrant investigation, your input would be greatly appreciated.

Answer 1

Hello Marc Hedgley,

Thank you for posting your query here!

You may consider using Azcopy tool to upload files from on-premises or cloud (Use this command-line tool to easily copy data to and blobs from Azure Blobs, Blob Files, and Table storage Storage with optimal performance. ) AzCopy supports concurrency and parallelism, and the ability to resume copy operations when interrupted. It provides high-performance for uploading, downloading larger files.

If you want to upload larger files to file share or blob storage, there is an Azure Storage Data Movement Library.

Here is a sample code:

using Microsoft.Azure.Storage;
using Microsoft.Azure.Storage.Blob;
using Microsoft.Azure.Storage.DataMovement;

class program
{
    public static void Main(string[] args)
    {
        string storageConnectionString = "<Connection string>";
        CloudStorageAccount account = CloudStorageAccount.Parse(storageConnectionString);
        CloudBlobClient blobClient = account.CreateCloudBlobClient();
        CloudBlobContainer blobContainer = blobClient.GetContainerReference("test");
        blobContainer.CreateIfNotExists();
        string sourceBlob = @"C:\Users\download\sample.docx";
        CloudBlockBlob destPath = blobContainer.GetBlockBlobReference("sample.docx");
        TransferManager.Configurations.ParallelOperations = 64;
        // Setup the transfer context and track the download progress
        SingleTransferContext context = new SingleTransferContext
        {
            ProgressHandler = new Progress<TransferStatus>(progress =>
            {
                Console.WriteLine("Bytes Upload: {0}", progress.BytesTransferred);
            })
        };
        // upload the blob
        var task = TransferManager.UploadAsync(
        sourceBlob, destPath, null, context, CancellationToken.None);
        task.Wait();
    }
}

Additional references:

Choose an Azure solution for data transfer: This article provides an overview of some of the common Azure data transfer solutions. The article also links out to recommended options depending on the network bandwidth in your environment and the size of the data you intend to transfer.

Upload large amounts of random data in parallel to Azure storage

https://stackoverflow.com/questions/74740392/upload-large-size-file-more-than-2-gb-what-will-be-the-best-approach

Kindly let us know if you have any further queries. I’m happy to assist you further.

---

Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.