Hello Wansart, Christian,
Thank you for posting your query here!
For uploading large files to Azure Blob Storage, Block Blobs are generally the best choice. Block Blobs are designed to handle large files efficiently by breaking them into blocks. You can upload each block in parallel to increase upload speed, and each block can be retried separately upon failure, saving bandwidth.
The azure-storage-cpplite library you’re using seems to have some limitations with large files. However, you can work around this by reading your large file into smaller chunks and uploading each chunk with the Put Block API. After all blocks are uploaded, you can call the Put Block List API to commit the blob.
Here’s a high-level overview of the steps:
· You can decide the size of the chunks based on your requirements. A common choice is to use chunks of a few megabytes each.
· Each chunk is uploaded as a separate block. Each block is identified by a block ID, which must be unique within the blob.
· Call the Put Block List API to commit the blob: After all blocks have been uploaded, you provide a list of block IDs to the Put Block List API in the order that they should appear in the blob.
As for the upload_block_blob_from_buffer function, it’s not suitable for large files because it requires the entire file to be loaded into memory. This can lead to out-of-memory errors for large files.
Regarding parallelism, it’s about uploading multiple blocks at the same time. This can significantly increase upload speed, especially for large files. However, it also requires more memory and CPU resources. If you want to upload a single large file, you can still use parallelism by uploading multiple chunks of the file at the same time.
https://inside.covve.com/uploading-block-blobs-larger-than-256-mb-in-azure/
Note: You can use Azure Storage Explorer, Azcopy, Power Shell, CLI or any programming language to upload large files.
Upload large amounts of random data in parallel to Azure storage
How to upload big files to Azure Blob Storage (.NET Core)
If you want to upload larger files to file share or blob storage, there is an Azure Storage Data Movement Library. It provides high-performance for uploading, downloading larger files. Please consider using this library for larger files.
Choose an Azure solution for data transfer : This article provides an overview of some of the common Azure data transfer solutions. The article also links out to recommended options depending on the network bandwidth in your environment and the size of the data you intend to transfer.
I hope this helps! Please let me know if the issue persists or if you have any other questions.
Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.