Handle the "ByteArrayOutputStream" for large image size more than 2GB while download/uploading data from blob storage

Question

From my application I am trying to upload/download large size images(specifically more than 2GB files) and we have used

ByteArrayOutputStream uploadImgFile = new ByteArrayOutputStream();

Problem with the ByteArrayOutputSteam is it can hold up to 2GB data. But my aplication support more than 2GB data sometimes it reach up to 60GB. In such case this array became stall and I dont get any output neither failure nor success. How to handle such situation ?

Answer 1

@JyotiPrakash Mallick

You can utilize the Azure Blob Storage REST API's "Put Block" and "Put Block List" operations to upload large files in chunks. Here are the high-level steps you can follow:

1. Divide the large file into smaller chunks.
2. Create a new blob in the Azure Blob Storage account using the "Put Blob" operation. This operation creates a new blob or updates an existing blob with the specified content.
3. Upload each chunk of the file using the "Put Block" operation. This operation uploads a block of data to the specified block blob.
4. Commit the blocks to the blob using the "Put Block List" operation. This operation commits the list of blocks that have been uploaded to the specified block blob.

These articles can provide guidance for implementing this solution along with sample Java code:

https://learn.microsoft.com/en-us/rest/api/storageservices/understanding-block-blobs--append-blobs--and-page-blobs#about-block-blobs

Put Block (REST API) - Azure Storage | Microsoft Learn

Upload a blob with Java - Azure Storage | Microsoft Learn

azure-sdk-for-java/sdk/storage/azure-storage-blob/src/samples/java/com/azure/storage/blob/FileTransferExample.java at main · Azure/azure-sdk-for-java · GitHub

 

There are few similar SO threads discussion here: https://stackoverflow.com/questions/61481720/upload-video-in-chunks-azure-blob-storage

https://stackoverflow.com/questions/61857337/how-to-upload-a-large-file-in-chunks-with-parallelism-in-azure-sdk-v12

Additional information: How to upload and download blobs from Azure Blob Storage with JavaScript: 

https://learn.microsoft.com/en-gb/samples/azure-samples/azure-sdk-for-js-storage-blob-upload-download/upload-download-blobs-javascript/

Please let us know if you have any further queries. I’m happy to assist you further.

---

Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.