Handle the "ByteArrayOutputStream" for large image size more than 2GB while download/uploading data from blob storage

JyotiPrakash Mallick 1 Reputation point
2023-10-11T06:22:00.99+00:00

From my application I am trying to upload/download large size images(specifically more than 2GB files) and we have used

ByteArrayOutputStream uploadImgFile = new ByteArrayOutputStream();

Problem with the ByteArrayOutputSteam is it can hold up to 2GB data. But my aplication support more than 2GB data sometimes it reach up to 60GB. In such case this array became stall and I dont get any output neither failure nor success. How to handle such situation ?

Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
2,493 questions
{count} votes

1 answer

Sort by: Most helpful
  1. Siva Villa 280 Reputation points Microsoft Employee
    2023-10-16T04:56:42.8266667+00:00

    @JyotiPrakash Mallick

    You can utilize the Azure Blob Storage REST API's "Put Block" and "Put Block List" operations to upload large files in chunks. Here are the high-level steps you can follow:

    1. Divide the large file into smaller chunks.
    2. Create a new blob in the Azure Blob Storage account using the "Put Blob" operation. This operation creates a new blob or updates an existing blob with the specified content.
    3. Upload each chunk of the file using the "Put Block" operation. This operation uploads a block of data to the specified block blob.
    4. Commit the blocks to the blob using the "Put Block List" operation. This operation commits the list of blocks that have been uploaded to the specified block blob.

    These articles can provide guidance for implementing this solution along with sample Java code:

    https://learn.microsoft.com/en-us/rest/api/storageservices/understanding-block-blobs--append-blobs--and-page-blobs#about-block-blobs

    Put Block (REST API) - Azure Storage | Microsoft Learn

    Upload a blob with Java - Azure Storage | Microsoft Learn

    azure-sdk-for-java/sdk/storage/azure-storage-blob/src/samples/java/com/azure/storage/blob/FileTransferExample.java at main · Azure/azure-sdk-for-java · GitHub

     

    There are few similar SO threads discussion here: https://stackoverflow.com/questions/61481720/upload-video-in-chunks-azure-blob-storage

    https://stackoverflow.com/questions/61857337/how-to-upload-a-large-file-in-chunks-with-parallelism-in-azure-sdk-v12

    Additional information: How to upload and download blobs from Azure Blob Storage with JavaScript: 

    https://learn.microsoft.com/en-gb/samples/azure-samples/azure-sdk-for-js-storage-blob-upload-download/upload-download-blobs-javascript/

    Please let us know if you have any further queries. I’m happy to assist you further.


    Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.

    0 comments No comments