Azure Blob Storage stageBlock: Return contentMd5 as a header

Jacob Nguyen 20

So I noticed that that the only way to get checksum validation on large files is to supply contentMd5 before upload on a particular block.

I don't have the block checksum before I upload the block. I'm trying to transfer the data from cloud to cloud. I'd rather just calculate checksums while I upload the file to azure, and I don't want to pull it into storage/memory as it won't be manageable since these are large files.

Is there a way to have a feature request where we can just ask the server to calculate checksum (if I send a header indicating for this) and the server returns the block checksum back so that I can use it to validate my upload?

Vinod Kumar Reddy Chilupuri 2,225 Reputation points Microsoft Vendor

2025-01-06T02:06:30.79+00:00

Hi @Jacob Nguyen
Just checking to see that the above suggestion helped you to solve your query. If this answers your query, do click Accept Answer and Yes for was this answer helpful. which might be beneficial to other community members reading this thread. If you have any further query do let us know.
Vinod Kumar Reddy Chilupuri 2,225 Reputation points Microsoft Vendor

2025-01-07T01:15:29.5+00:00

Hi @Jacob Nguyen
Just checking to see that the above suggestion helped you to solve your query. If this answers your query, do click Accept Answer or Upvote for was this answer helpful. Which might be beneficial to other community members reading this thread. If you have any further query do let us know.
Jacob Nguyen 20 Reputation points

2025-01-07T22:51:55.3266667+00:00

Manually setting the MD5 on the whole file from our calculation isn't very good of a validation.

There can be malformed bytes that can occur during that upload that we can't see but is stored on the Azure side.

Like it's fine that we can't calculate full checksum of the entire file because they're written in blocks. What's not fine is that I can't get the checksum of the blocks.

My file is on a cloud storage device. In order to calculate checksum I would need to download the file once just to calculate checksum. Then I would have to download the file again in order to upload to Azure Blob storage. My downloading application has zero way of storing the file. Storing that file would be a security risk. Letting the data go into the void after downloading it is also a waste of Egress.

If Azure Blob storage already calculates checksum when you supply an MD5 for the block, I'm just asking for the checksum when we don't have it on hand. It's seems like little work to add an extra header to accept that. And it lets us easily validate the checksum calculated on the stream.

But if it's not possible, there's nothing I can do. We just won't validate integrity of files uploaded to Azure blob.
Jacob Nguyen 20 Reputation points

2025-01-07T23:01:57.2933333+00:00

Ah, second thing to note is that I'm using the Azure Blob SDK for java. The context is that we're uploading from Cloud -> Application that pulls and sends data -> Azure.

If there's not a way to feature request this, then it's fine.
Jacob Nguyen 20 Reputation points

2025-01-07T23:09:28.0966667+00:00

I guess another workaround, is there a way to ask Azure to calculate the checksum after we've uploaded it?

Is there an API from the java SDK that I can request for that instead? That would work for me and is a lot better than nothing.
Vinod Kumar Reddy Chilupuri 2,225 Reputation points Microsoft Vendor

2025-01-08T06:12:58.1166667+00:00

Hi @Jacob Nguyen
Azure Blob Storage does not provide a direct way to retrieve the checksum (MD5) of blocks after they have been uploaded. When you upload a block blob and supply an MD5 hash, Azure calculates the checksum during the upload process.

There is no API in the Azure Storage Java SDK that allows you to request the checksum of a block after it has been uploaded. As I mentioned earlier, Azure Blob Storage calculates the checksum of a block when you supply an MD5 for the block. However, if you don't have the checksum on hand, there is no way to retrieve it from Azure Blob Storage.

If you are looking for a way to validate the integrity of the data after the upload, then you can consider implementing a mechanism where you calculate the MD5 checksum of the data before uploading it and store it securely in your application. This way, you can compare the stored checksum with the one you calculated before the upload to validate the integrity of the data.

Hope the above answer helps! Please let us know do you have any further queries.
Vinod Kumar Reddy Chilupuri 2,225 Reputation points Microsoft Vendor

2025-01-09T05:10:39.8533333+00:00

Hi @Jacob Nguyen
Following up to see that the above suggestion helped you to solve your query. If this answers your query, do click Accept Answer and Yes for was this answer helpful. which might be beneficial to other community members reading this thread. If you have any further query do let us know.
Vinod Kumar Reddy Chilupuri 2,225 Reputation points Microsoft Vendor

2025-01-10T03:48:29.88+00:00

Hi @Jacob Nguyen
Just checking to see that the above suggestion helped you to solve your query. If this answers your query, do click Accept Answer and Yes for was this answer helpful. which might be beneficial to other community members reading this thread.
Jacob Nguyen 20 Reputation points

2025-01-10T23:01:14.7766667+00:00

There is no API in the Azure Storage Java SDK that allows you to request the checksum of a block after it has been uploaded. As I mentioned earlier, Azure Blob Storage calculates the checksum of a block when you supply an MD5 for the block. However, if you don't have the checksum on hand, there is no way to retrieve it from Azure Blob Storage.

Thanks. There's just no way, and there's no way to feature request for it.

I'll just have to download the file twice, or store as much data as possible in memory.
Vinod Kumar Reddy Chilupuri 2,225 Reputation points Microsoft Vendor

2025-01-13T02:52:28.8566667+00:00

Hi @Jacob Nguyen
If the below answer helps you, please Accept the answer and Upvote. If you have any further queries do let us know.

Accepted answer

Vinod Kumar Reddy Chilupuri 2,225 Reputation points Microsoft Vendor

2025-01-04T01:01:11.11+00:00
Hi @Jacob Nguyen

Welcome to Microsoft Q&A, thanks for posting your query.

Azure Blob Storage does not natively support a feature where the server calculates and returns the MD5 checksum of a block after it has been uploaded. The typical workflow requires the client to compute the MD5 checksum before uploading the block and then send it as part of the request.

The best practice is that you always calculate hash of a downloaded blob and keep it as baseline

Blobs uploaded by PutBlob will have Content-MD5 calculated by Storage service. But Blobs uploaded PutBlock/PutBlockList won’t have it, and client needs to calculate locally and set it to x-ms-blob-content-md5 Blob property explicitly. When client doesn’t do it, it’s empty. Above recommendation is based on such different cases. Reference: https://technet2.github.io/Wiki/blogs/windowsazurestorage/windows-azure-blob-md5-overview.html

For larger files storage does not calculate the MD5 hash of the full blob because each block is written separately. You can work around this by calculating and manually setting the md5 hash when uploading your files. Note the md5 hash is in base64. See the example below for how to upload a blob while calculating and setting the Content-MD5 property:

az storage blob upload -c test -n md5test -f ./test.img --content-md5 cat test.img | openssl dgst -md5 -binary | base64

MD5 hash checks on Azure Blob Storage files

Smaller files are not an issue since all the files smaller than 64MB will have Content-MD5 populated by the platform. For larger files, we can either have an azure function that can react on the events or perform a batch operation by spinning up some VMs and calculating MD5.

“The MD5 hash calculated from the downloaded data does not match the MD5 hash stored in the property of source: For more information see here

Hope the above answer helps! Please let us know do you have any further queries.
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

1 additional answer

Deleted

This answer has been deleted due to a violation of our Code of Conduct. The answer was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.

Comments have been turned off. Learn more

Share via

Azure Blob Storage stageBlock: Return contentMd5 as a header

1 additional answer

Your answer