File Integrity monitoring for blobs in Archive storage

Question

File Integrity monitoring for blobs in Archive storage

Ruminski, Clayton - REE-ARS, Beltsville, MD 21

We are looking at implementing Azure Blob Archive storage for digital preservation storage. One concern we have is monitoring blobs for file integrity to ensure no changes to the data over time. Is there such a feature built in? Or must this be done via a third party tool? Doing some digging I found this third-party tool on GitHub: BlobIM but have not done any testing. We are trying to avoid extracting data from Archive storage, running third party fixity checks, and then putting them back into Archive storage. The costs associated with pulling large amounts of data from archive storage is prohibitive.

Accepted answer

1 additional answer

Your answer

Answer 1

SaiKishor-MSFT 17,336

@Ruminski, Clayton - REE-ARS, Beltsville, MD I reached out to our internal team regarding this and here is more information regarding your question-

"Archive storage validates data integrity in the backend as part of service compliance requirements. In any cases that customers want to validate explicitly, they need to rehydrate blobs to Hot, calculate hash, and compare with baseline hashes. The cost involved on rehydration is not avoidable."

As you mentioned, if this does not work for you, please feel free to use the 3rd party integrity check. Hope this helps!
Please let us know if you have any more questions and we will be glad to assist you further. Thank you!

Remember:

Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.

Want a reminder to come back and check responses? Here is how to subscribe to a notification.

Ruminski, Clayton - REE-ARS, Beltsville, MD 21 Reputation points

2022-09-01T14:18:41.263+00:00

Thank you for looking into this! This information will help us as we decided what we want/need.
Claus Juhl Knudsen 0 Reputation points

2023-07-10T08:07:05.4966667+00:00

Hi SaiKishor. Do you have any documentation or description of the processes and intervals at which "Archive storage validates data integrity"? Thanks

Answer 2

SaiKishor-MSFT 17,336

@Ruminski, Clayton - REE-ARS, Beltsville, MD Thanks for reaching out to Microsoft Q&A. I understand that you want monitoring options for your Blobs to ensure no changes happen to the data.

Here are some ways to protect your data using Azure resources-

You can monitor this information using Azure Monitor- https://learn.microsoft.com/en-us/azure/storage/blobs/monitor-blob-storage?tabs=azure-portal

You would first start with creating a Diagnostic Setting- https://learn.microsoft.com/en-us/azure/storage/blobs/monitor-blob-storage?tabs=azure-portal#creating-a-diagnostic-setting

And then view logged activity by using a Log Analytics query or you can go the destination that you are forwarding the logs to as setup in the diagnostics settings.

Here is a reference of Azure Blob Storage monitoring data collected using these steps - https://learn.microsoft.com/en-us/azure/storage/blobs/monitor-blob-storage-reference

If you want to rather lock this data in a way that data cannot be modified or deleted for a user-specified interval, you can store the data in a WORM (Write Once, Read Many) state. Please refer this for more information- https://learn.microsoft.com/en-us/azure/storage/blobs/immutable-storage-overview#summary-of-immutability-scenarios

Hope this helps. Please let us know if you have any more questions and we will be glad to assist you further. Thank you!

Remember:

Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.

Want a reminder to come back and check responses? Here is how to subscribe to a notification.

Ruminski, Clayton - REE-ARS, Beltsville, MD 21 Reputation points

2022-08-25T16:13:41.353+00:00

Thanks for this info. I did take a look at the documentation and see that in the diagnostic settings you can monitor read, write and delete operations. That seems to only monitor who and when does what to the storage account. But does it actively monitor if a file becomes corrupted or changed simply by sitting in storage over time rather than being changed or altered by a user? We are most concerned with the former.
SaiKishor-MSFT 17,336 Reputation points

2022-08-29T19:49:02.023+00:00

@Ruminski, Clayton - REE-ARS, Beltsville, MD Here are all the logged operations for the storage service- https://learn.microsoft.com/en-us/rest/api/storageservices/storage-analytics-logged-operations-and-status-messages#logged-operations

We do not have any monitoring specifically for files becoming corrupted etc., Do you have any example or use case where this happened so I can understand better? Thank you!
SaiKishor-MSFT 17,336 Reputation points

2022-08-30T20:08:17.787+00:00

@Ruminski, Clayton - REE-ARS, Beltsville, MD Any update?
Ruminski, Clayton - REE-ARS, Beltsville, MD 21 Reputation points

2022-08-31T12:50:13.97+00:00

We plan to use Archive tier blob storage for the long-term digital preservation storage of our data. One key component of digital preservation is to occasionally run integrity checks on data (sometimes monthly, or as needed). In past instances, data that has sat in storage for years has succumbed to data loss or corruption. It is not common, but running file integrity checks allows us to monitor this data periodically to identify any files that may have changed over time. That we can identify the corrupt file and replace it with a clean copy.

It seems like the only way to really do this in Azure is to remove the data from blob storage and run integrity checks with a third party tool and then put the data back in. Our concern with doing this is the potential cost associated with taking that data out.
SaiKishor-MSFT 17,336 Reputation points

2022-08-31T20:12:10.753+00:00

@Ruminski, Clayton - REE-ARS, Beltsville, MD Thanks for giving more clarity about your use case. Let me run this through our team and see if they have any other suggestions for your use case. I will update you soon. Thanks!

Share via

File Integrity monitoring for blobs in Archive storage

1 additional answer

Your answer