Need suggestion for malware scan for blob in Azure Storage, file size approx. 100GB

Question

We're uploading virtual machine backup files using AzCopy with extension .vmdk, .vdi, etc and size are huge around 100GB and it's a single file to Azure Storage as a blob.

We thought to do malware scan once file uploaded. The Defender of Azure Storage has a limit of 2GB and we can't this file scan.

Could someone please suggest any solution here?

Answer 1

Hello Mohammad Ajmal Yazdani,

Greetings! Welcome to Microsoft Q&A Platform.

As of today, Defender for Storage’s malware scanner indeed only scans files upon upload in Azure Blob Storage. It ensures that any content uploaded to your storage accounts is thoroughly inspected for malicious elements before being stored. This proactive approach helps maintain the security and integrity of your data. However, there are some limitations which are not supported by Malware Scanning mentioned https://learn.microsoft.com/en-us/azure/defender-for-cloud/defender-for-storage-malware-scan#limitations where blobs that are larger than the 2GB limit will not be scanned. In the file meta data, you will find the scan has been aborted with this message “SAM259206: Scan aborted - the requested blob exceeded the maximum allowed size of 2 GB. Correlation Id: xxx”

I understand that you would like to scan the files that are around 100GB and hence below are some considerations that you can try to do the same,

1.Consider splitting them into smaller chunks before uploading. This way, each chunk remains within the size limit and can be scanned individually using the the Azure Blob Storage REST API's "Put Block" and "Put Block List" operations to upload large files in chunks. Here are the high-level steps you can follow:

1. Divide the large file into smaller chunks.
2. Create a new blob in the Azure Blob Storage account using the "Put Blob" operation. This operation creates a new blob or updates an existing blob with the specified content.
3. Upload each chunk of the file using the "Put Block" operation. This operation uploads a block of data to the specified block blob.
4. Commit the blocks to the blob using the "Put Block List" operation. This operation commits the list of blocks that have been uploaded to the specified block blob.

These articles can provide guidance for implementing this solution along with sample Java code:

https://learn.microsoft.com/en-us/rest/api/storageservices/understanding-block-blobs--append-blobs--and-page-blobs#about-block-blobs

Put Block (REST API) - Azure Storage | Microsoft Learn

Upload a blob with Java - Azure Storage | Microsoft Learn

azure-sdk-for-java/sdk/storage/azure-storage-blob/src/samples/java/com/azure/storage/blob/FileTransferExample.java at main · Azure/azure-sdk-for-java · GitHub

 

There are few similar SO threads discussion here: https://stackoverflow.com/questions/61481720/upload-video-in-chunks-azure-blob-storage

https://stackoverflow.com/questions/61857337/how-to-upload-a-large-file-in-chunks-with-parallelism-in-azure-sdk-v12

Additional information: How to upload and download blobs from Azure Blob Storage with JavaScript: 

https://learn.microsoft.com/en-gb/samples/azure-samples/azure-sdk-for-js-storage-blob-upload-download/upload-download-blobs-javascript/

2.For your use case with files that are greater than 2GB, try using Azure Logic Apps for handling malware scan results and copying the blob to another storage account. Logic Apps provide a simple, no-code approach to setting up response, although the response time might be slower than the event-driven code-based approach. Please see Option 1: Logic App based on Microsoft Defender for Cloud security alerts for steps on configuring this, the default is deleted but you can modify to move it.

Reference - https://learn.microsoft.com/en-us/azure/defender-for-cloud/defender-for-storage-malware-scan, https://github.com/MicrosoftDocs/azure-docs/blob/main/articles/defender-for-cloud/defender-for-storage-malware-scan.mdSimilar thread for reference - https://learn.microsoft.com/en-us/answers/questions/1457488/how-to-get-round-the-microsoft-defender-2gb-file-l

Hope this answer helps! Please let us know if you have any further queries. I’m happy to assist you further.

---

Please "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.