Cannot see blob data in VM

Question

If I attempt to upload multiple files via the web portal, 75% of them will fail with 'InvalidBlobOrBlock'
I am therefore having to re-upload these using the Microsoft Azure Storage Explorer program. The other issue with this is: Whether or not I upload via the web portal or the storage explorer, when I then go to my Linux VM and perform a ls check on the directory, nothing is showing. I uploaded some files on Sunday and they’ve only just appeared in the VM mount today. I’ve also uploaded some this morning and they are still not showing in the VM. Why is this?

I am using PuTTy to SSH into the Linux VM

So far, I've been told there is latency on the data mounting but no one can see why

Answer 1

@Marc Hughes Welcome to Microsoft Q&A Forum, Thank you for posting your query here!

How much GB Of data are you trying upload?

Based on the error message it depend on a variety of factors, including your connection quality and bandwidth, among other things. I would try to keep the chunksizebytes at default and decreasing the number of workers in the range of 2 to 8 to see if that alleviates your problem. Then you can increase that number with something that's a good tradeoff between speed and consistency in transfer. ensure that your block ids of your blocks are of same length. If the block ids of your blocks are of different length, you’ll get this error.

Troubleshooting InvalidBlock 'The specified block list is invalid’ based errors

There is similar discussion thread here, please refer to the suggestion and let me know the status

You can also use Azcopy tool(Fast Data Transfer is a tool for fast upload of data into Azure – up to 4 terabytes per hour from a single client machine. It moves data from your premises to Blob Storage, to a clustered file system, or direct to an Azure VM. It can also move data between Azure regions)

This can be happen if there is an Uncommitted Block List, and to make sure that there is an Uncommitted Block List, customer have to validate this with REST API Call (https://learn.microsoft.com/en-us/rest/api/storageservices/get-block-list). There is no equivalent feature in Azure Portal or CLI.

• Verify the Uncommitted Block List by using Get Block List API Call, and check the length of existing block ID is same with block ID in failed request. You can use a SDK to do that.
• Commit empty block list to create an empty blockblob, which will remove all previous blocks. Then try to upload again(Can self-mitigate by doing the following: Delete the blob, then issue a PutBlob. This will resolve the issue by clearing the uncommitted block list)
• How to mount Blob storage as a file system with blobfuse
• Mount Blob storage on Linux VM using Managed Identities or Service Principal with Blobfuse

If the issue still persist, can you please share the screenshot of the error message?

Hope this helps!
Kindly let us know if the above helps or you need further assistance on this issue.

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.

Disclaimer: This response contains a reference to a third-party World Wide Web site. Microsoft is providing this information as a convenience to you. Microsoft does not control these sites and has not tested any software or information found on these sites; therefore, Microsoft cannot make any representations regarding the quality, safety, or suitability of any software or information found there.
There are inherent dangers in the use of any software found on the Internet, and Microsoft cautions you to make sure that you completely understand the risk before retrieving any software from the Internet.

Answer 2

@Marc Hughes Time sync is important for security and event correlation. Sometimes it is used for distributed transactions implementation. Time accuracy between multiple computer systems is achieved through synchronization. Synchronization can be affected by multiple things, including reboots and network traffic between the time source and the computer fetching the time. Learn more: https://learn.microsoft.com/en-us/azure/virtual-machines/linux/time-sync

• Allow interne to the TIME Sever’s using UDR(user defined routes) with the next hop being internet.
• Use Separate server as LOCAL TIME SERVER and give the IP of the local time server in all the clients.

In this scenario you can blobfuse , Latency could be with the Linux VM , region, bandwidth we need to check what exactly is causing the issue (Linux VM may be having time sync delay )

Azure Storage provides two latency metrics for block blobs. These metrics can be viewed in the Azure portal:
End-to-end (E2E) latency measures the interval from when Azure Storage receives the first packet of the request until Azure Storage receives a client acknowledgment on the last packet of the response.
Server latency measures the interval from when Azure Storage receives the last packet of the request until the first packet of the response is returned from Azure Storage.

This article will help you in resolving the Azure blob latency: https://techcommunity.microsoft.com/t5/azure-paas-blog/how-to-isolate-latency-issue-for-azure-storage-account/ba-p/1430656

Verify throughput and latency metrics for a storage account

Hope this helps!
Kindly let us know if the above helps or you need further assistance on this issue.

---------------------------------------------------------------------------------------------------------------------------------------

Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.