Failed blob writes with mcr.microsoft.com/azure-blob-storage:1.4.2-linux-amd64 in docker-compose stack

asked 2023-01-20T08:22:04.52+00:00
Andre du Toit 0 Reputation points

Hi

We are using mcr.microsoft.com/azure-blob-storage:1.4.2-linux-amd64 Docker image for a host-only/local blobstore.

Currently it is running in docker-compose, without IoTEdge runtime. This is a POC deployment on a desktop machine, so there is no integration with IoTEdge or Azure cloud at the moment (for various reasons).

The docker-compose configuration for the blob store looks something like this:

...
blobstore:
    image: mcr.microsoft.com/azure-blob-storage:$BLOBSTORE_VERSION
    hostname: blobstore
    restart: unless-stopped
    environment:
      - LOCAL_STORAGE_ACCOUNT_NAME=<REDACTED>
      - LOCAL_STORAGE_ACCOUNT_KEY=<REDACTED>
    ports:
      - "11002:11002"
    volumes:
      - blobstore:/blobroot

...
volumes:
  ...
  blobstore:

What might be pertinent (or a red herring?) to note here is that our Docker root directory is sitting on a secondary non-SSD (spinning) disk, 7TB in size, since the main SSD drive is too small to store the number of blobs (image files) for the period required.

We have a custom service for reading/writing the blobs from/to the blobstore. The setup works relatively well, except every now and then when we want to retrieve a given blob, it is missing. Looking at the blobstore docker logs I see a whole bunch of exceptions of the form:

[2023-01-17 07:31:52.636] [error   ] [tid 2251] UnexpectedXStoreError: An unexpected XStore error was encountered: {0}, p0="Microsoft.Cis.Services.Nephos.Common.Storage.StorageManagerException: No mappin
g available for WStoreException. HR: 54F (1359), Message: PutBlobAsync failed
 ---> Microsoft.Cis.Services.Nephos.Common.Storage.StorageManagerException: No mapping available for WStoreException. HR: 54F (1359), Message: PutBlobAsync failed
 ---> Microsoft.Cis.Services.Nephos.Common.Storage.StorageManagerException: No mapping available for WStoreException. HR: 54F (1359), Message: PutBlobAsync failed
 ---> Microsoft.AzureStack.Services.Storage.FrontEnd.WossProvider.WStoreException: PutBlobAsync failed
 ---> ErrorCode: 1359, Exception of type 'Microsoft.AzureStack.Services.Storage.Blob.BlobClientException' was thrown.
   at Microsoft.AzureStack.Services.Storage.Blob.WBlobClient.<>c__DisplayClass9_0.<PutBlobAsync>b__0()
   at System.Threading.Tasks.Task`1.InnerInvoke()
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AzureStack.Services.Storage.FrontEnd.WossProvider.WBlobStream.PutBlobSingleChunk()
   at Microsoft.AzureStack.Services.Storage.FrontEnd.WossProvider.WBlobStream.PutBlobAsync()
   at Microsoft.AzureStack.Services.Storage.FrontEnd.WossProvider.WBaseBlobObject.PutBlobAsync(String contentType, Int64 contentLength, Nullable`1 maxBlobSize, Byte[] serviceMetadata, Byte[] applicationM
etadata, Stream inputStream, CrcReaderStream crcReaderStream, Byte[] contentMD5, Boolean invokeGeneratePutBlobServiceMetadata, GeneratePutBlobServiceMetadata generatePutBlobServiceMetadata, ISequenceNumb
erUpdate sequenceNumberUpdate, OverwriteOption overwriteOption, IBlobObjectCondition condition)
   --- End of inner exception stack trace ---
   at Microsoft.AzureStack.Services.Storage.FrontEnd.WossProvider.WBaseBlobObject.PutBlobAsync(String contentType, Int64 contentLength, Nullable`1 maxBlobSize, Byte[] serviceMetadata, Byte[] applicationM
etadata, Stream inputStream, CrcReaderStream crcReaderStream, Byte[] contentMD5, Boolean invokeGeneratePutBlobServiceMetadata, GeneratePutBlobServiceMetadata generatePutBlobServiceMetadata, ISequenceNumb
erUpdate sequenceNumberUpdate, OverwriteOption overwriteOption, IBlobObjectCondition condition)
   at Microsoft.AzureStack.Services.Storage.Util.Await(Task t)
   at Microsoft.AzureStack.Services.Storage.FrontEnd.WossProvider.WBaseBlobObject.EndPutBlob(IAsyncResult asyncResult)
   --- End of inner exception stack trace ---
   at Microsoft.AzureStack.Services.Storage.FrontEnd.WossProvider.ExceptionManager.ReThrowIfWStoreException(Exception e)
   at Microsoft.AzureStack.Services.Storage.FrontEnd.WossProvider.WBaseBlobObject.EndPutBlob(IAsyncResult asyncResult)
   at Microsoft.Cis.Services.Nephos.Storage.Service.ServiceManager.RealServiceManager.PutBlobImpl(IAccountIdentifier identifier, String account, IStorageAccount storageAccount, String container, String blob, Stream inputStream, Int64 contentLength, IPutBlobProperties putBlobProperties, Boolean supportCrc64, Boolean calculateCrc64, Boolean storeCrc64, Boolean verifyCrc64, Boolean calculateMd5, Boolean storeMd5, Boolean verifyMd5, Boolean generationIdEnabled, Boolean isLargeBlockBlobRequest, Boolean is8TBPageBlobAllowed, BlobObjectCondition condition, BlobIdentifier blobIdentifier, OverwriteOption overwriteOption, TimeSpan timeout, RequestContext requestContext, AccessTier accessTier, String blobSasUrl, AsyncIteratorContext`1 context)+MoveNext()
   at AsyncHelper.AsyncIteratorContextBase.ExecuteIterator(Boolean inBegin)
   --- End of inner exception stack trace ---
   at Microsoft.Cis.Services.Nephos.Storage.Service.ServiceManager.RealServiceManager.EndPutBlob(IAsyncResult asyncResult)
   at Microsoft.Cis.Services.Nephos.Storage.Service.Protocols.Rest.HttpRestProcessor.PutBlobImpl(AsyncIteratorContext`1 async)+MoveNext()
   at AsyncHelper.AsyncIteratorContextBase.ExecuteIterator(Boolean inBegin)
   --- End of inner exception stack trace ---
   at Microsoft.Cis.Services.Nephos.Common.Protocols.Rest.BasicHttpProcessorWithAuthAndAccountContainer`1.EndPerformOperation(IAsyncResult ar)

These exception traces appear to occur quite frequently and may explain the missing blobs issue. I have been unable to find any meaningful info regarding the exceptions listed above and how to remedy them. More or less the same stack runs fine on my SSD-only dev machine, without any of these errors. The main difference between my dev machine and the site machine: the / SSD drive used as Docker root on my dev machine and the secondary non-SSD drive used as Docker root on the site machine. Also the blobstore on site has been running for about a month now, so perhaps the problem only starts manifesting itself when the blobstore content reaches a certain number of blobs/size?

Any feedback on what causes the above exception and how to remedy it would be greatly appreciated.

Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
1,388 questions
{count} votes

1 answer

Sort by: Most helpful
  1. answered 2023-01-20T13:34:28.7833333+00:00
    Andre du Toit 0 Reputation points

    I am now at the point where I am going to sic gdb on the dotnet process to get some hints, once I get the attach to work ;)

    No comments