Support Request for Deploy multiple module instances

Question

Hi Team,

I am following this MicrosoftDoc Deploy the Azure Blob Storage on IoT Edge module to your device

On this above article section - Deploy multiple module instances

We have deployed multiple storage modules with different host ports (11002 and 11003) as mentioned in the docs, we are able to create both the storage account separately.

For the first blob module, we are able to connect with storage explorer but for the second, storage explorer connection is not working.

So, while creating a new container OR adding a blob to second storage account two exposed on 11003 we are an error "Microsoft.Cis.Services.Nephos.Common.Authentication.AuthenticationFailureException: Signature did not match. String to sign used was rwl"

we also observed that the logs shows 11002 port for the storage module exposed on 11003
"[tid 56] Info: Accepting {0} request for '{1}'. RemoteEndPoint='{2}', Via='{3}', Expect='{4}', Connection='{5}', Version='{6}', ClientRequestId='{7}', UserAgent='{8}' DurationStayedInQueueInMs='{9}', RemainingRequestsInQueueAtEnqueue='{10}', RemainingRequestsInQueueAtDequeue='{11}', WasInHighPriorityQueue='{12}', ToSetKeepAliveToFalse='{13}', ActiveWorkerThreads='{14}', ActiveCompletionThreads='{15}' PassedInActivityId={16}, EnqueueTime={17}, DequeueTime={18}, RequestTraceActvityId = '{19}', p0="HEAD", p1="http://172.31.130.191:11002/devstoreaccount2/loader-container/S1001691%20-%20Copy.JPG?se=2022-11-20T13%3A52%3A22Z&sig=XXXXX&sp=rwl&sr=c&sv=2021-08-06&timeout=901", p2="172.31.128.1:64362", p3="", p4="", p5="", p6="2020-10-02", p7="7513e139-d4d7-4cf9-4a2b-f57e6b1d5781", p8="Microsoft Azure Storage Explorer", p9="0.0291", p10="0", p11="0", p12="False", p13="False", p14="1", p15="0", p16="null", p17="2022-10-21T13:52:24.5381891Z", p18="2022-10-21T13:52:24.5382140Z", p19="583596a7-1d79-4077-8036-fd81543fb904"
[2022-10-21 13:52:24.555] [info ] [tid 128] Info: Processing exception: {0}, p0="Microsoft.Cis.Services.Nephos.Common.Authentication.AuthenticationFailureException: Signature did not match. String to sign used was rwl

2022-11-20T13:52:22Z
/blob/devstoreaccount2/loader-container"

ASK: How do we fix this issue?
The storage account exposed on 11002 works as expected, but facing an issue for the second one when using a custom port 11003 ?

257648-errorlogs.txt

Answer 1

Hi @sekhar kumar ,

I have similar behavior when using Azure Storage Explorer or azCopy, even worse: While I can create containers in both blob module instances, I am not able to copy anything to one of the blob-module instances. Although it works well when using simple .Net code. Thus, I assume it is related to azCopy what is used by Azure Storage Explorer.

            string accountName = "localstorageacc2";  
            string accountKey = "EWby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==";  
            string blobUriString = "http://172.17.103.79:11003/localstorageacc2/localblobcont2";  
            string localFilePath = "test.txt";  
      
            // create SharedKeyCredential  
            StorageSharedKeyCredential credential = new StorageSharedKeyCredential(accountName, accountKey);  
      
            // create blob container client  
            BlobContainerClient container = new BlobContainerClient(new Uri(blobUriString), credential);  
              
            // create blob client and upload   
            BlobClient blobClient = container.GetBlobClient("newFile_" + Guid.NewGuid().ToString() + ".txt");  
            await blobClient.UploadAsync(localFilePath, true);  
      
            Console.WriteLine("Listing all blobs in container {0}\n", blobUriString);  
            // list all blobs in container  
            await foreach (BlobItem blobItem in container.GetBlobsAsync())  
            {  
                Console.WriteLine("\t" + blobItem.Name);  
            }  

Additional information:
The issue is between Azure Storage Explorer and the IoT Edge Blob Storage module as the module uses a lower API version than the public Azure Storage service, see issue 5999
After selecting Target Azure Stack Hub APIs in Azure Storage Explorer, uploading to both module instances worked like a charm.

----------

If your issue is solved and to help other users, please mark this as best answer and feel free to upvote.

Answer 2

Hi @sekhar kumar ,

Besides setting multiple port bindings for the host environment, did you also configure different blob storage paths for your blob modules?
Explained in how-to-deploy-blob

If you want to deploy multiple instances of the Azure Blob Storage on IoT Edge module, you need to provide a different storage path and change the HostPort value that the module binds to.

Answer 3

Hi @chbeier , thanks for the reply.

Below is the configuration of my edge where I am having two blob modules. For the second one the connection is not working. Could you please advise?

"FirstBlobStorageonIoTEdge": {
"version": "1.0",
"type": "docker",
"status": "running",
"restartPolicy": "always",
"settings": {
"image": "mcr.microsoft.com/azure-blob-storage:latest",
"createOptions": {
"HostConfig": {
"Binds": [
"/srv/containerdata:/blobroot"
],
"PortBindings": {
"11002/tcp": [
{
"HostPort": "11002"
}
]
}
}
}
},
"env": {
"LOCAL_STORAGE_ACCOUNT_NAME": {
"value": "devstoreaccount1"
},
"LOCAL_STORAGE_ACCOUNT_KEY": {
"value": "Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw=="
}
}
},

      "SecondBlobStorageonIoTEdge": {  
        "version": "1.0",  
        "type": "docker",  
        "status": "running",  
        "restartPolicy": "always",  
        "settings": {  
          "image": "mcr.microsoft.com/azure-blob-storage:latest",  
          "createOptions": {  
            "HostConfig": {  
              "Binds": [  
                "/srv/containerdata2:/blobroot"  
              ],  
              "PortBindings": {  
                "11002/tcp": [  
                  {  
                    "HostPort": "11003"  
                  }  
                ]  
              }  
            }  
          }  
        },  
        "env": {  
          "LOCAL_STORAGE_ACCOUNT_NAME": {  
            "value": "devstoreaccount2"  
          },  
          "LOCAL_STORAGE_ACCOUNT_KEY": {  
            "value": "EWby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw=="  
          }  
        }  
      }  

Answer 4

Hi @chbeier , thanks for the reply.

I am following the same way as explained the docs but still it is not working.

Please find the below configurations of two edge blob modules

FisrtStorage Account:

DefaultEndpointsProtocol=http;BlobEndpoint=http://172.23.80.35:11002/devstoreaccount1;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==

"createOptions": {
"HostConfig": {
"Binds": [
"/srv/containerdata:/blobroot"
],
"PortBindings": {
"11002/tcp": [
{
"HostPort": "11002"
}
]
}
}
}

Second Storage Account:

DefaultEndpointsProtocol=http;BlobEndpoint=**http://172.23.80.35:11003/devstoreaccount2**;AccountName=devstoreaccount2;AccountKey=JaknIU4eGvw5/nUclpZvGQZitVRg4EUhjzK7/Ci/AlTyluUzbPVW6S4Uq1YuoWtSCjQY6QS5A0L/U27o8YV6Dg==

"createOptions": {
"HostConfig": {
"Binds": [
"/srv/containerdata2:/blobroot"
],
"PortBindings": {
"11002/tcp": [
{
"HostPort": "11003"
}
]
}
}
}

First Storage Account is working as expected.

Here, for the reference I am pasting the error while uploading blobs to second storage account

[info ] [tid 33] Info: Accepting {0} request for '{1}'. RemoteEndPoint='{2}', Via='{3}', Expect='{4}', Connection='{5}', Version='{6}', ClientRequestId='{7}', UserAgent='{8}' DurationStayedInQueueInMs='{9}', RemainingRequestsInQueueAtEnqueue='{10}', RemainingRequestsInQueueAtDequeue='{11}', WasInHighPriorityQueue='{12}', ToSetKeepAliveToFalse='{13}', ActiveWorkerThreads='{14}', ActiveCompletionThreads='{15}' PassedInActivityId={16}, EnqueueTime={17}, DequeueTime={18}, RequestTraceActvityId = '{19}', p0="HEAD", p1="http://172.23.80.35:11002/devstoreaccount2/loader-container/S1001691%20-%20Copy.JPG?se=2022-12-09T05%3A03%3A50Z&sig=XXXXX&sp=rwl&sr=c&sv=2021-08-06&timeout=901", p2="172.23.80.1:64786", p3="", p4="", p5="", p6="2020-10-02", p7="c523fca1-0db5-4a46-5ca2-847ed5e32f8b", p8="Microsoft Azure Storage Explorer", p9="2.9105", p10="0", p11="0", p12="False", p13="False", p14="1", p15="0", p16="null", p17="2022-11-09T05:03:51.9381922Z", p18="2022-11-09T05:03:51.9390689Z", p19="3f7b93a6-7e5b-4002-8353-8c633c50f6c2"
[2022-11-09 05:03:51.980] [info ] [tid 160] Info: Processing exception: {0}, p0="Microsoft.Cis.Services.Nephos.Common.Authentication.AuthenticationFailureException: Signature did not match. String to sign used was rwl

2022-12-09T05:03:50Z
/blob/devstoreaccount2/loader-container

2021-08-06
c

---> Microsoft.Cis.Services.Nephos.Common.Authentication.AuthenticationFailureException: Signature did not match. String to sign used was rwl

2022-12-09T05:03:50Z
/blob/devstoreaccount2/loader-container

2021-08-06
c

at Microsoft.Cis.Services.Nephos.Common.Authentication.XFEBlobAuthenticationManager.AuthenticateImpl(IStorageAccount storageAccount, RequestContext requestContext, NephosUriComponents uriComponents, GetStringToSignCallback getStringToSignCallback, TimeSpan timeout, AsyncIteratorContext1 context)+MoveNext() at AsyncHelper.AsyncIteratorContextBase.ExecuteIterator(Boolean inBegin) --- End of inner exception stack trace --- at Microsoft.Cis.Services.Nephos.Common.Authentication.XFEBlobAuthenticationManager.EndAuthenticate(IAsyncResult asyncResult) at Microsoft.Cis.Services.Nephos.Common.Protocols.Rest.BasicHttpProcessorWithAuthAndAccountContainer1.ProcessImpl(AsyncIteratorContext`1 async)+MoveNext()"
[2022-11-09 05:03:51.981] [info ] [tid 160] Info: Setting status description as "{0}", p0="Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature."
[2022-11-09 05:03:51.989] [info ] [tid 160] Perf: PerfCounters: Account={0} Operation={1} on Container={2} with Status={3} RequestHeaderSize={4} RequestSize={5} ResponseHeaderSize={6} ResponseSize={7} ErrorResponseByte={8} TimeInMs={9} ProcessingTimeInMs={10} UserTimeoutInMs={11} OperationTimeoutInMs={12} MaxAllowedTimeoutInMs={13} SlaUsedInMs={14} ReadLatencyInMs={15} WriteLatencyInMs={16} ClientIP={17} UserAgent='{18}' RequestVersion='{19}' ProcessorVersionUsed='{20}' RequestUrl='{21}' ClientRequestId='{22}' MeasurementStatus={23} HttpStatusCode={24} TotalFeTimeInMs={25} TotalTableServerTimeInMs={26} TotalTableServerRoundTripCount={27} TotalPartitionWaitTimeInMs={28} TotalXStreamTimeInMs={29} TotalXStreamRoundTripCount={30} SmbOpLockBreakLatency={31} LastTableServerInstanceName={32} LastTableServerErrorCode={33} TotalAccountCacheWaitTimeInMs={34} TotalContainerCacheWaitTimeInMs={35} InternalStatus={36} RequestContentType='{37}' ResponseContentType='{38}' PartitionKey='{39}' ItemsReturned='{40}' BatchOperationCount='{41}' LastXStreamErrorCode={42} AuthenticationType='{43}' AccountConcurrentReq='{44}' OverallConcurrentReq='{45}' Range='{46}' EntityType='{47}' {48} LastTSPartition={49} TotalXCacheTimeInMs={50} TotalXCacheRoundTripCount={51} TotalTableServerOnlyTimeInMicroSeconds={52} TotalAuthTimeInMs={53} AuthenticationTimeInMs={54} AuthorizationTimeInMs={55} GetAuthenticationDataTimeInMs={56} GetAuthorizationDataTimeInMs={57} VnetPolicyCheckTimeInMs={58} PerfFeatureFlags={59}, p0="devstoreaccount2", p1="AuthenticationFailure", p2="null", p3="AuthenticationFailed", p4="0", p5="0", p6="0", p7="0", p8="0", p9="NA", p10="NA", p11="NA", p12="NA", p13="NA", p14="NA", p15="NA", p16="NA", p17="172.23.80.1:64786", p18="Microsoft Azure Storage Explorer, 1.26.1, win32, azcopy-node/2.7.1-rc.1 (win32) AzCopy/10.16.0 Azure-Storage/0.15 (go1.17.9; Windows_NT)", p19="2020-10-02", p20="2020-10-02", p21="http://172.23.80.35:11003/devstoreaccount2/loader-container/S1001691%20-%20Copy.JPG?se=2022-12-09T05%3A03%3A50Z&sig=XXXXX&sp=rwl&sr=c&sv=2021-08-06&timeout=901", p22="c523fca1-0db5-4a46-5ca2-847ed5e32f8b", p23="AuthenticationFailure", p24="403", p25="NA", p26="NA", p27="NA", p28="NA", p29="NA", p30="NA", p31="NA", p32="NA", p33="NA", p34="0", p35="0", p36="AuthenticationError", p37="", p38="", p39="", p40="0", p41="0", p42="NA", p43="NA", p44="", p45="1", p46="NA", p47="", p48="", p49="NA", p50="NA", p51="NA", p52="0", p53="18.667", p54="18.667", p55="0", p56="0", p57="0", p58="NA", p59="null"
[2022-11-09 05:03:52.334] [info ] [tid 160] [BlobInterface.cc:1494] [ListBlobsInOrder] ListBlobsInOrder received. Container:loader-container BlobNameStart:null MaxBlobNames:1 OrderType:0 Flags:1
[2022-11-09 05:03:52.335] [info ] [tid 160] [BlobInterface.cc:1528] [ListBlobsInOrder] ListBlobsInOrder completed. Container:loader-container BlobEntryCount:0 NextBlobName:null
[2022-11-09 05:03:53.336] [info ] [tid 160] [BlobInterface.cc:1494] [ListBlobsInOrder] ListBlobsInOrder received. Container:loader-container BlobNameStart:null MaxBlobNames:1 OrderType:0 Flags:1
[2022-11-09 05:03:53.336] [info ] [tid 160] [BlobInterface.cc:1528] [ListBlobsInOrder] ListBlobsInOrder completed. Container:loader-container BlobEntryCount:0 NextBlobName:null

Answer 5

Hello sekhar kumar

Thanks for your patience on this issue!

It seems that there is an authentication issue with the second storage account.

Here are a few things you can try:

1. Double-check the account name and account key for the second storage account. Make sure they are correct.
2. Make sure that the request URL includes the correct container name and blob name.
3. Ensure that the date and time on your local machine are set correctly.
4. Check if the second storage account has firewall rules enabled. If so, make sure that your IP address is added to the list of allowed IP addresses.
5. Try using HTTPS instead of HTTP to ensure a secure connection.
6. Try to regenerate the storage account key for the second storage account and try again.
7. Check the connection string for any typos or syntax errors.

If none of these steps work, you may need to contact Microsoft Azure support for further assistance.

Please comment in the below section, so that we get in touch with you!