Container Instances fail to start the majority of the time

Question

I have a function app that programmatically defines a container instance using the Azure Fluent SDK. This uses a ClamAV image from our private container registry and mounts a shared volume for use across our generated container instances.

The majority of the time this container fails to start and the error message received contains little/no detail as to the cause, yet sometimes the function app is able to succeed in creating and starting a container instance, with no changes whatsoever to when it fails.

The most common error I receive is:

Microsoft.Rest.Azure.CloudException: Long running operation failed with status 'Failed'. Additional Info:''
   at Microsoft.Rest.ClientRuntime.Azure.LRO.AzureLRO`2.CheckForErrors()
   at Microsoft.Rest.ClientRuntime.Azure.LRO.PutLRO`2.CheckForErrors()
   at Microsoft.Rest.ClientRuntime.Azure.LRO.AzureLRO`2.StartPollingAsync()
   at Microsoft.Rest.ClientRuntime.Azure.LRO.AzureLRO`2.BeginLROAsync()
   at Microsoft.Rest.Azure.AzureClientExtensions.GetLongRunningOperationResultAsync[TBody,THeader](IAzureClient client, AzureOperationResponse`2 response, Dictionary`2 customHeaders, CancellationToken cancellationToken)
   at Microsoft.Rest.Azure.AzureClientExtensions.GetLongRunningOperationResultAsync[TBody](IAzureClient client, AzureOperationResponse`1 response, Dictionary`2 customHeaders, CancellationToken cancellationToken)
   at Microsoft.Rest.Azure.AzureClientExtensions.GetPutOrPatchOperationResultAsync[TBody](IAzureClient client, AzureOperationResponse`1 response, Dictionary`2 customHeaders, CancellationToken cancellationToken)
   at Microsoft.Azure.Management.ContainerInstance.Fluent.ContainerGroupsOperations.CreateOrUpdateWithHttpMessagesAsync(String resourceGroupName, String containerGroupName, ContainerGroupInner containerGroup, Dictionary`2 customHeaders, CancellationToken cancellationToken)
   at Microsoft.Azure.Management.ContainerInstance.Fluent.ContainerGroupsOperationsExtensions.CreateOrUpdateAsync(IContainerGroupsOperations operations, String resourceGroupName, String containerGroupName, ContainerGroupInner containerGroup, CancellationToken cancellationToken)
   at Microsoft.Azure.Management.ContainerInstance.Fluent.ContainerGroupImpl.CreateResourceAsync(CancellationToken cancellationToken)
   at Microsoft.Azure.Management.ResourceManager.Fluent.Core.ResourceActions.Creatable`4.Microsoft.Azure.Management.ResourceManager.Fluent.Core.ResourceActions.IResourceCreator<IResourceT>.CreateResourceAsync(CancellationToken cancellationToken)
   at Microsoft.Azure.Management.ResourceManager.Fluent.Core.DAG.CreatorTaskItem`1.ExecuteAsync(CancellationToken cancellationToken)
   at Microsoft.Azure.Management.ResourceManager.Fluent.Core.DAG.TaskGroupBase`1.ExecuteNodeTaskAsync(DAGNode`1 node, CancellationToken cancellationToken)

By checking in the portal I can see that the container is in fact successfully created, but often has no IP and I'm unable to manually start it. Is there any reason why this would only work some of the time, and if so what's the reason for the failures?

Thanks.

Answer 1

The same problem is happening very often for me.
9 out of 10 containers failed to start today.

In my case, images are coming from dockerhub.

Microsoft.Rest.Azure.CloudException: Long running operation failed with status 'Failed'. Additional Info:''     at Microsoft.Rest.ClientRuntime.Azure.LRO.AzureLRO`2.CheckForErrors()     at Microsoft.Rest.ClientRuntime.Azure.LRO.PutLRO`2.CheckForErrors()     at Microsoft.Rest.ClientRuntime.Azure.LRO.AzureLRO`2.<StartPollingAsync>d__20.MoveNext()  --- End of stack trace from previous location where exception was thrown ---     at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)     at Microsoft.Rest.ClientRuntime.Azure.LRO.AzureLRO`2.<BeginLROAsync>d__15.MoveNext()  --- End of stack trace from previous location where exception was thrown ---     at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)     at Microsoft.Rest.Azure.AzureClientExtensions.<GetLongRunningOperationResultAsync>d__1`2.MoveNext()  --- End of stack trace from previous location where exception was thrown ---     at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)     at Microsoft.Rest.Azure.AzureClientExtensions.<GetLongRunningOperationResultAsync>d__0`1.MoveNext()  --- End of stack trace from previous location where exception was thrown ---     at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)     at Microsoft.Rest.Azure.AzureClientExtensions.<GetPutOrPatchOperationResultAsync>d__5`1.MoveNext()  --- End of stack trace from previous location where exception was thrown ---     at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)     at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()     at Microsoft.Azure.Management.ContainerInstance.Fluent.ContainerGroupsOperations.<CreateOrUpdateWithHttpMessagesAsync>d__8.MoveNext()  --- End of stack trace from previous location where exception was thrown ---     at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)     at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()     at Microsoft.Azure.Management.ContainerInstance.Fluent.ContainerGroupsOperationsExtensions.<CreateOrUpdateAsync>d__3.MoveNext()  --- End of stack trace from previous location where exception was thrown ---     at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)     at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()     at Microsoft.Azure.Management.ContainerInstance.Fluent.ContainerGroupImpl.<CreateResourceAsync>d__38.MoveNext()  --- End of stack trace from previous location where exception was thrown ---     at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)     at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()     at Microsoft.Azure.Management.ResourceManager.Fluent.Core.ResourceActions.Creatable`4.<Microsoft-Azure-Management-ResourceManager-Fluent-Core-ResourceActions-IResourceCreator<IResourceT>-CreateResourceAsync>d__15.MoveNext()  --- End of stack trace from previous location where exception was thrown ---     at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)     at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()     at Microsoft.Azure.Management.ResourceManager.Fluent.Core.DAG.CreatorTaskItem`1.<ExecuteAsync>d__6.MoveNext()  --- End of stack trace from previous location where exception was thrown ---     at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)     at System.Runtime.CompilerServices.TaskAwaiter.GetResult()     at Microsoft.Azure.Management.ResourceManager.Fluent.Core.DAG.TaskGroupBase`1.<ExecuteNodeTaskAsync>d__14.MoveNext()  --- End of stack trace from previous location where exception was thrown ---     at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)     at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()     at Microsoft.Azure.Management.ResourceManager.Fluent.Core.Extensions.Synchronize[TResult](Func`1 function)     at Microsoft.Azure.Management.ResourceManager.Fluent.Core.ResourceActions.Creatable`4.Create()   

Answer 2

I haven't heard from the OP in private messages and it's a bit difficult to diagnose the root cause of an intermittent issue without deeper insights and more details. If your Azure Subscriptions carry a support plan, I recommend opening a support case and working with a support engineer in a 1:1 setting to troubleshoot the root cause. If not, you can reach out to me at azcommunity at microsoft dotcom with subject line "Attn: murnun" and I'll be able to enable your subscription with a one-time free support case for resolving this issue -- be sure to include your subscription ID in your email.

Also, If it helps, we've had reports of ClamAV image not running reliably and switching to different image resolving issues, i.e: https://learn.microsoft.com/en-us/answers/questions/127797/vnet-intermittent-connection-problem-between-funct.html?childToView=167357#answer-167357