Share via

Azure Container Instance - Managed Identity not available when container first deployed or restarted

Andrew Shepherd 1 Reputation point
2020-09-09T03:59:58.23+00:00

I am running an ASP.NET Core application in an Azure Container instance.

I am utilizing Managed Identities, and am working with a User Assigned managed identity.

I deploy this container using an ARM template.

Here's the thing:

  • When I first deploy the container, the managed identity is not available.
  • If I RESTART the container via the portal, the managed identity is not available
  • If, via the portal, I STOP the container then START the container, the managed identity is available. (Then, if I Restart the container it goes back to being unavailable)

I suspect that this is a defect with the Azure framework. Is there a bug registry that I would be able to post this to?

To diagnose this problem, I output the 'Azure-Identity' events to the console.

Here is the happy path. When I STOP then START the container as two separate actions, the console output is as follows: 

03:49:30.5490 Azure-Identity Informational: GetToken "ManagedIdentityCredential.GetToken invoked. Scopes: [ https://vault.azure.net/.default ] ParentRequestId: "
03:49:30.5541 Azure-Identity Informational: ProbeImdsEndpoint "Probiing IMDS endpoint for availability. Endpoint: http://169.254.169.254/metadata/identity/oauth2/token"
03:49:30.5580 Azure-Identity Informational: ImdsEndpointFound "IMDS endpoint is available. Endpoint: http://169.254.169.254/metadata/identity/oauth2/token"
03:49:30.7716 Azure-Identity Informational: GetTokenSucceeded "ManagedIdentityCredential.GetToken succeeded. Scopes: [ https://vault.azure.net/.default ] ParentRequestId:  ExpiresOn: 2020-09-10T03:49:30.0000000+00:00"
03:49:31.0249 Azure-Identity Informational: GetToken "ManagedIdentityCredential.GetToken invoked. Scopes: [ https://vault.azure.net/.default ] ParentRequestId: "
03:49:31.0250 Azure-Identity Informational: ProbeImdsEndpoint "Probiing IMDS endpoint for availability. Endpoint: http://169.254.169.254/metadata/identity/oauth2/token"
03:49:31.0256 Azure-Identity Informational: ImdsEndpointFound "IMDS endpoint is available. Endpoint: http://169.254.169.254/metadata/identity/oauth2/token"
03:49:31.0281 Azure-Identity Informational: GetToken "ManagedIdentityCredential.GetToken invoked. Scopes: [ https://vault.azure.net/.default ] ParentRequestId: "
03:49:31.0283 Azure-Identity Informational: ProbeImdsEndpoint "Probiing IMDS endpoint for availability. Endpoint: http://169.254.169.254/metadata/identity/oauth2/token"
03:49:31.0286 Azure-Identity Informational: ImdsEndpointFound "IMDS endpoint is available. Endpoint: http://169.254.169.254/metadata/identity/oauth2/token"
03:49:31.0291 Azure-Identity Informational: GetToken "ManagedIdentityCredential.GetToken invoked. Scopes: [ https://vault.azure.net/.default ] ParentRequestId: "
03:49:31.0292 Azure-Identity Informational: ProbeImdsEndpoint "Probiing IMDS endpoint for availability. Endpoint: http://169.254.169.254/metadata/identity/oauth2/token"
03:49:31.0295 Azure-Identity Informational: ImdsEndpointFound "IMDS endpoint is available. Endpoint: http://169.254.169.254/metadata/identity/oauth2/token"
03:49:31.0299 Azure-Identity Informational: GetToken "ManagedIdentityCredential.GetToken invoked. Scopes: [ https://vault.azure.net/.default ] ParentRequestId: "
03:49:31.0300 Azure-Identity Informational: ProbeImdsEndpoint "Probiing IMDS endpoint for availability. Endpoint: http://169.254.169.254/metadata/identity/oauth2/token"
03:49:31.0303 Azure-Identity Informational: ImdsEndpointFound "IMDS endpoint is available. Endpoint: http://169.254.169.254/metadata/identity/oauth2/token"
03:49:31.0337 Azure-Identity Informational: GetTokenSucceeded "ManagedIdentityCredential.GetToken succeeded. Scopes: [ https://vault.azure.net/.default ] ParentRequestId:  ExpiresOn: 2020-09-10T03:49:30.0000000+00:00"
03:49:31.0360 Azure-Identity Informational: GetTokenSucceeded "ManagedIdentityCredential.GetToken succeeded. Scopes: [ https://vault.azure.net/.default ] ParentRequestId:  ExpiresOn: 2020-09-10T03:49:30.0000000+00:00"
03:49:31.1415 Azure-Identity Informational: GetTokenSucceeded "ManagedIdentityCredential.GetToken succeeded. Scopes: [ https://vault.azure.net/.default ] ParentRequestId:  ExpiresOn: 2020-09-10T03:49:30.0000000+00:00"
03:49:31.1447 Azure-Identity Informational: GetTokenSucceeded "ManagedIdentityCredential.GetToken succeeded. Scopes: [ https://vault.azure.net/.default ] ParentRequestId:  ExpiresOn: 2020-09-10T03:49:30.0000000+00:00"

Here is the unhappy path, when I RESTART the container instance via the console.
(I have implemented a retry mechanism in my code. Even after a whole minute the IMDS endpoint does not become available) 

03:53:37.7169 Azure-Identity Informational: GetToken "ManagedIdentityCredential.GetToken invoked. Scopes: [ https://vault.azure.net/.default ] ParentRequestId: "
03:53:37.7220 Azure-Identity Informational: ProbeImdsEndpoint "Probiing IMDS endpoint for availability. Endpoint: http://169.254.169.254/metadata/identity/oauth2/token"
03:53:38.7297 Azure-Identity Informational: ImdsEndpointUnavailable "IMDS endpoint is did not respond. Endpoint: http://169.254.169.254/metadata/identity/oauth2/token"
03:53:38.7354 Azure-Identity Informational: GetTokenFailed "ManagedIdentityCredential.GetToken was unable to retrieve an access token. Scopes: [ https://vault.azure.net/.default ] ParentRequestId:  Exception: Azure.Identity.CredentialUnavailableException (0x80131500): ManagedIdentityCredential authentication unavailable, no managed identity endpoint found."
03:53:40.7422 Azure-Identity Informational: GetToken "ManagedIdentityCredential.GetToken invoked. Scopes: [ https://vault.azure.net/.default ] ParentRequestId: "
03:53:40.7425 Azure-Identity Informational: ProbeImdsEndpoint "Probiing IMDS endpoint for availability. Endpoint: http://169.254.169.254/metadata/identity/oauth2/token"
03:53:41.7434 Azure-Identity Informational: ImdsEndpointUnavailable "IMDS endpoint is did not respond. Endpoint: http://169.254.169.254/metadata/identity/oauth2/token"
03:53:41.7436 Azure-Identity Informational: GetTokenFailed "ManagedIdentityCredential.GetToken was unable to retrieve an access token. Scopes: [ https://vault.azure.net/.default ] ParentRequestId:  Exception: Azure.Identity.CredentialUnavailableException (0x80131500): ManagedIdentityCredential authentication unavailable, no managed identity endpoint found."

The C# code that is generating this event is here: https://github.com/Azure/azure-sdk-for-net/blob/8d38ce2bc8eaa0a34d053ae94f03bbae6f7ee1c0/sdk/identity/Azure.Identity/src/ImdsManagedIdentitySource.cs#L31

Azure Container Instances
Azure Container Instances
An Azure service that provides customers with a serverless container experience.
656 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. prmanhas-MSFT 17,891 Reputation points Microsoft Employee
    2020-09-11T15:22:43.96+00:00

    @Andrew Shepherd I had discussion internally and got to know that there is a bug reported for the issue and our Product Team is working on it. There is no ETA as such when the bug will be fixed but this is definitely in pipeline.

    As a workaround can you stop ACI then start from portal and not Restart it should work as pe suggestion.

    Hope it helps!!!

    Please 'Accept as answer' if it helped, so that it can help others in the community looking for help on similar topics

    1 person found this answer helpful.