Container instances sporadically cannot connect to MSI endpoint for authentication token. Is the MSI endpoint unavailable?

Question

Background
I'm running a Data Factory pipeline which deploys two containers which use user assigned managed identities to connect to a Key Vault and retrieve secrets. One is running python and using the DefaultAzureCredential class to retrieve its token. The other one is connecting through the Azure CLI. The Data Factory pipeline has previously been running without problems for a couple of weeks, until two days ago. No changes has been made to the pipeline or container images since the problems arose.

Problem
The container instances started having connection problems to the MSI endpoint to retrieve the authentication token. In the Python app, I get

DefaultAzureCredential failed to retrieve a token from the included credentials. 

and in the container connecting by CLI, I get

ERROR: MSI endpoint is not responding. Please make sure MSI is configured correctly.\nError detail: MSI: Failed to acquire tokens after 12 times\nERROR: Please run 'az login' to setup account.\nERROR: Please run 'az login' to setup account.\nERROR: Please run 'az login' to setup account.\nERROR: Please run 'az login' to setup account. 

Sometimes one fails, sometimes both fail. Is there some problem with the MSI endpoint?

Edit: I should add, both are using User Assigned Managed Identities.

Answer 1

Hello @Oscar ,

Thank you for your post and I apologize for the delayed response!

This requires a deeper investigation by collecting debug logs also would request you to refer troubleshooting guide for details on how to diagnose various failure scenarios.

In additional to that, you can also test MSI endpoint by using HTTP GET REST call as mentioned here within Azure resources to make sure endpoint is accessible and If you see 403 or 404 responses, then something is wrong.

If non these fix the issue, then I request you file a support ticket if you have a support plan, else please send an email with the subject line “Attn: SivaKumarS” to AzCommunity[at]Microsoft[dot]com referencing this thread and your subscription id so that we will help you get a one-time free technical support.

Hope this helps. Thanks.

-----
Please "Accept the answer" if the information helped you. This will help us and others in the community as well.

Answer 2

Funny, am getting exact the same error in the MS Learning https://learn.microsoft.com/en-us/training/modules/aspnet-configurationbuilder/ where it performs this in the sandbox. Tried it a few times and it really doesnt return the object it in

$objectId = az ad signed-in-user show --query objectId -o tsv