How to update managed identity for connect AKS to ACR

Question

How to update managed identity for connect AKS to ACR

p.shapurau 16

I have a long ago created cluster connected to the Azure container register. A year later, the managed identity for the cluster expired and my cluster does not see containers to the Azure container register. Please tell me step by step and specifically what should I do, how to update the managed identity?
I have seen many links and articles on this topic on the Internet, but nothing specific. I think the solution should be clear and simple. this question will often come up in people after a year.

SRIJIT-BOSE-MSFT 4,291 Reputation points Microsoft Employee

2021-11-08T09:24:46.743+00:00

@p.shapurau , thank you for sharing your concern with us.

There might be a confusion. Can you please check if you are using managed identities or a service principal with the AKS cluster? Because,

Clusters using service principals eventually reach a state in which the service principal must be renewed to keep the cluster working. Managing service principals adds complexity, which is why it's easier to use managed identities instead.

Managed identities are essentially a wrapper around service principals, and make their management simpler. Credential rotation for MI happens automatically every 46 days according to Azure Active Directory default.

[Reference]

If you are using a Service Principal with the AKS cluster please follow the steps here to rotate the Service Principal credentials and update the AKS cluster with the new credentials.

[contd]
SRIJIT-BOSE-MSFT 4,291 Reputation points Microsoft Employee

2021-11-08T09:28:26.94+00:00
@p.shapurau ,

But first please perform the following checks:

Run az aks check-acr to find the error in ACR accessibility from the AKS cluster. [Reference]

Confirm your AKS cluster is using managed identity with the following CLI command [Reference]:
az aks show -g <RGName> -n <ClusterName> --query "servicePrincipalProfile"
If the cluster is using managed identities, you will see a clientId value of "msi". A cluster using a Service Principal instead will instead show the object ID. For example: Output: {
"clientId": "msi"
}
p.shapurau 16 Reputation points

2021-11-08T12:54:42.573+00:00

Yes, used managed identities. For this cluster, now I have already tried and updated the cluster to the mode service principal, And reconnected ACR.
But, The question is, what should I do if I have a default cluster and after exactly a year the connection to ACR stopped working ?

Here's what happened in my cluster when I tried to access ACR and deploy a container:

Failed to pull image "acr_name/container_name": [rpc error: code = Unknown desc = failed to pull and unpack image "acr_name/container_name": failed to resolve reference "acr_name/container_name": failed to authorize: failed to fetch oauth token: unexpected status: 401 Unauthorized, rpc error: code = Unknown desc = failed to pull and unpack image "acr_name/container_name": failed to resolve reference "acr_name/container_name": failed to authorize: failed to fetch anonymous token: unexpected status: 401 Unauthorized]

What should I do step by step, after a year of using the cluster, to renew the connection to my ACR in this case?

This command does not help in this case: az aks update -n namecluster -g namegroup --attach-acr nameacr

I was getting the following message:
SRIJIT-BOSE-MSFT 4,291 Reputation points Microsoft Employee

2021-11-09T11:53:05.26+00:00
@p.shapurau ,

If you are using Service Principals, then after the credentials expire,

Get the Service Principal ID of the AKS cluster using: SP_ID=$(az aks show --resource-group myResourceGroup --name myAKSCluster \
--query servicePrincipalProfile.clientId -o tsv)

Reset the Service Principal secret and store it in a variable like: SP_SECRET=$(az ad sp credential reset --name "$SP_ID" --query password -o tsv)

Update AKS cluster with new service principal credentials: az aks update-credentials \
--resource-group myResourceGroup \
--name myAKSCluster \
--reset-service-principal \
--service-principal "$SP_ID" \
--client-secret "$SP_SECRET"
If you are using managed identities with the AKS cluster, the managed identity secrets are refreshed every 46 days. The AKS cluster does not hold managed identity secrets. We are keeping the thread open for Managed Identity experts to share their view on troubleshooting steps if the managed identity credential rotation has discrepancies.
Suresh Bettadapur 46 Reputation points

2022-02-03T07:36:11.677+00:00

@SRIJIT-BOSE-MSFT I think original question is not answered. I have a somewhat similar situation. When the AKS cluster was created, I have used User assigned MI. That MI is of DevOps agent identity which has changed now. Now I have to update the MI in the AKS and am facing issues. I don't find any documentation that explains this. Can someone help here?

Thanks
Suresh

SRIJIT-BOSE-MSFT 4,291 Reputation points Microsoft Employee

2021-11-08T09:24:46.743+00:00

@p.shapurau , thank you for sharing your concern with us.

There might be a confusion. Can you please check if you are using managed identities or a service principal with the AKS cluster? Because,

Clusters using service principals eventually reach a state in which the service principal must be renewed to keep the cluster working. Managing service principals adds complexity, which is why it's easier to use managed identities instead.

Managed identities are essentially a wrapper around service principals, and make their management simpler. Credential rotation for MI happens automatically every 46 days according to Azure Active Directory default.

[Reference]

If you are using a Service Principal with the AKS cluster please follow the steps here to rotate the Service Principal credentials and update the AKS cluster with the new credentials.

[contd]
SRIJIT-BOSE-MSFT 4,291 Reputation points Microsoft Employee

2021-11-08T09:28:26.94+00:00

@p.shapurau ,

But first please perform the following checks:

Run az aks check-acr to find the error in ACR accessibility from the AKS cluster. [Reference]

Confirm your AKS cluster is using managed identity with the following CLI command [Reference]:
az aks show -g <RGName> -n <ClusterName> --query "servicePrincipalProfile"
If the cluster is using managed identities, you will see a clientId value of "msi". A cluster using a Service Principal instead will instead show the object ID. For example: Output: {
"clientId": "msi"
}
p.shapurau 16 Reputation points

2021-11-08T12:54:42.573+00:00

Yes, used managed identities. For this cluster, now I have already tried and updated the cluster to the mode service principal, And reconnected ACR.
But, The question is, what should I do if I have a default cluster and after exactly a year the connection to ACR stopped working ?

Here's what happened in my cluster when I tried to access ACR and deploy a container:

Failed to pull image "acr_name/container_name": [rpc error: code = Unknown desc = failed to pull and unpack image "acr_name/container_name": failed to resolve reference "acr_name/container_name": failed to authorize: failed to fetch oauth token: unexpected status: 401 Unauthorized, rpc error: code = Unknown desc = failed to pull and unpack image "acr_name/container_name": failed to resolve reference "acr_name/container_name": failed to authorize: failed to fetch anonymous token: unexpected status: 401 Unauthorized]

What should I do step by step, after a year of using the cluster, to renew the connection to my ACR in this case?

This command does not help in this case: az aks update -n namecluster -g namegroup --attach-acr nameacr

I was getting the following message:
SRIJIT-BOSE-MSFT 4,291 Reputation points Microsoft Employee

2021-11-09T11:53:05.26+00:00

@p.shapurau ,

If you are using Service Principals, then after the credentials expire,

Get the Service Principal ID of the AKS cluster using: SP_ID=$(az aks show --resource-group myResourceGroup --name myAKSCluster \
--query servicePrincipalProfile.clientId -o tsv)

Reset the Service Principal secret and store it in a variable like: SP_SECRET=$(az ad sp credential reset --name "$SP_ID" --query password -o tsv)

Update AKS cluster with new service principal credentials: az aks update-credentials \
--resource-group myResourceGroup \
--name myAKSCluster \
--reset-service-principal \
--service-principal "$SP_ID" \
--client-secret "$SP_SECRET"
If you are using managed identities with the AKS cluster, the managed identity secrets are refreshed every 46 days. The AKS cluster does not hold managed identity secrets. We are keeping the thread open for Managed Identity experts to share their view on troubleshooting steps if the managed identity credential rotation has discrepancies.
Suresh Bettadapur 46 Reputation points

2022-02-03T07:36:11.677+00:00

@SRIJIT-BOSE-MSFT I think original question is not answered. I have a somewhat similar situation. When the AKS cluster was created, I have used User assigned MI. That MI is of DevOps agent identity which has changed now. Now I have to update the MI in the AKS and am facing issues. I don't find any documentation that explains this. Can someone help here?

Thanks
Suresh

How to update managed identity for connect AKS to ACR

Activity