Private ACR with AKS: Failed to pull image... failed to resolve reference... unexpected status from HEAD request... 403 Forbidden

LyTien Dung 5

The kubelet try to pull an image from a private ACR (role assignment ArcPull granted for the -agentpool managed identity).
It gets an error like Failed to pull image... failed to resolve reference... unexpected status from HEAD request... 403 Forbidden
I think the network is ok, but if there is something missing about the permission on ACR private endpoint configuration.

In the cluster, using a pod to test network with ACR FQDN and private IP. I get these information:

nc -zv tempregistrydev.azurecr.io 443
Connection to tempregistrydev.azurecr.io (10.5.144.6) 443 port [tcp/https] succeeded!

telnet tempregistrydev.azurecr.io 443
Connected to tempregistrydev.azurecr.io

But, this ICMP ping is not passed
ping 10.5.144.6 PING 10.5.144.6 (10.5.144.6) 56(84) bytes of data.

Here are results after running Troubleshoot private link DNS configuration and connectivity issues

Have you provided a valid private endpoint resource?
Yes

Is your private endpoint in status 'Succeeded'?
Yes

What's the status of the private endpoint connection?
Approved

How are you trying to connect to the private endpoint from the client resource?
FQDN

What result do you get when resolving the FQDN?
Private IP

Run TCP pings to the FQDN or the private IP address and the corresponding TCP port. Is the TCP ping working?
Yes

Need further assistance If you have followed this troubleshooting guidance and reached this point, everything seems properly configured.

Prrudram-MSFT 24,661 Reputation points

2024-08-02T13:31:41.99+00:00
Hello @LyTien Dung
Thank you for reaching out to the Microsoft Q&A platform.

It seems like you are trying to pull an image from a private Azure Container Registry (ACR) using a Kubernetes cluster, but you are getting a "403 Forbidden" error. You have tested the network connection to the ACR using a pod and it seems to be working fine, but you are not able to ping the ACR's private IP address.

Based on the information you have provided, it seems like the private endpoint for the ACR is properly configured and the network connection to the ACR is working fine. However, you are still getting a "403 Forbidden" error when trying to pull the image from the ACR. One possible reason for this issue is that the managed identity used by the AKS cluster to access the ACR does not have the necessary permissions to pull the image. To resolve this issue, you can try granting the AcrPull role to the managed identity used by the AKS cluster.

Here are the steps to grant the AcrPull role to the managed identity used by the AKS cluster:

Open the Azure portal and navigate to the ACR resource.

Click on "Access keys" in the left-hand menu.

Click on "Add role assignment" and select the "AcrPull" role.

In the "Select" field, search for the name of the managed identity used by the AKS cluster.

Click on "Save" to grant the role assignment. After granting the AcrPull role to the managed identity used by the AKS cluster, try pulling the image again and see if the issue is resolved.

I hope this helps! Let me know if you have any further questions.

Share via

Private ACR with AKS: Failed to pull image... failed to resolve reference... unexpected status from HEAD request... 403 Forbidden

Your answer