AKS cannot pull image from GitHub Private registry (image not found) but Minikube could pull

Question

AKS cannot pull image from GitHub Private registry (image not found) but Minikube could pull

Kyle 96

Hi All,

I am trying to build an application, but when I try to create a Deployment, the container failed at creation stage with the error:

     "docker.pkg.github.com/XXXXX/XXXXXX/XXXXXXXXXXXX:latest": rpc error: code = NotFound desc = failed to pull and unpack image "docker.pkg.github.com/XXXXX/XXXXXXX/XXXXXXXXXX:latest": failed to copy: httpReaderSeeker: failed open: content at https://docker.pkg.github.com/v2/XXXXXXX/XXXXXX/XXXXXXX/manifests/sha........ not found: not found

And I tried with kubectl both on Cloudshell and on my local machine, same issue:

Then I tried minikube on my local machine, and deploy using the YAML, and the deployment created without any issue. Also, docker pull is also working. I guess then it should not be the github credential issue.

I really cannot figure out the reason for kubernetes behave so differently on azure and on minikube. Would appreciate any help on how to troubleshooting and solving this issue.

Just FYI, I use below shell cmd to create the resource on Azure, wondering if I miss any steps:

az group create \
    --name $RESOURCE_GROUP \
    --location $REGION_NAME

az network vnet create \
    --resource-group $RESOURCE_GROUP \
    --location $REGION_NAME \
    --name $VNET_NAME \
    --address-prefixes 10.0.0.0/8 \
    --subnet-name $SUBNET_NAME \
    --subnet-prefixes 10.240.0.0/16

SUBNET_ID=$(az network vnet subnet show \
    --resource-group $RESOURCE_GROUP \
    --vnet-name $VNET_NAME \
    --name $SUBNET_NAME \
    --query id -o tsv)

VERSION=$(az aks get-versions \
    --location $REGION_NAME \
    --query 'orchestrators[?!isPreview] | [-1].orchestratorVersion' \
    --output tsv)

az aks create \
    --resource-group $RESOURCE_GROUP \
    --name $AKS_CLUSTER_NAME \
    --vm-set-type VirtualMachineScaleSets \
    --node-count 1 \
    --load-balancer-sku standard \
    --location $REGION_NAME \
    --kubernetes-version $VERSION \
    --network-plugin azure \
    --vnet-subnet-id $SUBNET_ID \
    --service-cidr 10.2.0.0/24 \
    --dns-service-ip 10.2.0.10 \
    --docker-bridge-address 172.17.0.1/16 \
    --generate-ssh-keys   

echo 'xxxxxxx' | docker login https://docker.pkg.github.com -u xxxxxxx --password-stdin

SRIJIT-BOSE-MSFT 4,346 Reputation points Microsoft Employee

2021-06-21T06:55:51.48+00:00

anonymous user-6160 , Thank you for your question.

Did you run docker pull from your local machine when it succeeded? Were you logged into the private registry on the docker client?

Per https://learn.microsoft.com/en-us/azure/aks/cluster-container-registry-integration, if you need to pull an image from a private external registry, use an image pull secret.

Can you please try to deploy your application on the AKS cluster with the image pull secret and check if that helps?
Kyle 96 Reputation points

2021-06-21T07:05:34.133+00:00
@SRIJIT-BOSE-MSFT thanks for your response,

Did you run docker pull from your local machine when it succeeded?
- yes, though I get a warning like the below

WARNING: ⚠️ Failed to pull manifest by the resolved digest. This registry does not appear to conform to the distribution registry specification; falling back to pull by tag. This fallback is DEPRECATED, and will be removed in a future release. Please contact admins of https://docker.pkg.github.com. ⚠️

Were you logged into the private registry on the docker client?

Yes again, using the cmd: echo 'XXXXXXXXXXXXX' | docker login https://docker.pkg.github.com -u <my-username> --password-stdin

if you need to pull an image from a private external registry, use an image pull secret.

I have already specified imagePullSecrets: - name: github-credential

under the spec attribute
SRIJIT-BOSE-MSFT 4,346 Reputation points Microsoft Employee

2021-06-21T09:41:07.877+00:00

anonymous user-6160 , Can you please check the docker runtime version on the AKS nodes using kubectl get no -o wide and also share the version of kubernetes your AKS cluster is currently running?

Also can you please share the output of docker -v from the local machine where the pull has succeeded?
Kyle 96 Reputation points

2021-06-21T15:52:11.563+00:00

Hi @SRIJIT-BOSE-MSFT , I finally figure out the issue. The error was caused by the reason that I was using latest version 1.20 of AKS which has deprecated docker as the container runtime. After switching to the lower version 1.18 of AKS, the problem solved. Thanks.

Accepted answer

0 additional answers

Your answer

SRIJIT-BOSE-MSFT 4,346 Reputation points Microsoft Employee

2021-06-21T06:55:51.48+00:00

anonymous user-6160 , Thank you for your question.

Did you run docker pull from your local machine when it succeeded? Were you logged into the private registry on the docker client?

Per https://learn.microsoft.com/en-us/azure/aks/cluster-container-registry-integration, if you need to pull an image from a private external registry, use an image pull secret.

Can you please try to deploy your application on the AKS cluster with the image pull secret and check if that helps?
Kyle 96 Reputation points

2021-06-21T07:05:34.133+00:00

@SRIJIT-BOSE-MSFT thanks for your response,

Did you run docker pull from your local machine when it succeeded?
- yes, though I get a warning like the below

WARNING: ⚠️ Failed to pull manifest by the resolved digest. This registry does not appear to conform to the distribution registry specification; falling back to pull by tag. This fallback is DEPRECATED, and will be removed in a future release. Please contact admins of https://docker.pkg.github.com. ⚠️

Were you logged into the private registry on the docker client?

Yes again, using the cmd: echo 'XXXXXXXXXXXXX' | docker login https://docker.pkg.github.com -u <my-username> --password-stdin

if you need to pull an image from a private external registry, use an image pull secret.

I have already specified imagePullSecrets: - name: github-credential

under the spec attribute
SRIJIT-BOSE-MSFT 4,346 Reputation points Microsoft Employee

2021-06-21T09:41:07.877+00:00

anonymous user-6160 , Can you please check the docker runtime version on the AKS nodes using kubectl get no -o wide and also share the version of kubernetes your AKS cluster is currently running?

Also can you please share the output of docker -v from the local machine where the pull has succeeded?
Kyle 96 Reputation points

2021-06-21T15:52:11.563+00:00

Hi @SRIJIT-BOSE-MSFT , I finally figure out the issue. The error was caused by the reason that I was using latest version 1.20 of AKS which has deprecated docker as the container runtime. After switching to the lower version 1.18 of AKS, the problem solved. Thanks.

Answer 1

Kyle 96

This no longer an issue. Latest version of AKS on Azure has deprecated Docker as a container runtime (containerD instead). After downgrading the cluster to 1.18, the problem got solved.

Share via

AKS cannot pull image from GitHub Private registry (image not found) but Minikube could pull

0 additional answers

Your answer