How to update custom container register CA certificate in Aks node

Question

How to update custom container register CA certificate in Aks node

Haroon Rasheed 0

Im trying to pull the image from self hosted gitlab container registry. But it shows below error.

Events:

Type Reason Age From Message

Normal Scheduled 72s default-scheduler Successfully assigned default/python-microservice-78778df5d8-dwtzm to aks-agentpool-14210116-vmss000001

Normal Pulling 29s (x3 over 72s) kubelet Pulling image "dns.url/ecom/services/python-micro-services-ecom-v2:latest"

Warning Failed 28s (x3 over 71s) kubelet Failed to pull image "dns.url/ecom/services/python-micro-services-ecom-v2:latest": failed to pull and unpack image "dns.url/ecom/services/python-micro-services-ecom-v2:latest": failed to resolve reference "dns.url/ecom/services/python-micro-services-ecom-v2:latest": failed to do request: Head "https://dns.url/v2/ecom/services/python-micro-services-ecom-v2/manifests/latest": tls: failed to verify certificate: x509: certificate signed by unknown authority

Warning Failed 28s (x3 over 71s) kubelet Error: ErrImagePull

Normal BackOff 2s (x4 over 70s) kubelet Back-off pulling image "dns.url/ecom/services/python-micro-services-ecom-v2:latest"

Warning Failed 2s (x4 over 70s) kubelet Error: ImagePullBackOff

Dharani Reguri 990 Reputation points Microsoft External Staff Moderator

2025-04-23T15:23:42.88+00:00

Hi Haroon Rasheed,

I understand that when you are trying to pull an image from GitLab container registry, it was failed with TLS error.

Can you please try by running a CURL command inside any of the AKS nodes to check the TLS handshake and please share the result of it.

In order to resolve this problem, we can use a Kubernetes Daemon Set to make sure that every node in the cluster can have access to the Container Registry with the Self-Signed certificate or we can also use the AKS "Custom certificate authority (CA) in Azure Kubernetes Service (AKS).

Use the Custom certificate authority (CA) in Azure Kubernetes Service (AKS) feature, this will add the CUSTOM CA to the node's trust store. This can be implemented even if the cluster was already created. Use the Custom certificate authority (CA)

Please refer to the document about pull images using pull secret.

https://learn.microsoft.com/en-us/azure/container-registry/container-registry-auth-kubernetes

Please let me know if you have any further questions.

Thank you.
Haroon Rasheed 1 Reputation point

2025-04-23T20:20:50.8633333+00:00

When I try to login using docker login DNS.url it's logged in and succeeded. But pulling image from container registry it's show TLS. And I tried adding CA custom certs to AKS cluster it's also not working shows error as --enable-custom-ca-cert command not found.

Haroon Rasheed 0

@Dharani Reguri

When i try to docker login to CR i get below error from cloud shell CLI

INFO[0043] Error logging in to endpoint, trying next endpoint error="Get "https://fc-svm-cr.family.qa/v2/": tls: failed to verify certificate: x509: certificate signed by unknown authority"

Get "https://fc-svm-cr.family.qa/v2/": tls: failed to verify certificate: x509: certificate signed by unknown authority

Please find the curl results

curl fc-svm-cr.family.qa
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:03 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:04 --:--:--     0
100   162  100   162    0     0     31      0  0:00:05  0:00:05 --:--:--    34
<html>
<head><title>301 Moved Permanently</title></head>
<body>
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx</center>
</body>
</html>

Dharani Reguri 990 Reputation points Microsoft External Staff Moderator

2025-04-28T10:08:27.88+00:00
Hi Haroon Rasheed,

Thanks for sharing the details.

I would suggest by creating an image pull secret which store information required for authentication to your registry. To create the pull secret for your container registry, you need to provide the service principal ID, password, and the registry URL. Please use the below kubectl command for creating Image pull secret.

kubectl create secret docker-registry <secret-name> \ --namespace <namespace> \ --docker-server=<container-registry-name> \ --docker-username=<service-principal-ID> \ --docker-password=<service-principal-password>

Once you have created the image pull secret, you can use it to create Kubernetes pods and deployments. Provide the name of the secret under imagePullSecrets in the deployment file.

Please refer to the document below about create an image pullsecret

Additionally, could you please confirm whether you're attempting to run the docker login to the container registry from your local machine or from the node itself?

Please let me know if you have any further questions.

1 answer

Your answer

Dharani Reguri 990 Reputation points Microsoft External Staff Moderator

2025-04-23T15:23:42.88+00:00

Hi Haroon Rasheed,

I understand that when you are trying to pull an image from GitLab container registry, it was failed with TLS error.

Can you please try by running a CURL command inside any of the AKS nodes to check the TLS handshake and please share the result of it.

In order to resolve this problem, we can use a Kubernetes Daemon Set to make sure that every node in the cluster can have access to the Container Registry with the Self-Signed certificate or we can also use the AKS "Custom certificate authority (CA) in Azure Kubernetes Service (AKS).

Use the Custom certificate authority (CA) in Azure Kubernetes Service (AKS) feature, this will add the CUSTOM CA to the node's trust store. This can be implemented even if the cluster was already created. Use the Custom certificate authority (CA)

Please refer to the document about pull images using pull secret.

https://learn.microsoft.com/en-us/azure/container-registry/container-registry-auth-kubernetes

Please let me know if you have any further questions.

Thank you.
Haroon Rasheed 1 Reputation point

2025-04-23T20:20:50.8633333+00:00

When I try to login using docker login DNS.url it's logged in and succeeded. But pulling image from container registry it's show TLS. And I tried adding CA custom certs to AKS cluster it's also not working shows error as --enable-custom-ca-cert command not found.
Haroon Rasheed 0 Reputation points

2025-04-27T08:58:44.7733333+00:00

@Dharani Reguri

When i try to docker login to CR i get below error from cloud shell CLI

INFO[0043] Error logging in to endpoint, trying next endpoint error="Get "https://fc-svm-cr.family.qa/v2/": tls: failed to verify certificate: x509: certificate signed by unknown authority"

Get "https://fc-svm-cr.family.qa/v2/": tls: failed to verify certificate: x509: certificate signed by unknown authority

Please find the curl results

curl fc-svm-cr.family.qa % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:02 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:03 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:04 --:--:-- 0 100 162 100 162 0 0 31 0 0:00:05 0:00:05 --:--:-- 34 <html> <head><title>301 Moved Permanently</title></head> <body> <center><h1>301 Moved Permanently</h1></center> <hr><center>nginx</center> </body> </html>
Dharani Reguri 990 Reputation points Microsoft External Staff Moderator

2025-04-28T10:08:27.88+00:00

Hi Haroon Rasheed,

Thanks for sharing the details.

I would suggest by creating an image pull secret which store information required for authentication to your registry. To create the pull secret for your container registry, you need to provide the service principal ID, password, and the registry URL. Please use the below kubectl command for creating Image pull secret.

kubectl create secret docker-registry <secret-name> \ --namespace <namespace> \ --docker-server=<container-registry-name> \ --docker-username=<service-principal-ID> \ --docker-password=<service-principal-password>

Once you have created the image pull secret, you can use it to create Kubernetes pods and deployments. Provide the name of the secret under imagePullSecrets in the deployment file.

Please refer to the document below about create an image pullsecret

Additionally, could you please confirm whether you're attempting to run the docker login to the container registry from your local machine or from the node itself?

Please let me know if you have any further questions.

Answer 1

Im trying to access from node.

The issue was resolved using the following steps:

Encoded the CA certificate into Base64 format.

Exported the cluster configuration in JSON format.

az rest --method get --url "/subscriptions/<subscription-id>/resourceGroups/<resource-grou-name>/providers/Microsoft.ContainerService/managedClusters/<cluster-name>?api-version=2025-01-01" > body.json

Update Base64 values to the below porfile.

"securityProfile": {

"customCaTrustCertificates": ["values"]

Upload the update body json to cluster and it will take sometime to update the certificate.

az rest --method put --url "/subscriptions/<subscription-id>/resourceGroups/<resource-group-name>/providers/Microsoft.ContainerService/managedClusters/<cluster-name>?api-version=2025-01-01" --body @body.json

@Dharani Reguri , Thanks for the support and document "Use the Custom certificate authority (CA)"

Share via

How to update custom container register CA certificate in Aks node

1 answer

Your answer