AKS not able to resolve azure devops, dial tcp: lookup dev.azure.com on DNSIP:53: server misbehaving

Rajesh 6 Reputation points
2023-08-15T15:37:04.32+00:00

I am running a AKS cluster and Microsoft Azure is not able resolve its own service(dev.azure.com) in its backbone.

I am not using any costume DNS of my own use AKS provided once only.

To isolate the issue i can a container with nslookup and found once in 10 time it will fail to resolve dev.azure.com and this make it NOT THE Best choice for VCS (central repo server with code that need to be in available in HA mode)

Below is output :

root@tmp-shell:/# nslookup dev.azure.com
Server:         10.0.0.10
Address:        10.0.0.10#53
Name:   dev.azure.com
Address: 13.107.42.20


root@tmp-shell:/# nslookup dev.azure.com
Server:         10.0.0.10
Address:        10.0.0.10#53
** server can't find dev.azure.com: SERVFAIL


root@tmp-shell:/# nslookup dev.azure.com
Server:         10.0.0.10
Address:        10.0.0.10#53
** server can't find dev.azure.com.gsoaylwixxxxxnhrmmd3k1rfnb.frax.internal.cloudapp.net: SERVFAIL

dial tcp: lookup dev.azure.com on 10.0.0.10:53: server misbehaving"

dial tcp: lookup dev.azure.com on DNS_IP:53: server misbehaving"

Error message from logs

error resolving repo revision: rpc error: code = Unknown desc = Get "https://mycorp@dev.azure.com/mycorp/manifests/_git/manifests/info/refs?service=git-upload-pack": dial tcp: lookup dev.azure.com on 10.0.0.10:53: server misbehaving"

Kubernetes version:1.26.6

Type (plugin): Azure CNI

DNS service IP: 10.0.0.10

Network policy: None

Error log is same be it private or public cluster.

kubectl logs -f -l k8s-app=kube-dns -n kube-system

[ERROR] plugin/errors: 2 dev.azure.com. AAAA: read udp 10.244.0.16:35510->10.215.4.10:53: i/o timeout
[ERROR] plugin/errors: 2 dev.azure.com. A: read udp 10.244.0.16:41869->10.215.4.12:53: i/o timeout
[ERROR] plugin/errors: 2 dev.azure.com. A: read udp 10.244.0.16:52191->10.215.4.10:53: i/o timeout
[INFO] 10.244.0.19:58400 - 31861 "A IN dev.azure.com.3hrulajbi5nxxxxxxxnan5g.ax.internal.cloudapp.net. udp 94 false 1232" - - 0 2.000599712s

If you would ask what is status of core-dns, see blow

coredns-76b9877f49-cm2sv                    1/1     Running            0     
coredns-76b9877f49-d2bjg                    1/1     Running            0     
coredns-autoscaler-59b9d8bbb4-p5kp4         1/1     Running            0     

coredns config

kubectl describe cm coredns -n kube-system
Name:         coredns
Namespace:    kube-system
Labels:       addonmanager.kubernetes.io/mode=Reconcile
              k8s-app=kube-dns
              kubernetes.io/cluster-service=true
Annotations:  
Azure Kubernetes Service (AKS)
Azure Kubernetes Service (AKS)
An Azure service that provides serverless Kubernetes, an integrated continuous integration and continuous delivery experience, and enterprise-grade security and governance.
2,373 questions
{count} votes

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.