AKS not able to resolve azure devops, dial tcp: lookup dev.azure.com on DNSIP:53: server misbehaving
I am running a AKS cluster and Microsoft Azure is not able resolve its own service(dev.azure.com) in its backbone.
I am not using any costume DNS of my own use AKS provided once only.
To isolate the issue i can a container with nslookup and found once in 10 time it will fail to resolve dev.azure.com and this make it NOT THE Best choice for VCS (central repo server with code that need to be in available in HA mode)
Below is output :
root@tmp-shell:/# nslookup dev.azure.com
Server: 10.0.0.10
Address: 10.0.0.10#53
Name: dev.azure.com
Address: 13.107.42.20
root@tmp-shell:/# nslookup dev.azure.com
Server: 10.0.0.10
Address: 10.0.0.10#53
** server can't find dev.azure.com: SERVFAIL
root@tmp-shell:/# nslookup dev.azure.com
Server: 10.0.0.10
Address: 10.0.0.10#53
** server can't find dev.azure.com.gsoaylwixxxxxnhrmmd3k1rfnb.frax.internal.cloudapp.net: SERVFAIL
dial tcp: lookup dev.azure.com on 10.0.0.10:53: server misbehaving"
dial tcp: lookup dev.azure.com on DNS_IP:53: server misbehaving"
Error message from logs
error resolving repo revision: rpc error: code = Unknown desc = Get "https://mycorp@dev.azure.com/mycorp/manifests/_git/manifests/info/refs?service=git-upload-pack": dial tcp: lookup dev.azure.com on 10.0.0.10:53: server misbehaving"
Kubernetes version:1.26.6
Type (plugin): Azure CNI
DNS service IP: 10.0.0.10
Network policy: None
Error log is same be it private or public cluster.
kubectl logs -f -l k8s-app=kube-dns -n kube-system
[ERROR] plugin/errors: 2 dev.azure.com. AAAA: read udp 10.244.0.16:35510->10.215.4.10:53: i/o timeout
[ERROR] plugin/errors: 2 dev.azure.com. A: read udp 10.244.0.16:41869->10.215.4.12:53: i/o timeout
[ERROR] plugin/errors: 2 dev.azure.com. A: read udp 10.244.0.16:52191->10.215.4.10:53: i/o timeout
[INFO] 10.244.0.19:58400 - 31861 "A IN dev.azure.com.3hrulajbi5nxxxxxxxnan5g.ax.internal.cloudapp.net. udp 94 false 1232" - - 0 2.000599712s
If you would ask what is status of core-dns, see blow
coredns-76b9877f49-cm2sv 1/1 Running 0
coredns-76b9877f49-d2bjg 1/1 Running 0
coredns-autoscaler-59b9d8bbb4-p5kp4 1/1 Running 0
coredns config
kubectl describe cm coredns -n kube-system
Name: coredns
Namespace: kube-system
Labels: addonmanager.kubernetes.io/mode=Reconcile
k8s-app=kube-dns
kubernetes.io/cluster-service=true
Annotations: