Kubernetes DNS Resolution Failure - Intermittent Connectivity Issue to 168.63.129.16 (Working Previously)

Question

Kubernetes DNS Resolution Failure - Intermittent Connectivity Issue to 168.63.129.16 (Working Previously)

Severin Hersche 5

Hello,

Our Kubernetes cluster is experiencing DNS resolution failures. CoreDNS logs show "i/o timeout" errors when attempting to reach 168.63.129.16:53.

This issue began approximately two days ago. Prior to this, DNS resolution was functioning correctly.

Interestingly, we have found that restarting the Kubernetes cluster temporarily resolves the issue, but it reappears after a period of time. This suggests a potential underlying network state issue or a service that degrades over time.

We have not configured any custom firewalls or DNS settings, with the exception of an external DNS record forwarding specificURL.com to the cluster.

Thank you for your help.

Markapuram Sudheer Reddy 2,050 Reputation points Microsoft External Staff Moderator

2025-03-12T00:08:28.3566667+00:00
Hi Severin Hersche,

Thank You for reaching out to Microsoft Q&A forum.

Can you please confirm where does the DNS resolution fail? Pod, Node or Both pods and nodes.

and also, can you please share CoreDNS logging error messages.

Verify once if the CoreDNS pods are running:

kubectl get pods -n kube-system

Check once whether the CoreDNS pods are overused:

kubectl top pods -n kube-system

If the information is helpful, please click on "Upvote"

If you have any queries, please do let us know by sharing in detail errors/logs, we will help you.
Severin Hersche 5 Reputation points

2025-03-12T07:22:55.52+00:00

Hi @Markapuram Sudheer Reddy

Thank you for your comment. As far as I understand it it is failing on both levels pod and node. Two coredns pods are up and runnung. But I had a restart on the coredns-autoscaler pod - not sure if this is relevant. There are a lot of errors please find one example:

[ERROR] plugin/errors: 2 global.handler.control.monitor.azure.com. A: read udp 10.244.1.78:46160->168.63.129.16:53: i/o timeout

[ERROR] plugin/errors: 2 global.handler.control.monitor.azure.com. A: read udp 10.244.1.78:54988->168.63.129.16:53: i/o timeout

This is the output of the kubectl top pods:

NAME CPU(cores) MEMORY(bytes)

coredns- 2m 38Mi
Severin Hersche 5 Reputation points

2025-03-12T07:29:10.53+00:00

There is this error in the autoscaler pod logs - not sure if this is relevant:

E0310 08:14:37.627767 1 autoscaler_server.go:108] Error while getting cluster status: context deadline exceeded

E0310 08:14:47.627960 1 autoscaler_server.go:108] Error while getting cluster status: context deadline exceeded
Arko 4,150 Reputation points Microsoft External Staff Moderator

2025-04-01T13:46:16.9233333+00:00

Hello Severin Hersche, Looks like a networking degradation issue at the node level, affecting DNS resolution to Azure's internal IP 168.63.129.16. Restarting nodes temporarily helps because it clears these states. Check conntrack table usage, ensure your nodes have sufficient resources, and consider upgrading your CNI/kube-proxy versions. Restarting the azure-cni-networkmonitor and kube-proxy DaemonSets may also help stabilize networking. Try these and let me know how it goes.
Arko 4,150 Reputation points Microsoft External Staff Moderator

2025-04-02T05:55:58.0766667+00:00

I feel you're experiencing DNS resolution failures to 168.63.129.16, which is a critical Azure internal IP used for DNS relay, health probes, and other platform services. When CoreDNS logs show i/o timeout errors toward this IP, it usually means AKS node networking cannot reach Azure's internal services, breaking DNS resolution for certain internal or Azure-hosted domains.

Given that the issue started recently (was working before), and restarting the cluster temporarily resolves it and you're not using custom NSGs or UDRs and the failure affects both pods and nodes then this strongly points to a degraded or corrupted VNet subnet state in Azure’s backend SDN fabric, especially impacting routing to 168.63.129.16.

Provisioning a new subnet or new virtual network and deploying the AKS cluster into it should resolves the issue. Have you tried that? If not, then please create a new subnet (or even better, a new VNet) in the same region. Deploy a new AKS cluster or a new node pool using this fresh subnet.

Test DNS from a pod using nslookup and ping 168.63.129.16 — DNS resolution should now work consistently.

This avoids the degraded SDN state that can silently impact existing subnets.

If deploying into a new subnet/VNet still doesn't resolve the issue, let me know. I will raise a request with the backend fabric team to check on why the connectivity to 168.63.129.16 fails even in a clean deployment.

Additionally, you can checkout https://learn.microsoft.com/en-us/troubleshoot/azure/azure-kubernetes/connectivity/troubleshoot-dns-failures-across-an-aks-cluster-in-real-time to troubleshoot DNS failures across an AKS cluster in real time.
Severin Hersche 5 Reputation points

2025-04-02T07:36:42.4233333+00:00

Dear Arko Thank you for your message and help. I redeployed it to different clusters with different VM's. Unfortunately this is not solving the problem.
Arko 4,150 Reputation points Microsoft External Staff Moderator

2025-04-03T10:58:59.8566667+00:00
Hi Severin, thank you for testing and confirming. Since redeploying to different clusters and VM SKUs didn’t resolve the issue and given that the DNS failures are specifically targeting Azure’s internal IP 168.63.129.16, could you kindly check conntrack usage

cat /proc/sys/net/netfilter/nf_conntrack_count cat /proc/sys/net/netfilter/nf_conntrack_max

If you find it is nearing the max value, increase it

sudo sysctl -w net.netfilter.nf_conntrack_max=131072

Also, look for dropped packets or errors using netstat -s or ip -s link

Also try restarting the kube-proxy and azure-cni-networkmonitor, as they sometimes get into a degraded state.

If that too does not resolve the issue, consider setting up a DNS forwarder (like Azure DNS Proxy) to temporarily bypass reliance on 168.63.129.16 directly. This would be the final check before we can request a backend SDN health check, specifically regarding routing to 168.63.129.16 to the MS product team on behalf of you. Thanks
Arko 4,150 Reputation points Microsoft External Staff Moderator

2025-04-04T05:37:43.8366667+00:00

Following up on this. Hope your query is resolved now.
Severin Hersche 5 Reputation points

2025-04-06T18:44:31.0133333+00:00

Hi Arko

Thank you for your message. I will check in a few days I hope it is resolved.
Arko 4,150 Reputation points Microsoft External Staff Moderator

2025-04-07T04:21:07.0233333+00:00

Sure. Ping me here if anything else required. Thanks
Aleksander Miliukov 0 Reputation points

2025-04-22T11:57:23.3133333+00:00

We are experiencing a similar issue reaching 168.63.129.16:53, and additionally, the same issue occurs when trying to reach the kube-apiserver private endpoint, which is exposed to the private network.

1 answer

Your answer

Markapuram Sudheer Reddy 2,050 Reputation points Microsoft External Staff Moderator

2025-03-12T00:08:28.3566667+00:00

Hi Severin Hersche,

Thank You for reaching out to Microsoft Q&A forum.

Can you please confirm where does the DNS resolution fail? Pod, Node or Both pods and nodes.

and also, can you please share CoreDNS logging error messages.

Verify once if the CoreDNS pods are running:

kubectl get pods -n kube-system

Check once whether the CoreDNS pods are overused:

kubectl top pods -n kube-system

If the information is helpful, please click on "Upvote"

If you have any queries, please do let us know by sharing in detail errors/logs, we will help you.
Severin Hersche 5 Reputation points

2025-03-12T07:22:55.52+00:00

Hi @Markapuram Sudheer Reddy

Thank you for your comment. As far as I understand it it is failing on both levels pod and node. Two coredns pods are up and runnung. But I had a restart on the coredns-autoscaler pod - not sure if this is relevant. There are a lot of errors please find one example:

[ERROR] plugin/errors: 2 global.handler.control.monitor.azure.com. A: read udp 10.244.1.78:46160->168.63.129.16:53: i/o timeout

[ERROR] plugin/errors: 2 global.handler.control.monitor.azure.com. A: read udp 10.244.1.78:54988->168.63.129.16:53: i/o timeout

This is the output of the kubectl top pods:

NAME CPU(cores) MEMORY(bytes)

coredns- 2m 38Mi
Severin Hersche 5 Reputation points

2025-03-12T07:29:10.53+00:00

There is this error in the autoscaler pod logs - not sure if this is relevant:

E0310 08:14:37.627767 1 autoscaler_server.go:108] Error while getting cluster status: context deadline exceeded

E0310 08:14:47.627960 1 autoscaler_server.go:108] Error while getting cluster status: context deadline exceeded
Arko 4,150 Reputation points Microsoft External Staff Moderator

2025-04-01T13:46:16.9233333+00:00

Hello Severin Hersche, Looks like a networking degradation issue at the node level, affecting DNS resolution to Azure's internal IP 168.63.129.16. Restarting nodes temporarily helps because it clears these states. Check conntrack table usage, ensure your nodes have sufficient resources, and consider upgrading your CNI/kube-proxy versions. Restarting the azure-cni-networkmonitor and kube-proxy DaemonSets may also help stabilize networking. Try these and let me know how it goes.
Arko 4,150 Reputation points Microsoft External Staff Moderator

2025-04-02T05:55:58.0766667+00:00

I feel you're experiencing DNS resolution failures to 168.63.129.16, which is a critical Azure internal IP used for DNS relay, health probes, and other platform services. When CoreDNS logs show i/o timeout errors toward this IP, it usually means AKS node networking cannot reach Azure's internal services, breaking DNS resolution for certain internal or Azure-hosted domains.

Given that the issue started recently (was working before), and restarting the cluster temporarily resolves it and you're not using custom NSGs or UDRs and the failure affects both pods and nodes then this strongly points to a degraded or corrupted VNet subnet state in Azure’s backend SDN fabric, especially impacting routing to 168.63.129.16.

Provisioning a new subnet or new virtual network and deploying the AKS cluster into it should resolves the issue. Have you tried that? If not, then please create a new subnet (or even better, a new VNet) in the same region. Deploy a new AKS cluster or a new node pool using this fresh subnet.

Test DNS from a pod using nslookup and ping 168.63.129.16 — DNS resolution should now work consistently.

This avoids the degraded SDN state that can silently impact existing subnets.

If deploying into a new subnet/VNet still doesn't resolve the issue, let me know. I will raise a request with the backend fabric team to check on why the connectivity to 168.63.129.16 fails even in a clean deployment.

Additionally, you can checkout https://learn.microsoft.com/en-us/troubleshoot/azure/azure-kubernetes/connectivity/troubleshoot-dns-failures-across-an-aks-cluster-in-real-time to troubleshoot DNS failures across an AKS cluster in real time.
Severin Hersche 5 Reputation points

2025-04-02T07:36:42.4233333+00:00

Dear Arko Thank you for your message and help. I redeployed it to different clusters with different VM's. Unfortunately this is not solving the problem.
Arko 4,150 Reputation points Microsoft External Staff Moderator

2025-04-03T10:58:59.8566667+00:00

Hi Severin, thank you for testing and confirming. Since redeploying to different clusters and VM SKUs didn’t resolve the issue and given that the DNS failures are specifically targeting Azure’s internal IP 168.63.129.16, could you kindly check conntrack usage

cat /proc/sys/net/netfilter/nf_conntrack_count cat /proc/sys/net/netfilter/nf_conntrack_max

If you find it is nearing the max value, increase it

sudo sysctl -w net.netfilter.nf_conntrack_max=131072

Also, look for dropped packets or errors using netstat -s or ip -s link

Also try restarting the kube-proxy and azure-cni-networkmonitor, as they sometimes get into a degraded state.

If that too does not resolve the issue, consider setting up a DNS forwarder (like Azure DNS Proxy) to temporarily bypass reliance on 168.63.129.16 directly. This would be the final check before we can request a backend SDN health check, specifically regarding routing to 168.63.129.16 to the MS product team on behalf of you. Thanks
Arko 4,150 Reputation points Microsoft External Staff Moderator

2025-04-04T05:37:43.8366667+00:00

Following up on this. Hope your query is resolved now.
Severin Hersche 5 Reputation points

2025-04-06T18:44:31.0133333+00:00

Hi Arko

Thank you for your message. I will check in a few days I hope it is resolved.
Arko 4,150 Reputation points Microsoft External Staff Moderator

2025-04-07T04:21:07.0233333+00:00

Sure. Ping me here if anything else required. Thanks
Aleksander Miliukov 0 Reputation points

2025-04-22T11:57:23.3133333+00:00

We are experiencing a similar issue reaching 168.63.129.16:53, and additionally, the same issue occurs when trying to reach the kube-apiserver private endpoint, which is exposed to the private network.

Answer 1

Markapuram Sudheer Reddy 2,050 Microsoft External Staff Moderator

Hi Severin Hersche,

Thank You for the response.

Try to drain and reboot the affected worker nodes or regenerating the nodes.

If you are using Linux distribution-based worker nodes, try to check the iptables and update accordingly then try to reboot. sudo iptables --version

https://github.com/coredns/coredns/issues/2693#issuecomment-543208513

Check once if setting in this file /etc/sysctl conf and try to switch the setting to net.ipv4_forward = 1 and reboot the node.

Please try to check with the workarounds followed here as it may help to troubleshoot your query. https://github.com/Azure/AKS/issues/1320

Please check with below workarounds in the documentation to help in troubleshooting the query.

https://github.com/kubernetes/kubernetes/issues/86762

https://github.com/kubernetes/kubernetes/issues/113080

If the information is helpful, please click on "Upvote"

If you have any queries, please do let us know, we will help you.

Markapuram Sudheer Reddy 2,050 Reputation points Microsoft External Staff Moderator

2025-03-25T05:32:35.93+00:00

Hi Severin Hersche,

Try to delete core dns pods or try restarting those pods can sometimes resolve transient issues https://github.com/coredns/coredns/issues/5443

and also ensure that your CoreDNS pods have adequate resource limits and review your network policies which might block any DNS traffic.

https://learn.microsoft.com/en-us/troubleshoot/azure/azure-kubernetes/connectivity/troubleshoot-dns-failure-from-pod-but-not-from-worker-node

If you have any queries, please do let us know, we will help you.

Severin Hersche 5

Thank you Markapuram Sudheer Reddy for your answer. Deleteing the core dns pod did not help. For some reason the yarp I installed is also causing problems after some time. Is this connected to my other problem? I repoloyed everything to a new cluster with more memory.


Yarp.ReverseProxy.Forwarder.HttpForwarder[48]       Request: An error was encountered before receiving a response.       System.Net.Http.HttpRequestException: Name or service not known

I'm not able to start this command eighter:

kubectl debug node/aks-agentpool-29134186-vmss000000 -it --image=mcr.microsoft.com/cbl-mariner/busybox:2.0
Creating debugging pod node-debugger-aks-agentpool-29134186-vmss000000-kwbhd with container debugger on node aks-agentpool-29134186-vmss000000.
Warning: container debugger: rpc error: code = DeadlineExceeded desc = failed to pull and unpack image "mcr.microsoft.com/cbl-mariner/busybox:2.0": failed to copy: httpReadSeeker: failed open: failed to do request: Get "https://mcr.microsoft.com/v2/cbl-mariner/busybox/manifests/sha256:e4fb4d51fc9b70d6cdc1ce66a0af02ab40554d2ca632e1d188fabc760e432fdd": dial tcp 150.171.70.10:443: i/o timeout

Share via

Kubernetes DNS Resolution Failure - Intermittent Connectivity Issue to 168.63.129.16 (Working Previously)

1 answer

Your answer