DNS Resolution Issue on AKS due to i/o timeout

Erothu, Sita 1

Hi Team,

Our AKS cluster throwing an intermittent DNS issue(DNS resolution failed)

20210915T201241.606Z [0x80e00783][network][error] dns(Main-Name-Service): tid(1311): Cannot resolve host name 'xxxx.fsis.usda.gov' after 17046 ms..

When I checked the coreDNS logs, I found the below errors.

[ERROR] plugin/errors: 2 xxxx.fsis.usda.gov. A: read udp 10.201.x.20:43262->10.201.x.148:53: i/o timeout
[ERROR] plugin/errors: 2 xxxx.fsis.usda.gov. A: read udp 10.201.x.20:53789->10.201.x.148:53: i/o timeout
[WARNING] No files matching import glob pattern: custom/.override
[WARNING] No files matching import glob pattern: custom/.server

We are struggling understand why this is happening intermittently and fixed automatically after a few mins.

Could anyone please help in fixing the issue.

SRIJIT-BOSE-MSFT 4,331 Reputation points Microsoft Employee

2021-11-22T05:20:47.797+00:00

@Erothu, Sita , thank you for your question.

Can you please SSH to one of the AKS cluster nodes and check if nslookup xxxx.fsis.usda.gov resolves?
Erothu, Sita 1 Reputation point

2021-11-22T14:24:35.907+00:00

Hi @SRIJIT-BOSE-MSFT

nslookup from nodes and test tcp-connection from gw pods are working fine.

We are getting this error intermittently but not all the times.

Please let me know if you want to check any other details

Thanks
Sitaram
SRIJIT-BOSE-MSFT 4,331 Reputation points Microsoft Employee

2021-11-22T16:42:42.187+00:00

@Erothu, Sita , thank you for your response.

Based on the intermittent nature of the errors, I believe the best course of action would be to open a Microsoft technical Support Request, since this will require more traces and logs for our engineers to troubleshoot further and identify/eliminate the cause or recommend a fix/workaround.

Also, once you get the issue fixed, request you to reply back here on the thread with the resolution steps for the benefit of the community.
Rajesh 1 Reputation point

2023-08-16T09:44:18.2666667+00:00

@Erothu, Sita you found fix yet mate?
Cristian Gatjens 716 Reputation points Microsoft Employee

2023-08-16T15:25:24.5333333+00:00

@Erothu, Sita are you using custom DNS servers or Azure DNS?

If custom DNS servers, you can use the coreDNS hosts plugin as follows:

https://learn.microsoft.com/en-us/azure/aks/coredns-custom#hosts-plugin

This is basically to rule out issues with your custom DNS servers, by implementing the hosts plugin the name resolution happens locally in the nodes and never hit the DNS servers, if you don't see any more timeouts then there is an issue with your custom DNS servers that need investigation from your side.