Network issue on specific AKS node

Question

Network issue on specific AKS node

Alex Rankin 50

Hello,

A specific instance in my D4ads_v7 node pool (of 7 nodes) started experiencing outbound connection issues at about 5AM EST this weekend. This issue continued until I drained the node and a new one was spun up. No other nodes experienced this issue. Research led me to checking for SNAT port exhaustion, but according to the metrics, this was not an issue.

No changes were made by us to the NSG or LB for this VMSS.

The outbound node problem detector attached this event:

No client cert or token found; attempting anonymous healthz check. Required endpoints are unreachable (curl: (28) Connection timed out after 5001 milliseconds: https://management.azure.com ;curl: (28) Connection timed out after 5002 milliseconds: https://packages.microsoft.com ;curl: (28) Connection timed out after 5001 milliseconds: https://acs-mirror.azureedge.net/acs-mirror/healthz ;curl: (28) Connection timed out after 5002 milliseconds: https://d1-prod-cus-hl6vuoat.hcp.centralus.azmk8s.io/healthz ), aka.ms/AArpzy5 for more information.

Thank you in advance.

Himanshu Shekhar 6,335 Reputation points Microsoft External Staff Moderator

2026-06-01T00:48:20.2133333+00:00
Hey Alex, it looks like one of your D4ads_v7 nodes suddenly lost all outbound connectivity to Azure endpoints (management.azure.com, packages.microsoft.com, acs-mirror.azureedge.net, etc.) even though SNAT-port metrics weren’t maxed out and you hadn’t changed any NSGs or load-balancer rules. Draining and replacing the node resolved the issue, which tells me this was likely an intermittent host-level or Azure-platform network glitch rather than a Kubernetes/SNAT configuration problem.

Here’s what you can try and look at next time this happens:

Validate connectivity from inside the node – SSH to the problematic VMSS instance and run: • curl –v https://management.azure.com • curl –v https://packages.microsoft.com • telnet acs-mirror.azureedge.net 443 – This will confirm whether the failure is truly network-level.

Check SNAT and socket stats locally – netstat -an | grep TIME_WAIT | wc -l – iptables -t nat -L -v – dmesg | grep -i “conntrack” or “iptables” Even if Azure Monitor shows no exhaustion, local wire-level counters can give extra clues.

Review Azure Service Health and VMSS health events – Look for any platform maintenance or networking-infrastructure incidents around 5AM EST. – In the Azure Portal, under “Service Health” and under the VM scale set’s “Diagnose and solve problems.”

Ensure you have an eviction/auto-heal policy enabled on your node pool Azure AKS can automatically replace unhealthy instances. That way, if the host networking flakes out again, it’ll self-recover without manual intervention.

If this resurfaces regularly, consider migrating to Azure CNI overlay (which bypasses SNAT) or file a deeper support case so the Azure Fabric team can grab host-level logs.

Follow-up questions to pin this down:

Which network plugin are you using on this cluster (Azure CNI, Azure CNI Overlay or Kubenet)?

Do you have any custom NSGs, UDRs or Azure Firewall rules applied to the node subnet?

Can you share the SNAT-related metrics around the incident window (e.g. “SNATConnectionsFailed”)?

Were there any Azure Service Health alerts or VMSS “health” events logged around 5AM EST?

If you SSH into the replacement node, do you see the same curl/telnet failures?

References

• Troubleshoot an AKS node that changes to Not Ready status (steps for network-level checks & SNAT):

https://learn.microsoft.com/troubleshoot/azure/azure-kubernetes/availability-performance/node-not-ready-after-being-healthy#solution

• AKS outbound requirements (endpoints you must reach from nodes):

https://learn.microsoft.com/azure/aks/outbound-rules-control-egress

• AKS Network Troubleshooting Methodology:

https://supportability.visualstudio.com/AzureContainers/_wiki/wikis/Containers%20Wiki/1280910/Azure%20Kubernetes%20Service%20Wiki/AKS/TSG/AKS%20Network%20Troubleshooting%20Methodology

• [TSG] AKS & Network team common troubleshooting:

https://supportability.visualstudio.com/AzureContainers/_wiki/wikis/Containers%20Wiki/1280927/Azure%20Kubernetes%20Service%20Wiki/AKS/TSG/AKS%20Network%20Troubleshooting%20Methodology/[TSG]%20Aks%20and%20Network%20team%20common%20troubleshooting
Alex Rankin 50 Reputation points

2026-06-01T13:32:56.59+00:00
@Himanshu Shekhar , thank you for reaching out! To answer your questions:

I am using Azure CNI Overlay.

No

I am unable to find this metric. I don't see it under the outbound loadbalancer.

No

No

Additionally, I saw this issue occur on another instance overnight. Checking the node, the only problem events logged were from some of my workloads that were trying to access services on the internet. No system workloads tagged anything (including node problem detector). The node was still in a ready state. There were no dmsg kernel errors logged. After moving my workloads around, the issue was fixed. My guess is there is some kind of routing corruption that is occurring, and after moving workloads around, the bad state is replaced.

The only changes made recently were a K8s version update and upgrading the VM SKU to D4ads_v7 from D2as_v6. How can I get this problem escalated?

Thank you.
Alex Rankin 50 Reputation points

2026-06-01T14:57:19.7833333+00:00

We have updated our version of AKS from 1.35.4 to 1.36.0, and we will continue to monitor for issues.