NAT Gateway TCP idle timeout causes silent connection resets

Question

We are running workloads on AKS that make outbound HTTPS connections to external endpoints. Following Microsoft's recommendation to migrate away from the deprecated default outbound access via Load Balancer, we configured a NAT Gateway for outbound connectivity.

We have been experiencing intermittent java.net.SocketException: Connection reset errors on outbound connections since then. After investigation, we identified the root cause: the NAT Gateway silently discards SNAT mappings after the TCP idle timeout expires (default 4 minutes), without sending TCP RST or FIN to either endpoint. When the application subsequently attempts to reuse a pooled connection whose SNAT mapping has been discarded, it receives a RST and the request fails.

We were able to confirm this by increasing the NAT Gateway idle timeout to 10 minutes, combined with application-level connection pool TTL configuration set to 8 minutes. This reduced the errors to zero.

We understand why Microsoft recommends keeping the idle timeout at 4 minutes (https://docs.azure.cn/en-us/nat-gateway/nat-gateway-resource#:~:text=When%20a%20connection,of%204%20minutes.) — longer timeouts can contribute to SNAT port exhaustion by keeping mappings alive for connections that are no longer in use.

However, this creates a conflict:

1. Reconnection is expensive. Many of our components maintain persistent or pooled HTTPS connections precisely to amortize the cost of TLS handshakes. Configuring very short idle timeouts forces frequent reconnections, increasing latency and CPU overhead — particularly under load.
2. Not all products expose the necessary configuration. Resolving this at the application level requires every component running in the cluster to expose a connection pool TTL or idle timeout parameter that can be set below the NAT Gateway threshold. In practice, some third-party or proprietary components do not expose such a parameter, making a per-product remediation strategy difficult.

Given the above, what is the recommended infrastructure-level approach for workloads that require persistent outbound connections?

Is there a NAT Gateway configuration or complementary Azure networking feature that would notify endpoints of connection termination rather than silently discarding state?

For components that do not expose application-level idle timeout configuration, is configuring OS-level TCP keepalive (net.ipv4.tcp_keepalive_time, etc.) at the node level via AKS LinuxOSConfig a supported and recommended mitigation?

Thanks!

Answer 1

Hello @Andrés

Thank you for reaching out to Microsoft Q&A.

Based on the behavior you've described, the observed connection resets are consistent with the documented design of Azure NAT Gateway. When a TCP connection remains idle longer than the configured idle timeout (4 minutes by default), the NAT Gateway removes the associated SNAT flow state. The connection is not proactively closed with a TCP FIN or RST packet at the time the idle timeout expires. As a result, applications that attempt to reuse an idle pooled connection may encounter connection reset errors once traffic resumes.

From an infrastructure perspective, there is currently no NAT Gateway feature that sends immediate connection termination notifications to both endpoints when idle flow state is removed.

For workloads that rely on persistent outbound HTTPS connections, Microsoft generally recommends one of the following approaches:

• Configure application-level connection pool idle timeouts or connection lifetimes to be shorter than the NAT Gateway idle timeout.

Enable TCP keepalive so that idle connections periodically send traffic and remain active from the NAT Gateway's perspective.

Increase the NAT Gateway idle timeout where appropriate, while monitoring SNAT port consumption, as longer timeouts keep ports allocated for a longer duration.

For AKS workloads where individual applications do not expose connection pool settings, configuring OS-level TCP keepalive parameters through AKS Linux OS configuration is a supported mitigation. TCP keepalive probes can help prevent connections from becoming idle long enough for the NAT Gateway to remove the flow state.

Based on your testing results, increasing the NAT Gateway idle timeout and ensuring connection lifetime or keepalive settings remain below that threshold is a reasonable and supported approach for applications that maintain long-lived outbound connections.

Reference documentation:

• https://learn.microsoft.com/azure/virtual-network/nat-gateway/nat-gateway-resource
• https://learn.microsoft.com/azure/aks/improve-network-fault-tolerance-in-aks-using-tcp-keepalive

If the answer is helpful, please click "Accept Answer" and kindly upvote it. If you have extra questions about this answer, please click "Comment".

Answer 2

Azure NAT Gateway always silently drops flows when the TCP idle timeout expires; it does not support a configurable TCP reset-on-idle behavior. There is no NAT Gateway setting or companion Azure networking feature that will cause NAT Gateway to send FIN/RST on idle timeout. The documented behavior is that the flow is dropped, and a TCP reset is only sent later if traffic is seen on a non‑existent flow.

From the NAT Gateway documentation:

• When a connection reaches the idle timeout, the flow is silently dropped.
• A TCP reset is sent only when NAT Gateway detects traffic on a connection flow that no longer exists (for example, after idle timeout). This reset is unidirectional and is a consequence of the application trying to reuse the dead connection, not a proactive notification.

Because of this, the recommended pattern is to design applications and infrastructure so that:

• Outbound connections are used efficiently (to avoid SNAT exhaustion), and
• Long‑lived connections are either kept alive or refreshed before NAT idle timeout.

Relevant guidance from the NAT Gateway troubleshooting and best practices content:

1. Use connection pooling and reuse connections
   • Pool connections and reuse them rather than creating new TCP connections per request. This reduces SNAT port consumption and improves performance.
   • Ensure pooled connections are either actively used or refreshed/closed before the NAT idle timeout.
2. Use keepalives to reset the outbound idle timeout
   • The guidance explicitly calls out using TCP keepalives or application‑layer keepalives to refresh idle flows and reset the idle timeout timer.
   • Keepalives need to be enabled only on one side of the connection; when one side sends a keepalive, the other side ACKs, and the idle timer is reset on both sides.
   • This is the primary recommended mitigation for long‑lived connections that must survive NAT idle timeout.
3. Be cautious with increasing NAT idle timeout
   • The TCP idle timeout can be configured between 4 and 120 minutes.
   • Documentation explicitly states that increasing the idle timeout above the default 4 minutes increases the risk of SNAT port exhaustion, because flows are held longer and SNAT ports are not freed promptly.
   • Increasing idle timeout is described as a last resort and may not address the root cause.
4. Reduce SNAT pressure with architectural patterns
   • Use connection pooling and less aggressive retry logic so that failed connections do not cause bursts of new connections that consume SNAT ports.
   • Consider asynchronous polling patterns and Private Link for Azure PaaS destinations to reduce SNAT usage for internet‑bound traffic.
5. AKS and NAT Gateway resiliency
   • For AKS, the recommendation is to pair multi‑zone node pools with a StandardV2 NAT Gateway for zone‑redundant, resilient outbound connectivity. This does not change idle timeout behavior, but ensures consistent SNAT behavior across zones.

Regarding OS‑level TCP keepalives as mitigation:

• The documentation explicitly recommends using TCP keepalives or application‑layer keepalives to refresh idle flows and reset the NAT idle timeout.
• It does not distinguish between application‑configured keepalives and OS‑level keepalive configuration; both are mechanisms to generate periodic traffic on otherwise idle connections.
• Therefore, configuring TCP keepalive at the OS level (for example, via Linux TCP keepalive settings applied through AKS LinuxOSConfig) is aligned with the documented guidance, especially for components that do not expose connection pool TTL/idle settings.
• Care must be taken to choose keepalive intervals that are short enough to stay below the NAT idle timeout but not so aggressive that they create unnecessary traffic or mask inefficient connection usage.

In summary, for workloads that require persistent outbound connections behind NAT Gateway:

• There is no NAT Gateway option to send proactive FIN/RST on idle; silent drop plus reset on subsequent traffic is by design.
• The recommended infrastructure‑level mitigations are:
  • Use connection pooling and reuse connections efficiently.
  • Use TCP or application‑layer keepalives to keep important long‑lived connections active.
  • Avoid increasing NAT idle timeout except as a last resort, and monitor SNAT metrics if doing so.
  • Use Private Link where possible to offload traffic from NAT and reduce SNAT pressure.
• For components without application‑level idle controls, OS‑level TCP keepalive configuration is a supported pattern consistent with the official guidance to use keepalives to reset the NAT idle timeout.

---

References:

• Troubleshoot Azure NAT Gateway connectivity problems
• Troubleshoot outbound connectivity with NAT gateway and Azure services
• NAT gateway resource
• Load Balancer TCP Reset and Idle Timeout
• Troubleshoot Azure Load Balancer outbound connectivity issues