Application Gateway backend health failure despite successful NVA forwarding We need assistance with Azure Application Gateway AEGWAP_01 backend connectivity to 10.8.2.234:443

Question

Application Gateway backend health failure despite successful NVA forwarding We need assistance with Azure Application Gateway AEGWAP_01 backend connectivity to 10.8.2.234:443

Marcos Adriano 0

Application Gateway backend health failure despite successful NVA forwarding

We need assistance with Azure Application Gateway AEGWAP_01 backend connectivity to 10.8.2.234:443.

Impact

All App Gateway backends are unhealthy and public application hosts return 502 Bad Gateway:

tenant-andon.automatia.mx
operator-andon.automatia.mx
admin-andon.automatia.mx

The application stack and Caddy reverse proxy on 10.8.2.234 are healthy. Direct origin access returns 200.

Topology

AEGWAP_01 (10.8.3.6)
  -> NVA AESSRVVPN (10.8.1.4)
  -> OpenVPN tun0 (10.8.0.1 -> 10.8.0.2)
  -> backend origin 10.8.2.234:443

Initial issue and correction

Azure Connection Troubleshoot originally reported:

Local Error: UDRLoop

Cause: route table AES-Routing was attached to both:

APPGW_01 subnet: 10.8.3.0/27
default subnet: 10.8.1.0/24, containing the NVA

The route table includes:

E1200: 10.8.2.0/24 -> Virtual appliance 10.8.1.4

This route is required on the App Gateway subnet but caused a self-reference on the NVA subnet. We removed the route-table association from the default subnet. Connection Troubleshoot hop details now show healthy routing:

AEGWAP_01 (10.8.3.6) -> 10.8.1.4 -> 10.8.2.234

Current verification

Azure NIC aessrvvpn48 has IP forwarding enabled.
Linux forwarding is enabled: net.ipv4.ip_forward = 1.
UFW allows routed TCP/443 traffic between eth0 and tun0.
UFW NAT is configured:

-A POSTROUTING -s 10.8.3.0/27 -d 10.8.2.0/24 -o tun0 -j MASQUERADE

AESSRVVPN-nsg allows Virtual Network outbound traffic.
AEGWAP_nsg allows Virtual Network inbound traffic.

Packet capture evidence

On AESSRVVPN:

App Gateway SYN packets arrive:

10.8.3.6:<ephemeral> -> 10.8.2.234:443 SYN

They are forwarded and SNATed through tun0:

10.8.0.1:<ephemeral> -> 10.8.2.234:443 SYN

The backend replies:

10.8.2.234:443 -> 10.8.0.1:<ephemeral> SYN-ACK

The NVA reverse-NATs the reply and emits:

10.8.2.234:443 -> 10.8.3.6:<ephemeral> SYN-ACK

However, App Gateway never sends the final TCP ACK. Connection Troubleshoot still reports all probes failed, and App Gateway Backend Health remains unhealthy.

Request

Please investigate why AEGWAP_01 is not receiving or accepting the reverse-NATed SYN-ACK from 10.8.2.234:443, despite:

healthy effective routing,
Azure NIC IP forwarding enabled,
NSG allow rules,
and packet captures proving the complete path through the NVA up to the outbound return packet.

We can provide the packet captures, Connection Troubleshoot results, effective routes, and NSG screenshots.Application Gateway backend health failure despite successful NVA forwarding

We need assistance with Azure Application Gateway AEGWAP_01 backend connectivity to 10.8.2.234:443.

Impact

All App Gateway backends are unhealthy and public application hosts return 502 Bad Gateway:

tenant-andon.automatia.mx
operator-andon.automatia.mx
admin-andon.automatia.mx

The application stack and Caddy reverse proxy on 10.8.2.234 are healthy. Direct origin access returns 200.

Topology

AEGWAP_01 (10.8.3.6)
  -> NVA AESSRVVPN (10.8.1.4)
  -> OpenVPN tun0 (10.8.0.1 -> 10.8.0.2)
  -> backend origin 10.8.2.234:443

Initial issue and correction

Azure Connection Troubleshoot originally reported:

Local Error: UDRLoop

Cause: route table AES-Routing was attached to both:

APPGW_01 subnet: 10.8.3.0/27
default subnet: 10.8.1.0/24, containing the NVA

The route table includes:

E1200: 10.8.2.0/24 -> Virtual appliance 10.8.1.4

This route is required on the App Gateway subnet but caused a self-reference on the NVA subnet. We removed the route-table association from the default subnet. Connection Troubleshoot hop details now show healthy routing:

AEGWAP_01 (10.8.3.6) -> 10.8.1.4 -> 10.8.2.234

Current verification

Azure NIC aessrvvpn48 has IP forwarding enabled.
Linux forwarding is enabled: net.ipv4.ip_forward = 1.
UFW allows routed TCP/443 traffic between eth0 and tun0.
UFW NAT is configured:

-A POSTROUTING -s 10.8.3.0/27 -d 10.8.2.0/24 -o tun0 -j MASQUERADE

AESSRVVPN-nsg allows Virtual Network outbound traffic.
AEGWAP_nsg allows Virtual Network inbound traffic.

Packet capture evidence

On AESSRVVPN:

App Gateway SYN packets arrive:

10.8.3.6:<ephemeral> -> 10.8.2.234:443 SYN

They are forwarded and SNATed through tun0:

10.8.0.1:<ephemeral> -> 10.8.2.234:443 SYN

The backend replies:

10.8.2.234:443 -> 10.8.0.1:<ephemeral> SYN-ACK

The NVA reverse-NATs the reply and emits:

10.8.2.234:443 -> 10.8.3.6:<ephemeral> SYN-ACK

However, App Gateway never sends the final TCP ACK. Connection Troubleshoot still reports all probes failed, and App Gateway Backend Health remains unhealthy.

Request

Please investigate why AEGWAP_01 is not receiving or accepting the reverse-NATed SYN-ACK from 10.8.2.234:443, despite:

healthy effective routing,
Azure NIC IP forwarding enabled,
NSG allow rules,
and packet captures proving the complete path through the NVA up to the outbound return packet.

We can provide the packet captures, Connection Troubleshoot results, effective routes, and NSG screenshots.

Vinayteja Lellela 110 Reputation points Microsoft External Staff Moderator

2026-08-05T23:36:20.22+00:00
Hello Marcos Adriano,

From what you described, the core network path from Application Gateway (AEGWAP_01) to the backend is working through the NVA (SYN arrives, forward/SNAT happens, SYN-ACK comes back). Yet Application Gateway still reports all backends as unhealthy and your app returns 502 Bad Gateway.

The typical causes we should still validate are the ones that can prevent Application Gateway from successfully completing/accepting the probe or establishing a TCP session for backend health.

**Things to validate
**1) Make sure NSG isn’t blocking Application Gateway health probe ports

Even if general traffic seems to flow, backend health probes can be impacted by NSG rules.

If the NSG on the Application Gateway subnet blocks inbound access to certain ranges, you should allow inbound from the GatewayManager service tag:

v1 SKU: allow inbound to 65503–65534

v2 SKU: allow inbound to 65200–65535

Source: GatewayManager

Action: verify the NSG effective rules for the Application Gateway subnet allow those inbound probe-management ports from GatewayManager.

Confirm UDR behavior for probes specifically: default route must allow Internet egress (as required by AppGW health logic)

The documentation for “Backend health status: Unknown” (and related scenarios) highlights that if there’s a UDR default route (0.0.0.0/0) whose next hop is not set to Internet, App Gateway probes may fail because return packets aren’t reaching destinations correctly.

In your case you’ve fixed a specific route-table loop, which is great. But please re-check that:

the Application Gateway subnet doesn’t have a default route (0.0.0.0/0) pointing to a next hop (virtual appliance/VPN gateway) that modifies packets in a way that breaks health/probe validation.

Action: use the guidance to check effective UDR and NSG for the Application Gateway NIC:

Get-AzEffectiveNetworkSecurityGroup -NetworkInterfaceName "nic1" -ResourceGroupName "rgName" Get-AzEffectiveRouteTable -NetworkInterfaceName "nic1" -ResourceGroupName "rgName"

Validate what Application Gateway thinks the probe/health is doing (protocol/port/TLS expectations)

The documentation set you provided includes a case where a TLS version mismatch caused backend health to appear unhealthy until the backend minimum TLS version was aligned.

Action: confirm the backend health probes:

protocol (HTTP vs HTTPS)

port

expected status codes

and whether the backend TLS configuration supports what Application Gateway offers.

This matters even when a browser/test curl to the origin works, because probe behavior and TLS negotiation can differ.

If network is “fine,” the remaining bucket is backend application session establishment

If routing/NSG/UDR aren’t the issue, you should confirm the backend is actually listening and able to establish TCP sessions on the port Application Gateway probes.

Action: On the backend host, verify it’s listening on the configured port (example guidance uses netstat):

netstat

Confirm the service is LISTENING on the exact port App Gateway health probe uses (in your case, 443) and that it isn’t behaving differently when the source IP is SNATed by the NVA/OpenVPN.

Can you please update us if the action plan provided was helpful?

Should there be any follow-up questions or concerns, please let us know and we shall try to address them.

If these answer your question, click "Upvote" which may be beneficial to other community members reading this thread.
Vinayteja Lellela 110 Reputation points Microsoft External Staff Moderator

2026-08-07T00:43:11.2866667+00:00

Hello **Marcos Adriano,

**following up to see if you had a chance to check the above response and if it was helpful. Please let me know if you have any questions.
Vinayteja Lellela 110 Reputation points Microsoft External Staff Moderator

2026-08-11T18:01:53.4133333+00:00

Hello Marcos Adriano

Following up to see if you had a chance to check the above response and if it was helpful. Please let me know if you have any questions.

1 answer

Your answer

Vinayteja Lellela 110 Reputation points Microsoft External Staff Moderator

2026-08-07T00:43:11.2866667+00:00

Hello **Marcos Adriano,

**following up to see if you had a chance to check the above response and if it was helpful. Please let me know if you have any questions.
Vinayteja Lellela 110 Reputation points Microsoft External Staff Moderator

2026-08-11T18:01:53.4133333+00:00

Hello Marcos Adriano

Following up to see if you had a chance to check the above response and if it was helpful. Please let me know if you have any questions.

Answer 1

Resolution update: Application Gateway backend health never completed across an NVA/OpenVPN tunnel — fixed by moving the backend into the VNet

Update / resolution — thank you for the detailed checklist.

Posting the outcome in case it helps someone with the same symptom.

Background

The outage started after OS/package updates (and a restart) on our network virtual appliance (NVA). The NVA carried the only path from the Application Gateway to a backend origin that lived outside Azure, reached over an OpenVPN tunnel with SNAT applied. The update wiped the non-persistent Linux IP-forwarding and NAT/MASQUERADE state the path depended on, so all public hosts began returning 502 Bad Gateway while the origin itself stayed healthy (it answered correctly when reached directly).

We re-applied forwarding/NAT and also found and removed a route-table self-reference (a UDR loop where the same route table was associated with both the Application Gateway subnet and the NVA subnet). After that, effective routing showed healthy hops — but backend health still would not recover.

The persistent symptom

Packet captures proved the full path worked in both directions:

The probe SYN reached the NVA, was forwarded and SNATed across the tunnel to the backend.
The backend returned SYN-ACK.
The NVA reverse-NATed that reply back toward the Application Gateway.
…but the Gateway never sent the final ACK, so the handshake never completed. Backend health stayed Unhealthy and the public hosts kept returning 502.

What we validated (none of it restored backend health)

NSG on the Application Gateway subnet allowed inbound from the GatewayManager service tag on the probe-management port range for our SKU.
No 0.0.0.0/0 default route on the Application Gateway subnet forced probe traffic through a packet-modifying next hop; the earlier UDR self-reference was removed.
Backend probe settings (protocol / port / expected status) matched the reverse proxy; the origin returned 200 directly and presented a valid wildcard certificate; TLS negotiation to the origin succeeded from a direct client.
The backend was listening on the probed port and served correctly whenever it was reached directly, bypassing the Gateway.

Root cause (as far as we could establish)

The Application Gateway backend was outside the VNet, reachable only through an NVA + OpenVPN tunnel with SNAT. Even with captures proving packets traversed the whole path in both directions, the Gateway would not complete/accept the probe's TCP session across that asymmetric, SNATed, tunneled return path. We could not get backend health to go Healthy while the origin remained behind those hops, regardless of further NSG/UDR tuning — because the problem was not a block, it was the Gateway being unable to establish a usable TCP session over that path.

Resolution

We stopped trying to make the health check succeed across the tunnel and removed the cross-network hops entirely:

Relocated the application stack onto an Azure VM in the same VNet as the Application Gateway.
Repointed the backend pool at that VM's private IP (a native, in-VNet path — no NVA, no OpenVPN, no on-prem hop).
Changed the API backend setting from HTTPS:8001 to HTTPS:443 so it flows through the same reverse proxy as the other hosts.

As soon as the backend was an Azure-local VM with a direct in-VNet path, all backend targets went Healthy and every public host returned HTTP/2 200 immediately.

Takeaway

If Application Gateway backend health is stuck Unhealthy even though captures show a complete SYN → SYN-ACK exchange, and your backend is only reachable across an NVA/VPN tunnel with SNAT outside the VNet, the fix may not be another NSG/UDR change — the probe's TCP session simply may not survive that asymmetric tunneled path. Giving the backend a native, symmetric Azure path (e.g. putting it in the same VNet as the Gateway) resolved it cleanly for us.

Thanks again for the help.