Application gateway timeout issue

Question

The Application Gateway is unable to establish connections to the backend Container Apps, resulting in unhealthy backend pools. Health probes are failing with "Cannot connect to backend server" errors.

Network Flow

1. External users connect to Application Gateway public IP
2. Application Gateway attempts to proxy traffic to backend Container Apps
3. Application Gateway sends health probes to Container Apps on ports 8080/5464
4. Health probes fail with "Cannot connect to backend server" errors
5. Traffic from ACI test instance to Container Apps times out

Troubleshooting Steps Performed

1. Verified DNS resolution: DNS lookups for Container Apps FQDNs work correctly
2. Tested direct connectivity from ACI: TCP connection attempts to the Container Apps on ports 8080/5464 time out
3. Reviewed NSG rules: Added explicit rules to allow traffic from AppGW subnet (10.0.2.0/24) to Container subnet (10.0.3.0/24) on ports 8080/5464
4. Modified Container Apps ingress settings: Changed from "Limited to Container Apps Environment" to "Limited to VNet"
5. Verified Application Gateway health probe settings: Configured to use correct ports (8080/5464) for the respective backends

Root Cause

Despite having proper NSG rules in place and configuring Container Apps to accept VNet traffic, the connectivity continues to fail. The Container Apps platform's internal firewall ("GuestFirewall") appears to be blocking incoming connections from other resources in the VNet.

Are we missing any configuration steps in our infrastructure setup? We've configured an Application Gateway to route traffic to Container Apps (api and ui) within the same VNet (tutors24x7-staging-vnet), but the Application Gateway health probes fail with GuestFirewall errors.

Specifically:

1. Is there any additional configuration needed to allow Application Gateway to communicate with Container Apps in a VNet-integrated environment?
2. Are there known limitations when using Application Gateway with Container Apps in the same VNet?
3. Do we need to configure any specific DNS settings or private endpoints for this scenario to work?
4. Are there any platform-level settings in the Container Apps Environment that might be blocking the Application Gateway health probes?

We've followed all documentation but still can't establish connectivity. Any guidance would be greatly appreciated.

Answer 1

Hello Dinesh Nimmagadda

1. Deploy a VM on the same vnet and test the connectivity to the container apps on TCP port (8080/5464) and let me know what is the response?
2. Perform a connection troubleshoot from application gateway to the container app ip on TCP port 8080/5464, share me the result
3. How was the back pool configured on app gateway for Container app? I mean IP address or FQDN? If possible share the screenshot.

Meantime,

Refer: https://learn.microsoft.com/en-us/azure/application-gateway/application-gateway-backend-health-troubleshooting#tcp-connect-error

Check whether you can connect to the backend server on the port mentioned in the HTTP settings by using a browser or PowerShell. For example, run the following command:

Test-NetConnection -ComputerName www.bing.com -Port 443.

If the port mentioned isn't the desired port, enter the correct port number for Application Gateway to connect to the backend server.

If the specified port is not the correct one, enter the appropriate port number for the Application Gateway to connect to the backend server.

If you are unable to connect to the port from your local machine, then:

1. Verify the network security group (NSG) settings of the backend server's network adapter and subnet to ensure inbound connections to the configured port are permitted. If not, create a new rule to allow these connections.
2. Confirm that the NSG settings of the Application Gateway subnet permit outbound public and private traffic to establish a connection.

---

Can you please update us if the action plan provided by was helpful?

Should there be any follow-up questions or concerns, please let us know and we shall try to address them.