Azure Application Gateway to AKS internal load balancer using native Kubernetes NGINX controller

Duncan House 0

I've been struggling with a 502 Bad Gateway error when configuring an Azure Application Gateway, with a backend pool of an IP internal Load Balancer. When testing the nginx ingress, I can resolve internally via the nginx controller and ingress on the cluster. But, when I try and link up an Azure App Gateway v2, I receive a 502. Note, my ingress does not associate to host name, I'm just trying to test the setup initially all via IP. Any guidance would be most grateful.

Sai Prasanna Sinde 1,265 Reputation points Microsoft Vendor

2024-08-27T03:12:42.28+00:00

Hi@Duncan House,

We are looking into this issue to get more information related to your query and will get back to you as soon as we have an update.
Sai Prasanna Sinde 1,265 Reputation points Microsoft Vendor

2024-08-27T13:15:54.33+00:00
Hi @Duncan House

Welcome to Microsoft Q&A Platform. Thank you for reaching out & hope you are doing well.

A 502 bad gateway indicates that the Application Gateway is not communicating with the Backend Pool:

Verify the IP address configured in the backend pool of the Application Gateway which matches the Internal Load balancer. Verify the Port number configured in the backend pool of the Application Gateway is the correct port that the Internal Load balancer using.

Verify the configuration of the health probe, whether the backend Internal Load balancer is responding to the health probe or not.

Verify the NSG whether it is blocking the access to the backend and also check the NSG's which are associated to the Application Gateway subnet and make sure it is not blocking the access to the backend.

Make sure that the backend pool of the Internal Load balancer contains the IP address of the NGINX.

Make sure that the listener on the Application Gateway is configured with the correct frontend port and protocol.

Verify the routing rules which are correctly pointing to the backend pool.

Please try to increase the request time out, which might resolve the issue. You can increase the time out in BackendHttpSetting of the Application Gateway or else you can use the PowerShell scripting New-AzApplicationGatewayBackendHttpSettings -Name 'Setting01' -Port 80 -Protocol Http -CookieBasedAffinity Enabled -RequestTimeout 60. For your reference: https://learn.microsoft.com/en-us/azure/application-gateway/application-gateway-troubleshooting-502#solution-3

The Request timeout maximum to private backend is 24 hours. For your reference: https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/azure-subscription-service-limits#application-gateway-limits

Finally try to enable and review the AGW logs, you might get a reason for failing the request.Kindly let us know if the above helps or you need further assistance on this issue.
Duncan House 0 Reputation points

2024-08-27T15:06:25.95+00:00
Thankyou for responding....

Going through the points 1 - 9 above...

BEP is set as the internal LB IP

Backend health is not responding; not sure how to fix this?

Image below.

Checked and aligned with point 1.

First test setup HTTP port 80

Checked. Seems OK

Increased, see below.

Further notes...

The AKS cluster is within it's own private vnet.
The internal LB is an IP within the same vnet.

The AppGW is a Public IP, trying to reach backend pool of internal LB.

I'm missing something very simple, but sadly I just cannot spot it.... any more help gratefully received.
Sai Prasanna Sinde 1,265 Reputation points Microsoft Vendor

2024-08-28T05:09:57.2366667+00:00

Hi @Duncan House,

Looks like you have implemented App Gateway->Load Balancer->AKS

You can use AGIC (Application Gateway Ingress Controller) directly instead of Load Balancer here.

As documented here

https://learn.microsoft.com/en-us/azure/application-gateway/ingress-controller-overview

AGIC helps eliminate the need to have another load balancer/public IP address in front of the AKS cluster and avoids multiple hops in your DataPath before requests reach the AKS cluster. Application Gateway talks to pods using their private IP address directly and doesn't require Node Port or Kube Proxy services. This capability also brings better performance to your deployments.

Could you please tell us the reason why you are using App Gateway->Load Balancer->AKS in this way as you can use AGIC->AKS instead.

And as you mentioned Backend health is not responding,

502 errors can also be frequent indicators that the default health probe can't reach backend VMs.

If you are using a default health probe, you can refer to the below document and do the troubleshooting steps.

https://learn.microsoft.com/en-us/azure/application-gateway/application-gateway-troubleshooting-502#problems-with-default-health-probe

Please confirm whether you have created the health probe for backend pool. If not, please create HTTP settings first and then create a health probe by using the HTTP settings, after that you can test the backend pool health with the probe.

For your reference:

https://learn.microsoft.com/en-us/azure/application-gateway/application-gateway-create-probe-portal#createprobe

If you are using a custom health probe, you can refer to the below document and do the troubleshooting steps.

https://learn.microsoft.com/en-us/azure/application-gateway/application-gateway-troubleshooting-502#problems-with-custom-health-probe

For Additional reference, please go through this document related to 502 gateway errors.

https://learn.microsoft.com/en-us/azure/application-gateway/application-gateway-troubleshooting-502#overview

Kindly let us know if the above helps or you need further assistance on this issue.
Sai Prasanna Sinde 1,265 Reputation points Microsoft Vendor

2024-08-29T09:41:47.24+00:00

Hi @Duncan House,

Following up to see if the above suggestion was helpful. And, if you have any further query kindly do let us know.
Duncan House 0 Reputation points

2024-08-29T16:50:06.1833333+00:00

Hi, thankyou for the recommendation regarding AGIC, however not yet suitable for an overlay network. Our current AKS implementation adopts overlay, so my example is an AKS cluster with unmanaged NGINX, since we had recent issues either the app routing add on, which has led us down this path. Hence, the exploratory testing of NGINX with App Gateway.
Duncan House 0 Reputation points

2024-08-29T16:53:52.39+00:00

We are currently adopting AKS with network overlay, therefore AGIC not currently available, and our recent implementation of app routing had issues raised with Microsoft. Hence, we are now exploring the use of unmanaged NGINX with App GW.
Sai Prasanna Sinde 1,265 Reputation points Microsoft Vendor

2024-08-30T04:46:45.9833333+00:00

Hi @Duncan House,

Thanks for the reply. Just checking in to see if your issue got resolved or do you need any more information. If the issue is still on, please feel free to ask any more questions. We are happy to assist you.

Regards,

Sai Prasanna Sinde.

Share via

Azure Application Gateway to AKS internal load balancer using native Kubernetes NGINX controller

Your answer