Application gateway 502 bad gateway error intermittent

Question

Application gateway 502 bad gateway error intermittent

Ricardo Gabriel Nima Montalban 0

Good day

We have an AAGW: V2 standard (as Load balancer) in front of 2 VMs with a java spring boot application as backend pool. We have TLS end to end, but occurs that when we perform load tests with Jmeter, always show as between 3-16% bad gateway errors.

But the same backend with the next architecture: LB external --> 2 Nginx VMs --> LB internal --> 2 VMs backend pool, if we perfom the same load test we got cero errors in Jmeter.

In both we use the same custom probe with sucessful status.

So any idea about what we have to review or any suggestion.

Best regards

ChaitanyaNaykodi-MSFT 27,476 Reputation points Microsoft Employee Moderator

2023-08-22T19:49:59.5833333+00:00

@Ricardo Gabriel Nima Montalban

Just checking in to see if the below answer helped. If this answers your query, please don’t forget to click "Accept the answer" which might be beneficial to other community members reading this thread. And, if you have any further query do let us know.
scott 0 Reputation points

2024-01-18T22:37:58.2833333+00:00

We are experiencing the same issue with a similar configuration. We tried to migrate from a standard external load balancer to application gateway and started seeing these random 502 errors, but never had any issues with the standard load balancer. I have tried adjusting timeouts, probe settings, I even put a load balancer behind application gateway in a backend pool by itself, with my vms in a pool of the load balancer and am still seeing these 502 errors. I am currently working with MS support on this, but so far, they have not been able to diagnose the problem. When these random 502s happen, both backend vms are perfectly healthy. If I take one of the backend vms down, I immediately get 502 errors which also doesn't make sense, if application gateway is load balancing, why would it not fail over to the other vm? There is also no time lag in this situation, it's immediate even though the timeouts are set high and the probe timeouts are the default, should I not see at least a hang/wait while it's re-trying/timing out? We wanted to use application gateway to provide the WAF protection for our environment, but we can't get past this problem. Any help at this point would be appreciated. Thanks, Scott
scott 0 Reputation points

2024-01-18T22:38:36.3+00:00

Sorry, I meant to ask, was this issue ever resolved here?

1 answer

Your answer

ChaitanyaNaykodi-MSFT 27,476 Reputation points Microsoft Employee Moderator

2023-08-22T19:49:59.5833333+00:00

@Ricardo Gabriel Nima Montalban

Just checking in to see if the below answer helped. If this answers your query, please don’t forget to click "Accept the answer" which might be beneficial to other community members reading this thread. And, if you have any further query do let us know.
scott 0 Reputation points

2024-01-18T22:37:58.2833333+00:00

We are experiencing the same issue with a similar configuration. We tried to migrate from a standard external load balancer to application gateway and started seeing these random 502 errors, but never had any issues with the standard load balancer. I have tried adjusting timeouts, probe settings, I even put a load balancer behind application gateway in a backend pool by itself, with my vms in a pool of the load balancer and am still seeing these 502 errors. I am currently working with MS support on this, but so far, they have not been able to diagnose the problem. When these random 502s happen, both backend vms are perfectly healthy. If I take one of the backend vms down, I immediately get 502 errors which also doesn't make sense, if application gateway is load balancing, why would it not fail over to the other vm? There is also no time lag in this situation, it's immediate even though the timeouts are set high and the probe timeouts are the default, should I not see at least a hang/wait while it's re-trying/timing out? We wanted to use application gateway to provide the WAF protection for our environment, but we can't get past this problem. Any help at this point would be appreciated. Thanks, Scott
scott 0 Reputation points

2024-01-18T22:38:36.3+00:00

Sorry, I meant to ask, was this issue ever resolved here?

Answer 1

ChaitanyaNaykodi-MSFT 27,476 Microsoft Employee Moderator

@Ricardo Gabriel Nima Montalban

Welcome to the Microsoft Q&A forum.

Based on my understanding of your question above you are getting intermittent 502 bad gateway errors when performing load tests with JMeter on an Application Gateway v2 Standard (as Load Balancer) in front of 2 VMs with a Java Spring Boot application as a backend pool. However, when using the same backend with a different architecture (LB external --> 2 Nginx VMs --> LB internal --> 2 VMs backend pool), you got zero errors in JMeter. They use the same custom probe with a successful status in both architectures.

Regarding your test with Application Gateway as your load balancer. The bad gateway error might be due to Request time-out. When a user request is received, the application gateway applies the configured rules to the request and routes it to a backend pool instance. It waits for a configurable interval of time for a response from the backend instance. By default, this interval is 20 seconds. In Application Gateway v2, if the application gateway doesn't receive a response from the backend application in this interval, the request will be tried against a second backend pool member. If the second request fails, the user request gets a 502 error. The solution in cases will be to increase the request-timeout. Application Gateway allows you to configure this setting via the BackendHttpSetting. You can run the PowerShell command below to increase the time. The Request timeout maximum to private backend is 24 hours as per the limitation documentation. You can try this suggestion and see if that resolves the issue, if this does not help in resolving the issue you can enable diagnostic logging for your application gateway to help dig deeper into the issue.

New-AzApplicationGatewayBackendHttpSettings -Name 'Setting01' -Port 80 -Protocol Http -CookieBasedAffinity Enabled -RequestTimeout 60

Regarding your test using Azure Load Balancer, you observed no errors it might be because Azure Load Balancer is Azure's most performant Load Balancer all while keeping latency ultra-low. Additionally, unlike Application gateway Azure Load Balancer doesn't close or originate flows and idle timeout for Azure Load Balancer is set to 4 minutes by default.

Additional reference:

https://learn.microsoft.com/en-us/azure/application-gateway/application-gateway-troubleshooting-502#request-time-out

https://learn.microsoft.com/en-us/azure/architecture/guide/technology-choices/load-balancing-overview

Hope this answers your query. Please let me know if you have any additional questions. Thank you!

Please "Accept the answer" if the information helped you. This will help us and others in the community as well.

Ricardo Gabriel Nima Montalban 0 Reputation points

2023-08-11T20:01:05.49+00:00

Hello @ChaitanyaNaykodi-MSFT

Thanks for your help. I established to 240 sec (4 min like LB timeout) for timeout in the backendHttp settings but errors continuous. What about probes ? We use custom probe (interval: 30sec timeout: 30sec threshold:3) Do you consider is OK ?or we must increase timeouts ?

What about buffering and proxy options in Application gateway, to try to configure like nginx servers ?

As I said We use TLS with a well-know certificate for communication between AppGW and the application backend. The used certificate is installed on the application side and in the SO CA java store.
ChaitanyaNaykodi-MSFT 27,476 Reputation points Microsoft Employee Moderator

2023-08-15T00:25:35.1233333+00:00

@Ricardo Gabriel Nima Montalban

Thank you for sharing the additional details above.

I think it will be helpful if we could get to the exact cause for the issue above. Can you please check the diagnostic logs for you App Gateway and see if are able to get any additional details on the error above. If it helps you can take a look at the transaction id of the of the request which is getting the 502 error and use this ID to correlate the request to backend application.

It will also help if you could check the minimum instance count set for you Application Gateway. For Application Gateway v2 SKU, autoscaling takes six to seven minutes to scale out and provision additional set of instances ready to take traffic. Until then, if there are short spikes in traffic, your existing gateway instances might get under stress, and this may cause unexpected latency or loss of traffic. When under maximum load you can check the Compute Unit metric as discussed here. This will help us understand if the application gateway is getting overloaded during the load test.

Based on your questions above.

What about probes ? We use custom probe (interval: 30sec timeout: 30sec threshold:3) Do you consider is OK ?or we must increase timeouts ?

If you are getting Backend server timeout error for your custom health probe when under load, then increasing the timeout can help in this case. More information can be found here.

What about buffering and proxy options in Application gateway, to try to configure like nginx servers ?

I think this will depend on your backend application performance. By default, both Request and Response buffers are enabled on your Application Gateway and you can enable and disable them to see if this improves the web application performance.

You can also check this architecture guideline for application gateway.

Please let me know if you have any questions and please let me know if above does not help in resolving the issue. Thank you!
Ricardo Gabriel Nima Montalban 0 Reputation points

2023-08-23T14:20:01.1266667+00:00

Hello @ChaitanyaNaykodi-MSFT I continue with the issue after configuring 4min for request-timeout in BackendHttpSetting. In Diagnostic logs, I see 502 errors, but in the backend application side there is no error applications, CPU or RAM increase. There is a way for caching requests or what else can I check ? Because the only diference is that I am using Azure application gateway instead of Nginx.
ChaitanyaNaykodi-MSFT 27,476 Reputation points Microsoft Employee Moderator

2023-08-23T18:38:03.0066667+00:00

@Ricardo Gabriel Nima Montalban

Thank you for getting back. To troubleshoot the exact issue I think we will need specialized 1:1 session, where a support engineer can have a screen share session to pin point the issue. If you have a support plan you may file a support ticket, else could you please send an email to azcommunity@microsoft.com with the below details.

Subject : Attn Chaitanya

Thread URL: Link to this thread.

Subscription ID

Please let me know once you have done the same, I will enable a one-time free support ticket to you in this case. Thank you!

Share via

Application gateway 502 bad gateway error intermittent

1 answer

Your answer