Azure application gateway giving BAD Gateway 502

Question

I have upgraded my cluster to 1.24.9 and it started giving an error called

502 Bad Gateway

My Nginx Version

Here is my nginx File

`apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  namespace: develop
  name: develop-unified-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /$1
    nginx.ingress.kubernetes.io/proxy-body-size: 50m
    nginx.ingress.kubernetes.io/request-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
    er-snippets: |
      location / {
        proxy_set_header Upgrade $http_upgrade;
        proxy_http_version 1.1;
        proxy_set_header X-Forwarded-Host $http_host;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header Host $host;
        proxy_set_header Connection "Upgrade";
        proxy_cache_bypass $http_upgrade;
        }
spec:
  rules:
    - host: devgateway.com
      http:
        paths:
          - backend:
              service:
                name: adminer-dev
                port:
                  number: 8080
            path: /adminer.php(/|$)(.*)
            pathType: Prefix
          - backend:
              service:
                name: fms-fleet
                port:
                  number: 8010
            path: /fms-fleet/(.*)
            pathType: Prefix
          - backend:
              service:
                name: fms-driver
                port:
                  number: 8011
            path: /fms-driver/(.*)
            pathType: Prefix
          - backend:
              service:
                name: fms-maintenance
                port:
                  number: 8012
            path: /fms-maintenance/(.*)
            pathType: Prefix
          - backend:
              service:
                name: user-ms
                port:
                  number: 8013
            path: /user-ms/(.*)
            pathType: Prefix
          - backend:
              service:
                name: cob-packages
                port:
                  number: 8021
            path: /cob-packages/(.*)
            pathType: Prefix
          - backend:
              service:
                name: cob-customer
                port:
                  number: 8022
            path: /cob-customer/(.*)
            pathType: Prefix
          - backend:
              service:
                name: cob-inventory
                port:
                  number: 8023
            path: /cob-inventory/(.*)
            pathType: Prefix
          - backend:
              service:
                name: cob-new-customer
                port:
                  number: 8024
            path: /cob-new-customer/(.*)
            pathType: Prefix
          - backend:
              service:
                name: maptrail-generic
                port:
                  number: 8031
            path: /maptrail-generic/(.*)
            pathType: Prefix
          - backend:
              service:
                name: map-trail
                port:
                  number: 8032
            path: /map-trail/(.*)
            pathType: Prefix
          - backend:
              service:
                name: smpp-mail
                port:
                  number: 8033
            path: /smpp-mail/(.*)
            pathType: Prefix
          - backend:
              service:
                name: jubaili-be
                port:
                  number: 8035
            path: /jubaili-be/(.*)
            pathType: Prefix
          - backend:
              service:
                name: at-advance
                port:
                  number: 8036
            path: /at-advance/(.*)
            pathType: Prefix
          - backend:
              service:
                name: loki
                port:
                  number: 3100
            path: /loki/(.*)
            pathType: Prefix
          - backend:
              service:
                name: camera-service
                port:
                  number: 8037
            path: /camera-service/(.*)
            pathType: Prefix
          - backend:
              service:
                name: report-analytics
                port:
                  number: 8038
            path: /report-analytics/(.*)
            pathType: Prefix
          - backend:
              service:
                name: scheduler-ms
                port:
                  number: 8039
            path: /scheduler-ms/(.*)
            pathType: Prefix
          - backend:
              service:
                name: platform-audit
                port:
                  number: 8040
            path: /platform-audit/(.*)
            pathType: Prefix
          - backend:
              service:
                name: device-manager
                port:
                  number: 8041
            path: /device-manager/(.*)
            pathType: Prefix
          - backend:
              service:
                name: platform-role-access
                port:
                  number: 8042
            path: /platform-role-access/(.*)
            pathType: Prefix
          - backend:
              service:
                name: notification-alerts
                port:
                  number: 8043
            path: /notification-alerts/(.*)
            pathType: Prefix
          - backend:
              service:
                name: smart-building
                port:
                  number: 8044
            path: /smart-building/(.*)
            pathType: Prefix
          - backend:
              service:
                name: live-stream
                port:
                  number: 5000
            path: /live-stream/(.*)
            pathType: Prefix
          - backend:
              service:
                name: live-stream
                port:
                  number: 4400
            path: /ws-live-stream/(.*)
            pathType: Prefix
          - backend:
              service:
                name: sp-platform
                port:
                  number: 8087
            path: /sp-platform/(.*)
            pathType: Prefix`

Answer 1

Hello Taqi H. Rizvi,

Thank you for reaching out!

As per this Behavioral Change from AKS Release Notes:
"For Kubernetes 1.24+ the services of type LoadBalancer with appProtocol HTTP/HTTPS will switch to use HTTP/HTTPS as health probe protocol (while before v1.24.0 it uses TCP). And / will be used as the default health probe request path. If your service doesn’t respond 200 for /, please ensure you're setting the service annotation service.beta.kubernetes.io/port_{port}_health-probe_request-path or service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path (applies to all ports) with the correct request path to avoid service breakage."

This can lead to ingress traffic is not being forwarded to the respective services in the AKS cluster, due to the failure in the load balancer's health probes.

The most common scenario faced is with the nginx ingress controller, where the default path for the probe is /healthz.

As mentioned in the release notes, prior to the upgrade, the health probe would be using TCP, and thus the issue is not occurring.

After the upgrade, the health probe would start using HTTP/s on the / path, which would cause it to fail.

To resolve the issue, add the following annotation to the affected nginx-ingress controller Kubernetes service type LoadBalancer to point it to the correct path.

service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path: "/healthz"

---

I hope this is helpful. If any clarification needed, let me know and I will do my best to answer.

Please "Accept as Answer" and Upvote if it helped, so that it can help others in the community looking for help on similar topics.

Thank you!

Answer 2

This error may happen for the following main reasons:

• NSG, UDR, or Custom DNS is blocking access to backend pool members.
• Backend VMs or instances of virtual machine scale set aren't responding to the default health probe.
• Invalid or improper configuration of custom health probes.
• Azure Application Gateway's backend pool isn't configured or empty.
• None of the VMs or instances in virtual machine scale set are healthy.
• Request time-out or connectivity issues with user requests.

Please get more details from this website: https://learn.microsoft.com/en-us/azure/application-gateway/application-gateway-troubleshooting-502

Answer 3

Hello Taqi H. Rizvi

Welcome to Microsoft Q&A Platform, thanks for posting your query here.

The error message "502 Bad Gateway" indicates that the Azure Application Gateway is not able to connect to the backend service.

Can you try below troubleshooting steps to see if these help:

-Verify that your backend service is up and running, and responding to requests. You can use tools like kubectl to check the status of your Kubernetes pods and services.

-Ensure that the backend service is reachable from the Azure Application Gateway. You can use tools like traceroute and telnet to diagnose network connectivity issues.

-Review the Nginx logs to see if there are any errors or warnings related to the requests that are failing. You can find the logs by running the following command in the terminal:

kubectl logs -n ingress-nginx <nginx-pod-name>

-Check that the Ingress resource is configured correctly. Make sure that the backend service is specified correctly, and that the path and pathType match the actual endpoints of the backend service.

-Consider upgrading your Nginx version to the latest stable release, which may contain bug fixes and improvements that could resolve the issue.

-Try increasing the timeout settings in the Ingress annotations. This will allow the Azure Application Gateway to wait longer for a response from the backend service.

Ref: https://learn.microsoft.com/en-us/azure/application-gateway/ingress-controller-troubleshoot

Hope this helps.