I've upgraded kubernetes from 1.22.6 to 1.25.5, this required me to upgrade the ingress controller to latest version as it was incompatible before hand (1.1.2). As a result my connections to hosted applications failed with a gateway timeout.

Question

I've upgraded the aks cluster kubernetes version from 1.22.6 to 1.25.5, following this no more request came into the ingress controller, this was due to incompatibility that wasn't picked up. The ingress controller was then upgraded from 1.1.2 to latest 1.7.0. The connectivity issues persisted, i've checked the load balancer and the data and health availability is at zero. The ports are correctly defined in the yaml for ingress service.

I've checked the ingress pod events and the health check to the cluster ips returning 400 .

I believe some of the config has been deprecated following the upgrade and so removed, any ideas what else is needed to get the cluster reachable?

Yaml for ingress service:

kind: Service
apiVersion: v1
metadata:
  name: ingress-nginx-controller
  namespace: ingress
  uid: 14deb8a1-bd22-44cd-b255-1c5280dd11b7
  resourceVersion: '172443872'
  creationTimestamp: '2022-03-29T13:46:18Z'
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.7.0
    helm.sh/chart: ingress-nginx-4.6.0
  annotations:
    meta.helm.sh/release-name: ingress-nginx
    meta.helm.sh/release-namespace: ingress
    service.beta.kubernetes.io/azure-dns-label-name: uk1d-kc-001-dns
    service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path: /healthz
  finalizers:
    - service.kubernetes.io/load-balancer-cleanup
  managedFields:
    - manager: kube-controller-manager
      operation: Update
      apiVersion: v1
      time: '2022-03-29T13:46:32Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:finalizers:
            .: {}
            v:"service.kubernetes.io/load-balancer-cleanup": {}
    - manager: helm
      operation: Update
      apiVersion: v1
      time: '2023-03-28T09:46:38Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            .: {}
            f:meta.helm.sh/release-name: {}
            f:meta.helm.sh/release-namespace: {}
            f:service.beta.kubernetes.io/azure-dns-label-name: {}
            f:service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path: {}
          f:labels:
            .: {}
            f:app.kubernetes.io/component: {}
            f:app.kubernetes.io/instance: {}
            f:app.kubernetes.io/managed-by: {}
            f:app.kubernetes.io/name: {}
            f:app.kubernetes.io/part-of: {}
            f:app.kubernetes.io/version: {}
            f:helm.sh/chart: {}
        f:spec:
          f:allocateLoadBalancerNodePorts: {}
          f:externalTrafficPolicy: {}
          f:internalTrafficPolicy: {}
          f:ipFamilies: {}
          f:ipFamilyPolicy: {}
          f:ports:
            .: {}
            k:{"port":80,"protocol":"TCP"}:
              .: {}
              f:appProtocol: {}
              f:name: {}
              f:port: {}
              f:protocol: {}
              f:targetPort: {}
            k:{"port":443,"protocol":"TCP"}:
              .: {}
              f:appProtocol: {}
              f:name: {}
              f:port: {}
              f:protocol: {}
              f:targetPort: {}
          f:selector: {}
          f:sessionAffinity: {}
          f:type: {}
    - manager: cloud-controller-manager
      operation: Update
      apiVersion: v1
      time: '2023-03-28T10:19:14Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:finalizers:
            .: {}
            v:"service.kubernetes.io/load-balancer-cleanup": {}
        f:status:
          f:loadBalancer:
            f:ingress: {}
      subresource: status
    - manager: Mozilla
      operation: Update
      apiVersion: v1
      time: '2023-03-28T14:25:09Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            f:service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path: {}
        f:spec:
          f:loadBalancerSourceRanges: {}
spec:
  ports:
    - name: http
      protocol: TCP
      appProtocol: http
      port: 80
      targetPort: http
      nodePort: 31418
    - name: https
      protocol: TCP
      appProtocol: https
      port: 443
      targetPort: https
      nodePort: 30526
  selector:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
  clusterIP: 10.0.112.80
  clusterIPs:
    - 10.0.112.80
  type: LoadBalancer
  sessionAffinity: None
  externalTrafficPolicy: Cluster
  ipFamilies:
    - IPv4
  ipFamilyPolicy: SingleStack
  allocateLoadBalancerNodePorts: true
  internalTrafficPolicy: Cluster
status:
  loadBalancer:
    ingress:
      - ip: 20.90.247.53

Answer 1

I saw a similar issue and solution provided in the following thread:

https://learn.microsoft.com/en-us/answers/questions/1186078/upgraded-kubernetes-version-cant-connect-to-ingres

It looks like you need to add the following annotation to the affected load balancer service to point it to the correct path.

service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path: "/healthz"

The most common scenario that users would face is with the nginx ingress controller, where the default path for the probe is /healthz and it was necessary to change the health probe to use TCP before upgrade

---

--please don't forget to upvote and Accept as answer if the reply is helpful--

Answer 2

Hello, @Sachithya Gamage !

Why am I having connectivity issues with my ingress controller after an upgrade?

I reached out to an AKS specialist and am confirming what @Manu Philip has said and wanted to add a bit more information from the release notes:

https://github.com/Azure/AKS/blob/master/CHANGELOG.md#release-notes-1

Behavioral Changes

For Kubernetes 1.24+ the services of type LoadBalancer with appProtocol HTTP/HTTPS will switch to use HTTP/HTTPS as health probe protocol (while before v1.24.0 it uses TCP). And / will be used as the default health probe request path. If your service doesn’t respond 200 for /, please ensure you're setting the service annotation service.beta.kubernetes.io/port_{port}_health-probe_request-path or service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path (applies to all ports) with the correct request path to avoid service breakage.

The most common scenario with the nginx ingress controller is where the default path for the probe is /healthz. As you can see from the release notes, prior to the upgrade the health probe would be using TCP so you wouldn't run into the issue. After the upgrade, the health probe would start using HTTP/s on the / path which would cause it to fail.

To resolve this issue, you would add the following annotation to the affected load balancer service to point it to the correct path:

service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path: "/healthz"

You can also see this in the portal:

Before:

After: