IP is not getting assigned to Azure Application Gateway Ingress hence pod is not coming up due to readiness probe failure

Reshma Nair 120

I have created a private AKS cluster(subnet: AKSSubnet) with Application Gateway as Ingress controller using terraform.

ingress_application_gateway {
    gateway_name  = "appgw-ingress"
    subnet_cidr   = ".../24"
  }

After cluster creation I deployed the Kubernetes manifests(namespace, service, ingress. deployment....) which are terraform files. My ingress class is azure-application-gateway.

resource "kubernetes_ingress_v1" "ingress" {
  metadata {
    name = "appgw-ingress"
    namespace = "aks-ns"
    annotations = {
        "appgw.ingress.kubernetes.io/cookie-based-affinity" = "true"
        "appgw.ingress.kubernetes.io/ssl-redirect" = "true"
        
        
    }
  }

  spec {
    ingress_class_name = "azure-application-gateway"

Everything got deployed successfully but IP address is not assigned to the Ingress. User's image

Because of this I am not able to make the DNS entry for our domain for Ingress host. Hence my pod is also not coming up due to readiness probe failure as it is redirecting to the same ingress host. I tried adding the application-gateway-pip from the MC* Resource Group but of no use.

readiness_probe {
            http_get {
              path = "/"
              port = 80
              host = <ingress-host">
            }
            initial_delay_seconds = 10
            period_seconds        = 10
            timeout_seconds       = 10
          }
        }
      }
    }

I tried many ways and everything failed. I don't have deep knowledge in networking. Any leads will be helpful. Thanks in advance.

Luis Arias 6,461 Reputation points

2024-02-21T17:18:45.5266667+00:00
Hi Reshma Nair, You might need to do some extra checks to find the root cause of problem:

Ingress Controller Logs: Use kubectl logs -n kube-system -l app=ingress-appgw -f to check the logs.

Test Application: Deploy a simple test application to verify your Application Gateway and AKS setup.- Readiness Probe: Check the configuration of your readiness probe and ensure the service it is checking is ready and accessible.

Check that the public IP (Application Gateway Public IP) is correctly configured.

Check your subnet and network security group (NSG) rules. Ensure necessary ports are open and routing is correctly configured.

Could you share some of this addtional information to help you better.

Luis
Luis Arias 6,461 Reputation points

2024-02-21T17:20:48.7833333+00:00
Anveshreddy Nimmala 3,540 Reputation points Microsoft Vendor

2024-02-23T05:37:29.4333333+00:00

Hi Reshma Nair, Welcome to microsoft Q&A, Thankyou for posting your query here. You might need to do more troubleshooting to the root cause of the issue. Here are some steps to troubleshoot. Check if the subnet where the Azure Application Gateway Ingress is deployed has the correct service endpoints. The subnet should have service endpoints for Microsoft.ContainerRegistry and Microsoft.ContainerService. Verify that the Azure Application Gateway Ingress is associated with the correct subnet and virtual network. The subnet should be the same subnet where the AKS cluster is deployed, and the virtual network should be the same virtual network where the AKS cluster is deployed. Check if the Azure Application Gateway Ingress is associated with the correct subnet delegation. The subnet should be delegated to Microsoft.Network/applicationGateways. Verify that the Azure Application Gateway Ingress is associated with the correct backend pool, listener, rule, and public IP address. These should be the same backend pool, listener, rule, and public IP address where the Kubernetes service and ingress are deployed. Check the logs of the Ingress Controller pods in your AKS cluster. Look for any errors or warnings that might indicate why the IP address is not being assigned. You can use the kubectl logs command to view the logs. Ensure that there are no network policies in place that might be blocking traffic between the Application Gateway and the AKS cluster. Network security groups (NSGs) or Azure Firewall configurations could potentially block traffic. Check the configuration settings of the Application Gateway, including backend pools, listeners, rules, and probes. Ensure that the Application Gateway is properly configured to work with AKS. After running Terraform apply, check the output for any errors or warnings related to the creation of the AKS cluster and the Application Gateway. Terraform might provide useful information that can help diagnose the issue. Ensure that the service principal has at least Network Contributor or higher permissions on the resource group containing the Application Gateway. hope this helps you get the issue root cause, if not please provide more information about the error message you are seeing and the steps you have taken so far.
Reshma Nair 120 Reputation points

2024-02-23T14:15:23.0833333+00:00
I altered the readiness probe code as below and the pods came up but still application is not accessible.

readiness_probe { http_get { path = "/" port = 80 # host = <ingress-host> http_header { name = "Host" value = <ingress-host> } }

@Anveshreddy Nimmala I created my AKS cluster under a subnet(no NSG is associated with this subnet) which is in a particular address range and in the cluster added an ingress appgateway in a different subnet CIDR.

ingress_application_gateway { gateway_name = "appgw-ingress" subnet_cidr = "../24" }

This AppGW is not an already existing resource. When you deploy your AKS cluster a new resource group will be created with name starting MC*. In this resource group AppGW is created as a Managed resource along with a Kubernetes load balancer, PIP, NSG etc.(so far this is my understanding). I cannot see a subnet created under this resource group for AppGW Ingress. After cluster creation I deploy other Kubernetes manifests(Namespace, storage, Ingress, service, deployment...). For Ingress I am expecting the same AppGW PIP to get assigned but no IP address is getting assigned to my Ingress.

So I made DNS entry with the AppGW PIP for Ingress host soon after cluster creation and then deployed the kubernetes manifests. Everything created sucessfully, pods are up and running but application is not accessible.

No error logs are generated by pods.

Some questions in my mind are

Is this an Azure issue?

Kubernetes version has some role here (My AKS cluster version : 1.27.7)

If these are not then I am confused and not sure where to look for more details.

Many thanks for your responses @Anveshreddy Nimmala @Luis Arias

Accepted answer

Anveshreddy Nimmala 3,540 Reputation points Microsoft Vendor

2024-02-26T05:22:15.2166667+00:00

Hi Reshma Nair,

thankyou for replying back.

Ensure that the Application Gateway and the AKS cluster are in the same virtual network but in different subnets.

Associate the same route table created by AKS to the Application Gateway's subnet.

Check if the ingress resource is created successfully and annotated with kubernetes.io/ingress.class: azure/application-gateway.

Verify that the AGIC pod is running.

Check the AGIC pod logs for any useful information.

If necessary, increase the verbosity of the AGIC logs by setting the appgw.ingress.kubernetes.io/log-level annotation to debug.

Hope this helps you.
Please sign in to rate this answer.

1 person found this answer helpful.
Reshma Nair 120 Reputation points

2024-02-27T09:14:35.9933333+00:00

Hi @Anveshreddy Nimmala ,

Thank you for your guidance in this issue.

Everything is working great now. I have an IP assigned to my Ingress, all my deployments(pods) are up and able to access my application. Adding my workarounds/my working configurations below.

I have everything(AKS cluster, Ingress AppGateway etc.) under same network(AKS Agent pool virtual network/aks vnet) in MC* resource group

The Subnet CIDR for ingress appgateway is a /16 subnet in the usable host range of 10.224.0.0/12(10.224.0.0 - 10.239.0.0).

ingress_application_gateway { gateway_name = "appgateway-ingress" subnet_cidr = "10.225.0.0/16"

I think this is where my configuration went wrong. After I updated the subnet CIDR to the above mentioned range, the gateway module is deployed successfully(new aks vnet is created, new appgateway resource is provisioned and a new PIP is assigned for the resource)

But now I would like to implement Firewall configuration(Azure Firewall) to my AKS cluster. So my question is can I peer my AKS subnet with my Firewall Subnet for this configuration.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

IP is not getting assigned to Azure Application Gateway Ingress hence pod is not coming up due to readiness probe failure

0 additional answers

Your answer