Create an Application Gateway Ingress Controller in Azure Kubernetes Service using Terraform

Article tested with the following Terraform and Terraform provider versions:

Terraform enables the definition, preview, and deployment of cloud infrastructure. Using Terraform, you create configuration files using HCL syntax. The HCL syntax allows you to specify the cloud provider - such as Azure - and the elements that make up your cloud infrastructure. After you create your configuration files, you create an execution plan that allows you to preview your infrastructure changes before they're deployed. Once you verify the changes, you apply the execution plan to deploy the infrastructure.

Azure Kubernetes Service (AKS) manages your hosted Kubernetes environment. AKS makes it quick and easy to deploy and manage containerized applications without container orchestration expertise. AKS also eliminates the burden of taking applications offline for operational and maintenance tasks. Using AKS, you can do such tasks as provisioning, upgrading, and scaling resources on-demand.

An Application Gateway Ingress Controller (AGIC) provides various features for Kubernetes services. These features include reverse proxy, configurable traffic routing, and TLS termination. Kubernetes Ingress resources are used to configure the Ingress rules for individual Kubernetes services. An Ingress controller allows a single IP address to route traffic to multiple services in a Kubernetes cluster. All this functionality is provided by Azure Application Gateway, making it an ideal Ingress controller for Kubernetes on Azure.

In this article, you learn how:

  • Create a Kubernetes cluster using AKS with Application Gateway as Ingress controller
  • Define a Kubernetes cluster
  • Create Application Gateway resource
  • Create a Kubernetes cluster
  • Test the availability of a Kubernetes cluster

Note

The example code in this article is located in the Microsoft Terraform GitHub repo.

1. Configure your environment

  • Azure subscription: If you don't have an Azure subscription, create a free account before you begin.
  • Azure service principal: The demo requires a service principal that can assign roles. If you already have a service principal that can assign roles, you can use that service principal. If you need to create a service principal, you have two options:

    You'll need the following service principal values for the demo code: appId, displayName, password, tenant.

  • Service principal object ID: Run the following command to get the object ID of the service principal: az ad sp list --display-name "<display_name>" --query "[].{\"Object ID\":objectId}" --output table

  • SSH key pair: Use one of the following articles:

  • Install Helm: Helm is the Kubernetes package manager.

  • Install GNU wget: Ensure you have access to wget by running wget at any command line without any parameters. You can install wget from the official GNU wget website.

2. Configure Azure storage to store Terraform state

Terraform tracks state locally via the terraform.tfstate file. This pattern works well in a single-person environment. However, in a more practical multi-person environment, you need to track state on the server using Azure storage. In this section, you learn to retrieve the necessary storage account information and create a storage container. The Terraform state information is then stored in that container.

  1. Use one of the following options to create an Azure storage account:

  2. Browse to the Azure portal.

  3. Under Azure services, select Storage accounts. (If the Storage accounts option isn't visible on the main page, select More services to locate the option.)

  4. On the Storage accounts page, On the Storage accounts page, select the storage account where Terraform will store the state information.

  5. On the Storage account page, in the left menu, in the Security + networking section, select Access keys.

    The Storage account page has a menu option to get the access keys.

  6. On the Access keys page, select Show keys to display the key values.

    The Access keys page has an option to display the key values.

  7. Locate the key1 key on the page and select the icon to its right to copy the key value to the clipboard.

    A handy icon button allows you to copy the key values to the clipboard.

  8. From a command line prompt, run az storage container create. This command creates a container in your Azure storage account. Replace the placeholders with the appropriate values for your Azure storage account.

    az storage container create -n tfstate \
       --account-name <storage_account_name> \
       --account-key <storage_account_key>
    
  9. When the command successfully completes, it displays a JSON block with a key of "created" and a value of true. You can also run az storage container list to verify the container was successfully created.

    az storage container list \
       --account-name <storage_account_name> \
       --account-key <storage_account_key>
    

3. Implement the Terraform code

  1. Create a directory in which to test the sample Terraform code and make it the current directory.

  2. Create a file named providers.tf and insert the following code.

    terraform {
    
      required_version = ">=0.12"
    
      required_providers {
        azurerm = {
          source  = "hashicorp/azurerm"
          version = "~>2.0"
        }
      }
      backend "azurerm" {
        resource_group_name  = "<storage_account_resource_group>"
        storage_account_name = "<storage_account_name>"
        container_name       = "tfstate"
        key                  = "codelab.microsoft.tfstate"
      }
    }
    
    provider "azurerm" {
      features {}
    }
    

    Key points:

    • Set resource_group_name to the resource group of the storage account.
    • Set storage_account_name to the storage account name.
  3. Create a file named main.tf and insert the following code:

    resource "random_pet" "rg-name" {
      prefix = var.resource_group_name_prefix
    }
    
    resource "azurerm_resource_group" "rg" {
      name     = random_pet.rg-name.id
      location = var.resource_group_location
    }
    
    # Locals block for hardcoded names
    locals {
      backend_address_pool_name      = "${azurerm_virtual_network.test.name}-beap"
      frontend_port_name             = "${azurerm_virtual_network.test.name}-feport"
      frontend_ip_configuration_name = "${azurerm_virtual_network.test.name}-feip"
      http_setting_name              = "${azurerm_virtual_network.test.name}-be-htst"
      listener_name                  = "${azurerm_virtual_network.test.name}-httplstn"
      request_routing_rule_name      = "${azurerm_virtual_network.test.name}-rqrt"
      app_gateway_subnet_name        = "appgwsubnet"
    }
    
    # User Assigned Identities 
    resource "azurerm_user_assigned_identity" "testIdentity" {
      resource_group_name = azurerm_resource_group.rg.name
      location            = azurerm_resource_group.rg.location
    
      name = "identity1"
    
      tags = var.tags
    }
    
    resource "azurerm_virtual_network" "test" {
      name                = var.virtual_network_name
      location            = azurerm_resource_group.rg.location
      resource_group_name = azurerm_resource_group.rg.name
      address_space       = [var.virtual_network_address_prefix]
    
      subnet {
        name           = var.aks_subnet_name
        address_prefix = var.aks_subnet_address_prefix
      }
    
      subnet {
        name           = "appgwsubnet"
        address_prefix = var.app_gateway_subnet_address_prefix
      }
    
      tags = var.tags
    }
    
    data "azurerm_subnet" "kubesubnet" {
      name                 = var.aks_subnet_name
      virtual_network_name = azurerm_virtual_network.test.name
      resource_group_name  = azurerm_resource_group.rg.name
      depends_on           = [azurerm_virtual_network.test]
    }
    
    data "azurerm_subnet" "appgwsubnet" {
      name                 = "appgwsubnet"
      virtual_network_name = azurerm_virtual_network.test.name
      resource_group_name  = azurerm_resource_group.rg.name
      depends_on           = [azurerm_virtual_network.test]
    }
    
    # Public Ip 
    resource "azurerm_public_ip" "test" {
      name                = "publicIp1"
      location            = azurerm_resource_group.rg.location
      resource_group_name = azurerm_resource_group.rg.name
      allocation_method   = "Static"
      sku                 = "Standard"
    
      tags = var.tags
    }
    
    resource "azurerm_application_gateway" "network" {
      name                = var.app_gateway_name
      resource_group_name = azurerm_resource_group.rg.name
      location            = azurerm_resource_group.rg.location
    
      sku {
        name     = var.app_gateway_sku
        tier     = "Standard_v2"
        capacity = 2
      }
    
      gateway_ip_configuration {
        name      = "appGatewayIpConfig"
        subnet_id = data.azurerm_subnet.appgwsubnet.id
      }
    
      frontend_port {
        name = local.frontend_port_name
        port = 80
      }
    
      frontend_port {
        name = "httpsPort"
        port = 443
      }
    
      frontend_ip_configuration {
        name                 = local.frontend_ip_configuration_name
        public_ip_address_id = azurerm_public_ip.test.id
      }
    
      backend_address_pool {
        name = local.backend_address_pool_name
      }
    
      backend_http_settings {
        name                  = local.http_setting_name
        cookie_based_affinity = "Disabled"
        port                  = 80
        protocol              = "Http"
        request_timeout       = 1
      }
    
      http_listener {
        name                           = local.listener_name
        frontend_ip_configuration_name = local.frontend_ip_configuration_name
        frontend_port_name             = local.frontend_port_name
        protocol                       = "Http"
      }
    
      request_routing_rule {
        name                       = local.request_routing_rule_name
        rule_type                  = "Basic"
        http_listener_name         = local.listener_name
        backend_address_pool_name  = local.backend_address_pool_name
        backend_http_settings_name = local.http_setting_name
      }
    
      tags = var.tags
    
      depends_on = [azurerm_virtual_network.test, azurerm_public_ip.test]
    }
    
    resource "azurerm_role_assignment" "ra1" {
      scope                = data.azurerm_subnet.kubesubnet.id
      role_definition_name = "Network Contributor"
      principal_id         = var.aks_service_principal_object_id
    
      depends_on = [azurerm_virtual_network.test]
    }
    
    resource "azurerm_role_assignment" "ra2" {
      scope                = azurerm_user_assigned_identity.testIdentity.id
      role_definition_name = "Managed Identity Operator"
      principal_id         = var.aks_service_principal_object_id
      depends_on           = [azurerm_user_assigned_identity.testIdentity]
    }
    
    resource "azurerm_role_assignment" "ra3" {
      scope                = azurerm_application_gateway.network.id
      role_definition_name = "Contributor"
      principal_id         = azurerm_user_assigned_identity.testIdentity.principal_id
      depends_on           = [azurerm_user_assigned_identity.testIdentity, azurerm_application_gateway.network]
    }
    
    resource "azurerm_role_assignment" "ra4" {
      scope                = azurerm_resource_group.rg.id
      role_definition_name = "Reader"
      principal_id         = azurerm_user_assigned_identity.testIdentity.principal_id
      depends_on           = [azurerm_user_assigned_identity.testIdentity, azurerm_application_gateway.network]
    }
    
    resource "azurerm_kubernetes_cluster" "k8s" {
      name       = var.aks_name
      location   = azurerm_resource_group.rg.location
      dns_prefix = var.aks_dns_prefix
    
      resource_group_name = azurerm_resource_group.rg.name
    
      http_application_routing_enabled = false
    
      linux_profile {
        admin_username = var.vm_user_name
    
        ssh_key {
          key_data = file(var.public_ssh_key_path)
        }
      }
    
      default_node_pool {
        name            = "agentpool"
        node_count      = var.aks_agent_count
        vm_size         = var.aks_agent_vm_size
        os_disk_size_gb = var.aks_agent_os_disk_size
        vnet_subnet_id  = data.azurerm_subnet.kubesubnet.id
      }
    
      service_principal {
        client_id     = var.aks_service_principal_app_id
        client_secret = var.aks_service_principal_client_secret
      }
    
      network_profile {
        network_plugin     = "azure"
        dns_service_ip     = var.aks_dns_service_ip
        docker_bridge_cidr = var.aks_docker_bridge_cidr
        service_cidr       = var.aks_service_cidr
      }
    
      role_based_access_control {
        enabled = var.aks_enable_rbac
      }
    
      depends_on = [azurerm_virtual_network.test, azurerm_application_gateway.network]
      tags       = var.tags
    }
    
  4. Create a file named variables.tf and insert the following code:

    variable "resource_group_name_prefix" {
      default     = "rg"
      description = "Prefix of the resource group name that's combined with a random ID so name is unique in your Azure subscription."
    }
    
    variable "resource_group_location" {
      default     = "eastus"
      description = "Location of the resource group."
    }
    
    variable "aks_service_principal_app_id" {
      description = "Application ID/Client ID  of the service principal. Used by AKS to manage AKS related resources on Azure like vms, subnets."
    }
    
    variable "aks_service_principal_client_secret" {
      description = "Secret of the service principal. Used by AKS to manage Azure."
    }
    
    variable "aks_service_principal_object_id" {
      description = "Object ID of the service principal."
    }
    
    variable "virtual_network_name" {
      description = "Virtual network name"
      default     = "aksVirtualNetwork"
    }
    
    variable "virtual_network_address_prefix" {
      description = "VNET address prefix"
      default     = "192.168.0.0/16"
    }
    
    variable "aks_subnet_name" {
      description = "Subnet Name."
      default     = "kubesubnet"
    }
    
    variable "aks_subnet_address_prefix" {
      description = "Subnet address prefix."
      default     = "192.168.0.0/24"
    }
    
    variable "app_gateway_subnet_address_prefix" {
      description = "Subnet server IP address."
      default     = "192.168.1.0/24"
    }
    
    variable "app_gateway_name" {
      description = "Name of the Application Gateway"
      default     = "ApplicationGateway1"
    }
    
    variable "app_gateway_sku" {
      description = "Name of the Application Gateway SKU"
      default     = "Standard_v2"
    }
    
    variable "app_gateway_tier" {
      description = "Tier of the Application Gateway tier"
      default     = "Standard_v2"
    }
    
    variable "aks_name" {
      description = "AKS cluster name"
      default     = "aks-cluster1"
    }
    variable "aks_dns_prefix" {
      description = "Optional DNS prefix to use with hosted Kubernetes API server FQDN."
      default     = "aks"
    }
    
    variable "aks_agent_os_disk_size" {
      description = "Disk size (in GB) to provision for each of the agent pool nodes. This value ranges from 0 to 1023. Specifying 0 applies the default disk size for that agentVMSize."
      default     = 40
    }
    
    variable "aks_agent_count" {
      description = "The number of agent nodes for the cluster."
      default     = 3
    }
    
    variable "aks_agent_vm_size" {
      description = "VM size"
      default     = "Standard_D3_v2"
    }
    
    variable "kubernetes_version" {
      description = "Kubernetes version"
      default     = "1.11.5"
    }
    
    variable "aks_service_cidr" {
      description = "CIDR notation IP range from which to assign service cluster IPs"
      default     = "10.0.0.0/16"
    }
    
    variable "aks_dns_service_ip" {
      description = "DNS server IP address"
      default     = "10.0.0.10"
    }
    
    variable "aks_docker_bridge_cidr" {
      description = "CIDR notation IP for Docker bridge."
      default     = "172.17.0.1/16"
    }
    
    variable "aks_enable_rbac" {
      description = "Enable RBAC on the AKS cluster. Defaults to false."
      default     = "false"
    }
    
    variable "vm_user_name" {
      description = "User name for the VM"
      default     = "vmuser1"
    }
    
    variable "public_ssh_key_path" {
      description = "Public key path for SSH."
      default     = "~/.ssh/id_rsa.pub"
    }
    
    variable "tags" {
      type = map(string)
    
      default = {
        source = "terraform"
      }
    }
    
  5. Create a file named output.tf and insert the following code.

    output "resource_group_name" {
      value = azurerm_resource_group.rg.name
    }
    
    output "client_key" {
      value = azurerm_kubernetes_cluster.k8s.kube_config.0.client_key
    }
    
    output "client_certificate" {
      value = azurerm_kubernetes_cluster.k8s.kube_config.0.client_certificate
    }
    
    output "cluster_ca_certificate" {
      value = azurerm_kubernetes_cluster.k8s.kube_config.0.cluster_ca_certificate
    }
    
    output "cluster_username" {
      value = azurerm_kubernetes_cluster.k8s.kube_config.0.username
    }
    
    output "cluster_password" {
      value = azurerm_kubernetes_cluster.k8s.kube_config.0.password
    }
    
    output "kube_config" {
      value     = azurerm_kubernetes_cluster.k8s.kube_config_raw
      sensitive = true
    }
    
    output "host" {
      value = azurerm_kubernetes_cluster.k8s.kube_config.0.host
    }
    
    output "identity_resource_id" {
      value = azurerm_user_assigned_identity.testIdentity.id
    }
    
    output "identity_client_id" {
      value = azurerm_user_assigned_identity.testIdentity.client_id
    }
    
    output "application_ip_address" {
      value = azurerm_public_ip.test.ip_address
    }
    
  6. Create a file named terraform.tfvars and insert the following code.

    aks_service_principal_app_id = "<service_principal_app_id>"
    
    aks_service_principal_client_secret = "<service_principal_password>"
    
    aks_service_principal_object_id = "<service_principal_object_id>"
    

    Key points:

    • Set aks_service_principal_app_id to the service principal appId value.
    • Set aks_service_principal_client_secret to the service principal password value.
    • Set aks_service_principal_object_id to the service principal object ID. (The Azure CLI command for obtaining this value is in the Configure your environment section.)

4. Initialize Terraform

Run terraform init to initialize the Terraform deployment. This command downloads the Azure modules required to manage your Azure resources.

terraform init

5. Create a Terraform execution plan

Run terraform plan to create an execution plan.

terraform plan -out main.tfplan

Key points:

  • The terraform plan command creates an execution plan, but doesn't execute it. Instead, it determines what actions are necessary to create the configuration specified in your configuration files. This pattern allows you to verify whether the execution plan matches your expectations before making any changes to actual resources.
  • The optional -out parameter allows you to specify an output file for the plan. Using the -out parameter ensures that the plan you reviewed is exactly what is applied.
  • To read more about persisting execution plans and security, see the security warning section.

6. Apply a Terraform execution plan

Run terraform apply to apply the execution plan to your cloud infrastructure.

terraform apply main.tfplan

Key points:

  • The terraform apply command above assumes you previously ran terraform plan -out main.tfplan.
  • If you specified a different filename for the -out parameter, use that same filename in the call to terraform apply.
  • If you didn't use the -out parameter, call terraform apply without any parameters.

7. Verify the results: Test the Kubernetes cluster

The Kubernetes tools can be used to verify the newly created cluster.

  1. Run az aks get-credentials to get the Kubernetes configuration and access credentials from Azure.

    az aks get-credentials --name <aks_cluster_name>  \
       --resource-group <resource_group_name> \
       --overwrite-existing
    

    Key points:

    • Replace the <aks_cluster_name> placeholder with the aks_name block's default value (from the variables.tf file).
    • Replace the <resource_group_name> placeholder with the randomly generated resource group name. Get the resource group name by running echo "$(terraform output resource_group_name)".
  2. Verify the health of the cluster.

    kubectl get nodes
    

    Key points:

    • The details of your worker nodes are displayed with a status of Ready.

    The kubectl tool allows you to verify the health of your Kubernetes cluster

8. Install Azure AD Pod Identity

Azure Active Directory Pod Identity provides token-based access to Azure Resource Manager.

Azure AD Pod Identity adds the following components to your Kubernetes cluster:

To install Azure AD Pod Identity to your cluster, you need to know if RBAC is enabled or disabled. RBAC is disabled by default for this demo. Enabling or disabling RBAC is done in the variables.tf file via the aks_enable_rbac block's default value.

  • If RBAC is enabled, run the following command:

    kubectl create -f https://raw.githubusercontent.com/Azure/aad-pod-identity/master/deploy/infra/deployment-rbac.yaml
    
  • If RBAC is disabled, run the following command:

    kubectl create -f https://raw.githubusercontent.com/Azure/aad-pod-identity/master/deploy/infra/deployment.yaml
    

9. Install Helm

Use Helm to install the application-gateway-kubernetes-ingress package:

  1. Run the following helm commands to add the AGIC Helm repo.

    helm repo add application-gateway-kubernetes-ingress https://appgwingress.blob.core.windows.net/ingress-azure-helm-package/
    
  2. Update the AGIC Helm repo.

    helm repo update
    

10. Install AGIC Helm Chart

  1. Download helm-config.yaml to configure AGIC. (If you don't have access to wget, see the Configure your environment section.)

    wget https://raw.githubusercontent.com/Azure/application-gateway-kubernetes-ingress/master/docs/examples/sample-helm-config.yaml -O helm-config.yaml
    
  2. Open the helm-config.yaml file in a text editor.

  3. Enter values for the top level keys.

  4. Enter values for the appgw block.

    • appgw.subscriptionId: Specify the Azure subscription ID used to create the App Gateway.
    • appgw.resourceGroup: Specify the randomly generated resource group name. Get the resource group name by running echo "$(terraform output resource_group_name)"
    • appgw.name: Specify the name of the Application Gateway. This value is set in the variables.tf file via the app_gateway_name block's default value.
    • appgw.shared: This boolean flag defaults to false. Set it to true if you need a Shared App Gateway.
  5. Enter values for the kubernetes block.

    • kubernetes.watchNamespace: Specify the name space, which AGIC should watch. The namespace can be a single string value, or a comma-separated list of namespaces. Leaving this variable commented out, or setting it to a blank or an empty string results in the Ingress controller observing all accessible namespaces.
  6. Enter values for the armAuth block.

    • If you specify armAuth.type as aadPodIdentity:

      • armAuth.identityResourceID: Get the identity resource ID by running echo "$(terraform output identity_resource_id)".
      • armAuth.identityClientId: Get the identity client ID by running echo "$(terraform output identity_client_id)".
    • If you specify armAuth.type as servicePrincipal, see Using a service principal.

  7. Install the Application Gateway Ingress controller package:

    helm install -f helm-config.yaml application-gateway-kubernetes-ingress/ingress-azure --generate-name
    
  8. To get the key values from your identity, you can run az identity show .

    az identity show -g <resource_group_name> -n <identity_name>
    

    Key points:

    • Replace the <resource_group_name> placeholder with the randomly generated resource group name. Get the resource group name by running echo "$(terraform output resource_group_name)".
    • Replace the <identity_name> placeholder with the identity name for this demo. The identity name defaults to identity1 in the main.tf file.
    • All identities for a given subscription can be by running az identity list.

11. Install a sample app

Once you have the App Gateway, AKS, and AGIC installed, install a sample app.

  1. Use the curl command to download the YAML file:

    curl https://raw.githubusercontent.com/Azure/application-gateway-kubernetes-ingress/master/docs/examples/aspnetapp.yaml -o aspnetapp.yaml
    
  2. Apply the YAML file:

    kubectl apply -f aspnetapp.yaml
    

12. Verify the results: Test the sample app

  1. Run the following Terraform command to get the app's IP address.

    echo "$(terraform output application_ip_address)"
    
  2. Using a browser, go to the IP address indicated in the previous step.

    Running the sample app.

13. Clean up resources

Delete App Gateway, AKS, and AGIC resources

When you no longer need the resources created via Terraform, do the following steps:

  1. Run terraform plan and specify the destroy flag.

    terraform plan -destroy -out main.destroy.tfplan
    

    Key points:

    • The terraform plan command creates an execution plan, but doesn't execute it. Instead, it determines what actions are necessary to create the configuration specified in your configuration files. This pattern allows you to verify whether the execution plan matches your expectations before making any changes to actual resources.
    • The optional -out parameter allows you to specify an output file for the plan. Using the -out parameter ensures that the plan you reviewed is exactly what is applied.
    • To read more about persisting execution plans and security, see the security warning section.
  2. Run terraform apply to apply the execution plan.

    terraform apply main.destroy.tfplan
    

Delete storage account

Caution

Only delete the resource group containing storage account you used in this demo if you're not using either for anything else.

Run az group delete to delete the resource group (and its storage account you used in this demo).

az group delete --name <storage_resource_group_name> --yes

Key points:

  • Replace the storage_resource_group_name placeholder with the resource_group_name value in the providers.tf file.

Delete service principal

Caution

Only delete the service principal you used in this demo if you're not using it for anything else.

az ad sp delete --id <service_principal_object_id>

Troubleshoot Terraform on Azure

If you receive a "403 error" when applying the Terraform execution plan during the role assignment, it usually means your service principal role doesn't include permission to assign roles in Azure RBAC. For more information about the built-in roles, see Azure built-in roles. The following options will enable you to resolve the error:

  • Create the service principal with the "Owner" role. As a recommended practice, you should grant the least privilege needed to perform a given job. Therefore, only use the "Owner" role if the service principal is meant to be used in that capacity.
  • Create a custom role based on the role you want - such as Contributor. Depending on the base role you use, either add the Microsoft.Authorization/*/Write action to the Actions block or remove it from the NotActions block. For more information on custom roles, see Azure custom roles.

Troubleshoot common problems when using Terraform on Azure

Next steps