Edit

Azure Kubernetes Service (AKS) ingress with the application routing add-on (preview)

  • Article

The application routing add-on configures an ingress controller in your Azure Kubernetes Service (AKS) cluster with SSL termination through certificates stored in Azure Key Vault. It can optionally integrate with Open Service Mesh (OSM) for end-to-end encryption of inter-cluster communication using mutual TLS (mTLS). When you deploy ingresses, the add-on creates publicly accessible DNS names for endpoints on an Azure DNS zone.

Important

AKS preview features are available on a self-service, opt-in basis. Previews are provided "as is" and "as available," and they're excluded from the service-level agreements and limited warranty. AKS previews are partially covered by customer support on a best-effort basis. As such, these features aren't meant for production use. For more information, see the following support articles:

Application routing add-on overview

The application routing add-on deploys the following components:

  • nginx ingress controller: This ingress controller is exposed to the internet.
  • external-dns controller: This controller watches for Kubernetes ingress resources and creates DNS A records in the cluster-specific DNS zone. This is only deployed when you pass in the --dns-zone-resource-id argument.

Prerequisites

  • An Azure subscription. If you don't have an Azure subscription, you can create a free account.
  • Azure CLI version 2.47.0 or later installed and configured. Run az --version to find the version. If you need to install or upgrade, see Install Azure CLI.
  • An Azure Key Vault to store certificates.
  • The aks-preview Azure CLI extension version 0.5.137 or later installed. If you need to install or update, see Install or update the aks-preview extension.
  • Optionally, a DNS solution, such as Azure DNS.

Install or update the aks-preview Azure CLI extension

  • Install the aks-preview Azure CLI extension using the az extension add command.

    az extension add --name aks-preview

  • If you need to update the extension version, you can do this using the az extension update command.

    az extension update --name aks-preview

Create and export a self-signed SSL certificate

Note

If you already have an SSL certificate, you can skip this step.

  1. Create a self-signed SSL certificate to use with the ingress using the openssl req command. Make sure you replace <Hostname> with the DNS name you're using.

    openssl req -new -x509 -nodes -out aks-ingress-tls.crt -keyout aks-ingress-tls.key -subj "/CN=<Hostname>" -addext "subjectAltName=DNS:<Hostname>"

  2. Export the SSL certificate and skip the password prompt using the openssl pkcs12 -export command.

    openssl pkcs12 -export -in aks-ingress-tls.crt -inkey aks-ingress-tls.key -out aks-ingress-tls.pfx

Create an Azure Key Vault to store the certificate

Note

If you already have an Azure Key Vault, you can skip this step.

  • Create an Azure Key Vault using the az keyvault create command.

    az keyvault create -g <ResourceGroupName> -l <Location> -n <KeyVaultName>

Import certificate into Azure Key Vault

  • Import the SSL certificate into Azure Key Vault using the az keyvault certificate import command. If your certificate is password protected, you can pass the password through the --password flag.

    az keyvault certificate import --vault-name <KeyVaultName> -n <KeyVaultCertificateName> -f aks-ingress-tls.pfx [--password <certificate password if specified>]

Create an Azure DNS zone

Note

If you want the add-on to automatically manage creating host names via Azure DNS, you need to create an Azure DNS zone if you don't have one already.

  • Create an Azure DNS zone using the az network dns zone create command.

    az network dns zone create -g <ResourceGroupName> -n <ZoneName>

Enable application routing using Azure CLI

The following extra add-on is required:

  • azure-keyvault-secrets-provider: The Secret Store CSI provider for Azure Key Vault is required to retrieve the certificates from Azure Key Vault.

Important

To enable the add-on to reload certificates from Azure Key Vault when they change, you should to enable the secret autorotation feature of the Secret Store CSI driver with the --enable-secret-rotation argument. When the autorotation is enabled, the driver updates the pod mount and the Kubernetes secret by polling for changes periodically, based on the rotation poll interval you can define. The default rotation poll interval is two minutes.

Enable application routing on a new cluster

  • Enable application routing on a new AKS cluster using the az aks create command and the --enable-addons parameter with the following add-ons:

    az aks create -g <ResourceGroupName> -n <ClusterName> -l <Location> --enable-addons azure-keyvault-secrets-provider,web_application_routing --generate-ssh-keys --enable-secret-rotation

Enable application routing on an existing cluster

  • Enable application routing on an existing cluster using the az aks enable-addons command and the --addons parameter with the following add-ons:

    az aks enable-addons -g <ResourceGroupName> -n <ClusterName> --addons azure-keyvault-secrets-provider,web_application_routing --enable-secret-rotation

Retrieve the add-on's managed identity object ID

You use the managed identity in the next steps to grant permissions to manage the Azure DNS zone and retrieve secrets and certificates from the Azure Key Vault.

  • Get the add-on's managed identity object ID using the az aks show command and setting the output to a variable named MANAGEDIDENTITY_OBJECTID.

    # Provide values for your environment
RGNAME=<ResourceGroupName>
CLUSTERNAME=<ClusterName>
MANAGEDIDENTITY_OBJECTID=$(az aks show -g ${RGNAME} -n ${CLUSTERNAME} --query ingressProfile.webAppRouting.identity.objectId -o tsv)

Configure the add-on to use Azure DNS to manage DNS zones

Note

If you plan to use Azure DNS, you need to update the add-on to pass in the --dns-zone-resource-id.

  1. Retrieve the resource ID for the DNS zone using the az network dns zone show command and setting the output to a variable named ZONEID.

    ZONEID=$(az network dns zone show -g <ResourceGroupName> -n <ZoneName> --query "id" --output tsv)

  2. Grant DNS Zone Contributor permissions on the DNS zone using the az role assignment create command.

    az role assignment create --role "DNS Zone Contributor" --assignee $MANAGEDIDENTITY_OBJECTID --scope $ZONEID

  3. Update the add-on to enable the integration with Azure DNS and install the external-dns controller using the az aks addon update command.

    az aks addon update -g <ResourceGroupName> -n <ClusterName> --addon web_application_routing --dns-zone-resource-id=$ZONEID

Grant the add-on permissions to retrieve certificates from Azure Key Vault

The application routing add-on creates a user-created managed identity in the cluster resource group. You need to grant permissions to the managed identity so it can retrieve SSL certificates from the Azure Key Vault.

Azure Key Vault offers two authorization systems: Azure role-based access control (Azure RBAC), which operates on the management plane, and the access policy model, which operates on both the management plane and the data plane. To find out which system your key vault is using, you can query the enableRbacAuthorization property.

az keyvault show --name <KeyVaultName> --query properties.enableRbacAuthorization

If Azure RBAC authorization is enabled for your key vault, you should configure permissions using Azure RBAC. Add the Key Vault Secrets User role assignment to the key vault.

KEYVAULTID=$(az keyvault show --name <KeyVaultName> --query "id" --output tsv)
az role assignment create --role "Key Vault Secrets User" --assignee $MANAGEDIDENTITY_OBJECTID --scope $KEYVAULTID

If Azure RBAC authorization is not enabled for your key vault, you should configure permissions using the access policy model. Grant GET permissions for the application routing add-on to retrieve certificates from Azure Key Vault using the az keyvault set-policy command.

az keyvault set-policy --name <KeyVaultName> --object-id $MANAGEDIDENTITY_OBJECTID --secret-permissions get --certificate-permissions get

Connect to your AKS cluster

To connect to the Kubernetes cluster from your local computer, you use kubectl, the Kubernetes command-line client. You can install it locally using the az aks install-cli command. If you use the Azure Cloud Shell, kubectl is already installed.

  • Configure kubectl to connect to your Kubernetes cluster using the az aks get-credentials command.

    az aks get-credentials -g <ResourceGroupName> -n <ClusterName>

Deploy an application

Application routing uses annotations on Kubernetes ingress objects to create the appropriate resources, create records on Azure DNS, and retrieve the SSL certificates from Azure Key Vault.

Create the application namespace

  • Create a namespace called hello-web-app-routing to run the example pods using the kubectl create namespace command.

    kubectl create namespace hello-web-app-routing

Create the deployment

  • Copy the following YAML into a new file named deployment.yaml and save the file to your local computer.

    apiVersion: apps/v1
kind: Deployment
metadata:
  name: aks-helloworld  
  namespace: hello-web-app-routing
spec:
  replicas: 1
  selector:
    matchLabels:
      app: aks-helloworld
  template:
    metadata:
      labels:
        app: aks-helloworld
    spec:
      containers:
      - name: aks-helloworld
        image: mcr.microsoft.com/azuredocs/aks-helloworld:v1
        ports:
        - containerPort: 80
        env:
        - name: TITLE
          value: "Welcome to Azure Kubernetes Service (AKS)"

Create the service

  • Copy the following YAML into a new file named service.yaml and save the file to your local computer.

    apiVersion: v1
kind: Service
metadata:
  name: aks-helloworld
  namespace: hello-web-app-routing
spec:
  type: ClusterIP
  ports:
  - port: 80
  selector:
    app: aks-helloworld

Create the ingress

The application routing add-on creates an ingress class on the cluster called webapprouting.kubernetes.azure.com. When you create an ingress object with this class, it activates the add-on.

  1. Get the certificate URI to use in the ingress from Azure Key Vault using the az keyvault certificate show command.

    az keyvault certificate show --vault-name <KeyVaultName> -n <KeyVaultCertificateName> --query "id" --output tsv

  2. Copy the following YAML into a new file named ingress.yaml and save the file to your local computer.

    Note

    Update <Hostname> with your DNS host name and <KeyVaultCertificateUri> with the ID returned from Azure Key Vault. The secretName key in the tls section defines the name of the secret that contains the certificate for this Ingress resource. This certificate will be presented in the browser when a client browses to the URL defined in the <Hostname> key. Make sure that the value of secretName is equal to keyvault- followed by the value of the Ingress resource name (from metadata.name). In the example YAML, secretName will need to be equal to keyvault-aks-helloworld.

    apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.azure.com/tls-cert-keyvault-uri: <KeyVaultCertificateUri>
  name: aks-helloworld
  namespace: hello-web-app-routing
spec:
  ingressClassName: webapprouting.kubernetes.azure.com
  rules:
 - host: <Hostname>
    http:
      paths:
      - backend:
          service:
            name: aks-helloworld
            port:
              number: 80
        path: /
        pathType: Prefix
  tls:
 - hosts:
    - <Hostname>
    secretName: keyvault-<Ingress resource name>

Create the resources on the cluster

  • Create the resources on the cluster using the kubectl apply command.

    kubectl apply -f deployment.yaml -n hello-web-app-routing
kubectl apply -f service.yaml -n hello-web-app-routing
kubectl apply -f ingress.yaml -n hello-web-app-routing

    The following example output shows the created resources:

    deployment.apps/aks-helloworld created
service/aks-helloworld created
ingress.networking.k8s.io/aks-helloworld created

Verify the managed ingress was created

  • Verify the managed ingress was created using the kubectl get ingress command.

    kubectl get ingress -n hello-web-app-routing

    The following example output shows the created managed ingress:

    NAME             CLASS                                HOSTS               ADDRESS       PORTS     AGE
aks-helloworld   webapprouting.kubernetes.azure.com   myapp.contoso.com   20.51.92.19   80, 443   4m

Access the endpoint over a DNS hostname

If you haven't configured Azure DNS integration, you need to configure your own DNS provider with an A record pointing to the ingress IP address and the host name you configured for the ingress, for example myapp.contoso.com.

Remove the application routing add-on

  1. Remove the associated namespace using the kubectl delete namespace command.

    kubectl delete namespace hello-web-app-routing

  2. Remove the application routing add-on from your cluster using the az aks disable-addons command.

    az aks disable-addons --addons web_application_routing --name myAKSCluster --resource-group myResourceGroup

When the application routing add-on is disabled, some Kubernetes resources may remain in the cluster. These resources include configMaps and secrets and are created in the app-routing-system namespace. You can remove these resources if you want.