Edit

Share via

Configure Istio ingress with the Kubernetes Gateway API for Azure Kubernetes Service (AKS) (preview)

Important

AKS preview features are available on a self-service, opt-in basis. Previews are provided "as is" and "as available," and they're excluded from the service-level agreements and limited warranty. AKS previews are partially covered by customer support on a best-effort basis. As such, these features aren't meant for production use. For more information, see the following support articles:

The Istio service mesh add-on supports both Istio's own ingress traffic management API and the Kubernetes Gateway API for ingress traffic management. You can use the Istio Gateway API automated deployment model or the manual deployment model. This article describes how to configure ingress traffic management for the Istio service mesh add-on using the Kubernetes Gateway API with the automated deployment model.

Limitations and considerations

Prerequisites

  • Enable the Managed Gateway API on your AKS cluster.
  • Install the Istio service mesh add-on revision asm-1-26 or higher. Follow the installation guide if you don't have the Istio service mesh add-on installed yet, or the upgrade guide if you're on a lower minor revision.

Set environment variables

Set the following environment variables to use throughout this article:

Variable Description
RESOURCE_GROUP The name of the resource group containing your AKS cluster.
CLUSTER_NAME The name of your AKS cluster.
LOCATION The Azure region where your AKS cluster is deployed.
KEY_VAULT_NAME The name of the Azure Key Vault resource to be created for storing TLS secrets. If you have an existing resource, use that name.

Deploy sample application

  • Deploy the sample httpbin application in the default namespace using the kubectl apply command.

    kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.26/samples/httpbin/httpbin.yaml

Create Kubernetes Gateway and HTTPRoute

The example manifest creates an external ingress load balancer service that's accessible from outside the cluster. You can add annotations to create an internal load balancer and customize other load balancer settings.

  • Deploy a Gateway API configuration in the default namespace with the gatewayClassName set to istio and an HTTPRoute that routes traffic to the httpbin service using the following manifest:

    kubectl apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: httpbin-gateway
spec:
  gatewayClassName: istio
  listeners:
  - name: http
    port: 80
    protocol: HTTP
    allowedRoutes:
      namespaces:
        from: Same
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: http
  namespace: default
spec:
  parentRefs:
  - name: httpbin-gateway
  hostnames: ["httpbin.example.com"]
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /get
    backendRefs:
    - name: httpbin
      port: 8000
EOF

    Note

    If you're performing a minor revision upgrade and have two Istio service mesh add-on revisions installed on your cluster simultaneously, the control plane for the higher minor revision takes ownership of the Gateways by default. You can add the istio.io/rev label to the Gateway to control which control plane revision owns it. If you add the revision label, make sure that you update it accordingly to the appropriate control plane revision before rolling back or completing the upgrade operation.

Verify resource creation

  • Verify the Deployment, Service, HorizontalPodAutoscaler, and PodDisruptionBudget resources were created using the following kubectl get commands:

    kubectl get deployment httpbin-gateway-istio
kubectl get service httpbin-gateway-istio
kubectl get hpa httpbin-gateway-istio
kubectl get pdb httpbin-gateway-istio

    Example output:

    # Deployment resource
NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
httpbin-gateway-istio   2/2     2            2           31m

# Service resource
NAME                    TYPE           CLUSTER-IP   EXTERNAL-IP      PORT(S)                        AGE
httpbin-gateway-istio   LoadBalancer   10.0.65.45   <external-ip>    15021:32053/TCP,80:31587/TCP   33m

# HPA resource
NAME                    REFERENCE                          TARGETS       MINPODS   MAXPODS   REPLICAS   AGE
httpbin-gateway-istio   Deployment/httpbin-gateway-istio   cpu: 3%/80%   2         5         3          34m

# PDB resource
NAME                    MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
httpbin-gateway-istio   1               N/A               2                     36m

Send request to sample application

  1. Try sending a curl request to the httpbin application. First, set the INGRESS_HOST environment variable:

    kubectl wait --for=condition=programmed gateways.gateway.networking.k8s.io httpbin-gateway
export INGRESS_HOST=$(kubectl get gateways.gateway.networking.k8s.io httpbin-gateway -ojsonpath='{.status.addresses[0].value}')

  2. Try sending an HTTP request to httpbin.

    curl -s -I -HHost:httpbin.example.com "http://$INGRESS_HOST/get"

    In the output, you should see an HTTP 200 response.

Secure Istio ingress traffic with the Kubernetes Gateway API

The Istio service mesh add-on supports syncing secrets from Azure Key Vault for securing Gateway API-based ingress traffic with Transport Layer Security (TLS) termination or Server Name Indication (SNI) passthrough. In the following sections, you sync secrets from Azure Key Vault onto your AKS cluster using the Azure Key Vault provider for Secrets Store Container Storage Interface (CSI) Driver add-on and terminate TLS at the ingress gateway.

Create client/server certificates and keys

  1. Create a root certificate and private key for signing the certificates for sample services:

    mkdir httpbin_certs
openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -subj '/O=example Inc./CN=example.com' -keyout httpbin_certs/example.com.key -out httpbin_certs/example.com.crt

  2. Generate a certificate and a private key for httpbin.example.com:

    openssl req -out httpbin_certs/httpbin.example.com.csr -newkey rsa:2048 -nodes -keyout httpbin_certs/httpbin.example.com.key -subj "/CN=httpbin.example.com/O=httpbin organization"
openssl x509 -req -sha256 -days 365 -CA httpbin_certs/example.com.crt -CAkey httpbin_certs/example.com.key -set_serial 0 -in httpbin_certs/httpbin.example.com.csr -out httpbin_certs/httpbin.example.com.crt

Set up Azure Key Vault and create secrets

  1. Create an Azure Key Vault instance to supply the certificate and key inputs to the Istio service mesh add-on using the az keyvault create command. If you already have an Azure Key Vault instance, you can skip this step.

    az keyvault create --name $KEY_VAULT_NAME --resource-group $RESOURCE_GROUP --location $LOCATION

  2. Enable the Azure Key Vault provider for Secrets Store (CSI) Driver add-on on your cluster using the az aks enable-addons command.

    az aks enable-addons --addons azure-keyvault-secrets-provider --resource-group $RESOURCE_GROUP --name $CLUSTER_NAME

  3. If your key vault uses Azure role-based access control (RBAC) for the permissions model, follow the instructions in Provide access to Azure Key Vault keys, certificates, and secrets with Azure role-based access control to assign an Azure role of Key Vault Secrets User for the add-on's user-assigned managed identity. Alternatively, if your key vault uses the vault access policy permissions model, authorize the user-assigned managed identity of the add-on to access Azure Key Vault resource using access policy using the az keyvault set-policy command.

    OBJECT_ID=$(az aks show --resource-group $RESOURCE_GROUP --name $CLUSTER_NAME --query 'addonProfiles.azureKeyvaultSecretsProvider.identity.objectId' -o tsv | tr -d '\r')
CLIENT_ID=$(az aks show --resource-group $RESOURCE_GROUP --name $CLUSTER_NAME --query 'addonProfiles.azureKeyvaultSecretsProvider.identity.clientId')
TENANT_ID=$(az keyvault show --resource-group $RESOURCE_GROUP --name $KEY_VAULT_NAME --query 'properties.tenantId')

az keyvault set-policy --name $KEY_VAULT_NAME --object-id $OBJECT_ID --secret-permissions get list

  4. Create secrets in Azure Key Vault using the certificates and keys using the following az keyvault secret set commands:

    az keyvault secret set --vault-name $KEY_VAULT_NAME --name test-httpbin-key --file httpbin_certs/httpbin.example.com.key
az keyvault secret set --vault-name $KEY_VAULT_NAME --name test-httpbin-crt --file httpbin_certs/httpbin.example.com.crt

Deploy SecretProviderClass and sample pod

  1. Deploy the SecretProviderClass to provide Azure Key Vault specific parameters to the CSI driver using the following manifest. In this example, test-httpbin-key and test-httpbin-crt are the names of the secret objects in Azure Key Vault.

    cat <<EOF | kubectl apply -f -
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: httpbin-credential-spc
spec:
  provider: azure
  secretObjects:
  - secretName: httpbin-credential
    type: kubernetes.io/tls
    data:
    - objectName: test-httpbin-key
      key: tls.key
    - objectName: test-httpbin-crt
      key: tls.crt
  parameters:
    useVMManagedIdentity: "true"
    userAssignedIdentityID: $CLIENT_ID 
    keyvaultName: $KEY_VAULT_NAME
    cloudName: ""
    objects:  |
      array:
        - |
          objectName: test-httpbin-key
          objectType: secret
          objectAlias: "test-httpbin-key"
        - |
          objectName: test-httpbin-crt
          objectType: secret
          objectAlias: "test-httpbin-crt"
    tenantId: $TENANT_ID
EOF

    Note

    Alternatively, to reference a certificate object type directly from Azure Key Vault, use the following manifest to deploy SecretProviderClass. In this example, test-httpbin-cert-pxf is the name of the certificate object in Azure Key Vault.

    cat <<EOF | kubectl apply -f -
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: httpbin-credential-spc
spec:
  provider: azure
  secretObjects:
  - secretName: httpbin-credential
    type: kubernetes.io/tls
    data:
    - objectName: test-httpbin-key
      key: tls.key
    - objectName: test-httpbin-crt
      key: tls.crt
  parameters:
    useVMManagedIdentity: "true"
    userAssignedIdentityID: $CLIENT_ID 
    keyvaultName: $KEY_VAULT_NAME
    cloudName: ""
    objects:  |
      array:
        - |
          objectName: test-httpbin-cert-pfx  #certificate object name from keyvault
          objectType: secret
          objectAlias: "test-httpbin-key"
        - |
          objectName: test-httpbin-cert-pfx #certificate object name from keyvault
          objectType: cert
          objectAlias: "test-httpbin-crt"
    tenantId: $TENANT_ID
EOF

  2. Deploy a sample pod using the following manifest. The Azure Key Vault provider for Secrets Store (CSI) Driver add-on requires a pod to reference the SecretProviderClass resource to ensure secrets sync from Azure Key Vault to the cluster.

    cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: secrets-store-sync-httpbin
spec:
  containers:
    - name: busybox
      image: mcr.microsoft.com/oss/busybox/busybox:1.33.1
      command:
        - "/bin/sleep"
        - "10"
      volumeMounts:
      - name: secrets-store01-inline
        mountPath: "/mnt/secrets-store"
        readOnly: true
  volumes:
    - name: secrets-store01-inline
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: "httpbin-credential-spc"
EOF

Verify TLS secret creation

  • Verify the httpbin-credential secret was created in the default namespace as defined in the SecretProviderClass resource using the kubectl describe secret command.

    kubectl describe secret/httpbin-credential

    Example output:

    Name:         httpbin-credential
Namespace:    default
Labels:       secrets-store.csi.k8s.io/managed=true
Annotations:  <none>

Type:  kubernetes.io/tls

Data
====
tls.crt:  1180 bytes
tls.key:  1675 bytes

Deploy TLS Gateway

  1. Create a Kubernetes Gateway that references the httpbin-credential secret under the TLS configuration using the following manifest:

    cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: httpbin-gateway
spec:
  gatewayClassName: istio
  listeners:
  - name: https
    hostname: "httpbin.example.com"
    port: 443
    protocol: HTTPS
    tls:
      mode: Terminate
      certificateRefs:
      - name: httpbin-credential
    allowedRoutes:
      namespaces:
        from: Selector
        selector:
          matchLabels:
            kubernetes.io/metadata.name: default
EOF

    Note

    In the gateway definition, tls.certificateRefs.name must match the secretName in SecretProviderClass resource.

  2. Create a corresponding HTTPRoute to configure ingress traffic routing to the httpbin service over HTTPS using the following manifest:

    cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: httpbin
spec:
  parentRefs:
  - name: httpbin-gateway
  hostnames: ["httpbin.example.com"]
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /status
    - path:
        type: PathPrefix
        value: /delay
    backendRefs:
    - name: httpbin
      port: 8000
EOF

  3. Get the ingress gateway's external IP address and secure port using the following commands:

    kubectl wait --for=condition=programmed gateways.gateway.networking.k8s.io httpbin-gateway
export INGRESS_HOST=$(kubectl get gateways.gateway.networking.k8s.io httpbin-gateway -o jsonpath='{.status.addresses[0].value}')
export SECURE_INGRESS_PORT=$(kubectl get gateways.gateway.networking.k8s.io httpbin-gateway -o jsonpath='{.spec.listeners[?(@.name=="https")].port}')

  4. Send an HTTPS request to access the httpbin service:

    curl -v -HHost:httpbin.example.com --resolve "httpbin.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST" \
--cacert httpbin_certs/example.com.crt "https://httpbin.example.com:$SECURE_INGRESS_PORT/status/418"

    The output should show the httpbin service return the 418 I’m a Teapot code.

    Note

    To configure HTTPS ingress access to an HTTPS service, update the TLS mode in the gateway definition to Passthrough. This configuration instructs the gateway to pass the ingress traffic as is, without terminating TLS.

Annotation customizations

You can add annotations under spec.infrastructure.annotations to configure load balancer settings for the Gateway. For instance, to create an internal load balancer attached to a specific subnet, you can create a Gateway with the following annotations:

spec:
  # ... existing spec content ...
  infrastructure:
    annotations: 
      service.beta.kubernetes.io/azure-load-balancer-internal: "true"
      service.beta.kubernetes.io/azure-load-balancer-internal-subnet: "my-subnet"

ConfigMap customizations

The Istio service mesh add-on supports customizations of the resources generated for the Gateways, including:

  • Service
  • Deployment
  • Horizontal Pod Autoscaler (HPA)
  • Pod Disruption Budget (PDB)

The default settings for these resources are set in the istio-gateway-class-defaults ConfigMap in the aks-istio-system namespace. This ConfigMap must have the gateway.istio.io/defaults-for-class label set to istio for the customizations to take effect for all Gateways with spec.gatewayClassName: istio. The GatewayClass-level ConfigMap is installed by default in the aks-istio-system namespace when the Managed Gateway API installation is enabled. It could take up to five minutes for the istio-gateway-class-defaults ConfigMap to get deployed after installing the Managed Gateway API CRDs.

kubectl get configmap istio-gateway-class-defaults -n aks-istio-system -o yaml

...
data:
  horizontalPodAutoscaler: |
    spec:
      minReplicas: 2
      maxReplicas: 5
  podDisruptionBudget: |
    spec:
      minAvailable: 1
...

You can modify these settings for all Istio Gateways at a GatewayClass level by updating the istio-gateway-class-defaults ConfigMap, or you can set them for individual Gateway resources. For both the GatewayClass-level and Gateway-level ConfigMaps, you must add fields to the allow list for the given resource. If there are customizations both for the GatewayClass and an individual Gateway, the Gateway-level configuration takes precedence.

Deployment customization allow list fields

Field path Description
metadata.labels Deployment labels
metadata.annotations Deployment annotations
spec.replicas Deployment replica count
spec.template.metadata.labels Pod labels
spec.template.metadata.annotations Pod annotations
spec.template.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms Node affinity
spec.template.spec.affinity.nodeAffinity.preferredDuringSchedulingIgnoredDuringExecution Node affinity
spec.template.spec.affinity.podAffinity.requiredDuringSchedulingIgnoredDuringExecution Pod affinity
spec.template.spec.affinity.podAffinity.preferredDuringSchedulingIgnoredDuringExecution Pod affinity
spec.template.spec.affinity.podAntiAffinity.requiredDuringSchedulingIgnoredDuringExecution Pod anti-affinity
spec.template.spec.affinity.podAntiAffinity.preferredDuringSchedulingIgnoredDuringExecution Pod anti-affinity
spec.template.spec.containers.resizePolicy Container resource utilization
spec.template.spec.containers.resources.limits Container resource utilization
spec.template.spec.containers.resources.requests Container resource utilization
spec.template.spec.containers.stdin Container debugging
spec.template.spec.containers.stdinOnce Container debugging
spec.template.spec.nodeSelector Pod scheduling
spec.template.spec.nodeName Pod scheduling
spec.template.spec.tolerations Pod scheduling
spec.template.spec.topologySpreadConstraints Pod scheduling

Service customization allow list fields

Field path Description
metadata.labels Service labels
metadata.annotations Service annotations
spec.type Service type
spec.loadBalancerSourceRanges Service load balancer settings
spec.loadBalancerClass Service load balancer settings
spec.externalTrafficPolicy Service traffic policy
spec.internalTrafficPolicy Service traffic policy

HorizontalPodAutoscaler (HPA) customization allow list fields

Field path Description
metadata.labels HPA labels
metadata.annotations HPA annotations
spec.behavior.scaleUp.stabilizationWindowSeconds HPA scale-up behavior
spec.behavior.scaleUp.selectPolicy HPA scale-up behavior
spec.behavior.scaleUp.policies HPA scale-up behavior
spec.behavior.scaleDown.stabilizationWindowSeconds HPA scale-down behavior
spec.behavior.scaleDown.selectPolicy HPA scale-down behavior
spec.behavior.scaleDown.policies HPA scale-down behavior
spec.metrics HPA scaling resource metrics
spec.minReplicas HPA minimum replica count. Must not be below 2.
spec.maxReplicas HPA maximum replica count

PodDisruptionBudget (PDB) customization allow list fields

Field path Description
metadata.labels PDB labels
metadata.annotations PDB annotations
spec.minAvailable PDB minimum availability
spec.unhealthyPodEvictionPolicy PDB eviction policy

Note

Modifying the PDB minimum availability and eviction policy can lead to potential errors during cluster/node upgrade and deletion operations. Follow the PDB troubleshooting guide to address UpgradeFailed errors due to PDB eviction failures.

Configure GatewayClass-level settings

  1. Update the GatewayClass-level ConfigMap in the aks-istio-system namespace using the kubectl edit configmap command:

    kubectl edit cm istio-gateway-class-defaults -n aks-istio-system

  2. Edit the resource settings in the data section as needed. For example, to update the HPA min/max replicas and add a label to the Deployment, modify the ConfigMap as follows:

    ...
data:
  deployment: |
    metadata:
      labels:
        test.azureservicemesh.io/deployment-config: "updated"
  horizontalPodAutoscaler: |
    spec:
      minReplicas: 3
      maxReplicas: 6
  podDisruptionBudget: |
    spec:
      minAvailable: 1
...

    Note

    Only one ConfigMap per GatewayClass is allowed.

  3. Now, you should see the HPA for httpbin-gateway that you created earlier get updated with the new min/max values. Verify the HPA settings using the kubectl get hpa command.

    kubectl get hpa httpbin-gateway-istio

    Example output:

    NAME                    REFERENCE                          TARGETS       MINPODS   MAXPODS   REPLICAS   AGE
httpbin-gateway-istio   Deployment/httpbin-gateway-istio   cpu: 3%/80%   3         6         3          36m

  4. Verify the Deployment is updated with the new label using the kubectl get deployment command.

    kubectl get deployment httpbin-gateway-istio -ojsonpath='{.metadata.labels.test\.azureservicemesh\.io\/deployment-config}'

    Example output:

    updated

Configure settings for a specific gateway

  1. Create a ConfigMap with resource customizations for the httpbin Gateway using the following manifest:

    kubectl apply -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: gw-options
data:
  horizontalPodAutoscaler: |
    spec:
      minReplicas: 2
      maxReplicas: 4
  deployment: |
    metadata:
      labels:
        test.azureservicemesh.io/deployment-config: "updated-per-gateway"
EOF

  2. Update the httpbin Gateway to reference the ConfigMap:

    spec:
  # ... existing spec content ...
  infrastructure:
    parametersRef:
      group: ""
      kind: ConfigMap
      name: gw-options

  3. Apply the update using the kubectl apply command.

    kubectl apply -f httpbin-gateway-updated.yaml

  4. Verify the HPA is updated with the new min/max values using the kubectl get hpa command. If you also configured the GatewayClass-level ConfigMap, the Gateway-level settings should take precedence.

    kubectl get hpa httpbin-gateway-istio

    Example output:

    NAME                    REFERENCE                          TARGETS       MINPODS   MAXPODS   REPLICAS   AGE
httpbin-gateway-istio   Deployment/httpbin-gateway-istio   cpu: 3%/80%   2         4         2          4h14m

  5. Inspect the Deployment labels to ensure that the test.azureservicemesh.io/deployment-config is updated to the new value using the kubectl get deployment command.

    kubectl get deployment httpbin-gateway-istio -ojsonpath='{.metadata.labels.test\.azureservicemesh\.io\/deployment-config}'

    Example output:

    updated-per-gateway

Clean up resources

If you no longer need the resources created in this article, you can delete them to avoid incurring any charges.

  1. Delete the Gateway and HTTPRoute resources using the following kubectl delete commands:

    kubectl delete gateways.gateway.networking.k8s.io httpbin-gateway
kubectl delete httproute httpbin

  2. If you created a ConfigMap to customize your Gateway resources, delete it using the kubectl delete configmap command.

    kubectl delete configmap gw-options

  3. If you created a SecretProviderClass and secret to use for TLS termination delete the resources using the following kubectl delete commands:

    kubectl delete secret httpbin-credential
kubectl delete pod secrets-store-sync-httpbin
kubectl delete secretproviderclass httpbin-credential-spc

Related content

  • Last updated on