Use dual-stack kubenet networking in Azure Kubernetes Service (AKS)

You can deploy your AKS clusters in a dual-stack mode when using kubenet networking and a dual-stack Azure virtual network. In this configuration, nodes receive both an IPv4 and IPv6 address from the Azure virtual network subnet. Pods receive both an IPv4 and IPv6 address from a logically different address space to the Azure virtual network subnet of the nodes. Network address translation (NAT) is then configured so that the pods can reach resources on the Azure virtual network. The source IP address of the traffic is NAT'd to the node's primary IP address of the same family (IPv4 to IPv4 and IPv6 to IPv6).

This article shows you how to use dual-stack networking with an AKS cluster. For more information on network options and considerations, see Network concepts for Kubernetes and AKS.

Limitations

  • Azure route tables have a hard limit of 400 routes per table.
    • Each node in a dual-stack cluster requires two routes, one for each IP address family, so dual-stack clusters are limited to 200 nodes.
  • In Azure Linux node pools, service objects are only supported with externalTrafficPolicy: Local.
  • Dual-stack networking is required for the Azure virtual network and the pod CIDR.
    • Single stack IPv6-only isn't supported for node or pod IP addresses. Services can be provisioned on IPv4 or IPv6.
  • The following features are not supported on dual-stack kubenet:

Prerequisites

  • All prerequisites from configure kubenet networking apply.
  • AKS dual-stack clusters require Kubernetes version v1.21.2 or greater. v1.22.2 or greater is recommended.
  • If using Azure Resource Manager templates, schema version 2021-10-01 is required.

Overview of dual-stack networking in Kubernetes

Kubernetes v1.23 brings stable upstream support for IPv4/IPv6 dual-stack clusters, including pod and service networking. Nodes and pods are always assigned both an IPv4 and an IPv6 address, while services can be dual-stack or single-stack on either address family.

AKS configures the required supporting services for dual-stack networking. This configuration includes:

  • If using a managed virtual network, a dual-stack virtual network configuration.
  • IPv4 and IPv6 node and pod addresses.
  • Outbound rules for both IPv4 and IPv6 traffic.
  • Load balancer setup for IPv4 and IPv6 services.

Note

When using Dualstack with an outbound type of user-defined routing, you can choose to have a default route for IPv6 depending on if you need your IPv6 traffic to reach the internet or not. If you don't have a default route for IPv6, a warning will surface when creating a cluster but will not prevent cluster creation.

Deploying a dual-stack cluster

The following attributes are provided to support dual-stack clusters:

  • --ip-families: Takes a comma-separated list of IP families to enable on the cluster.
    • Only ipv4 or ipv4,ipv6 are supported.
  • --pod-cidrs: Takes a comma-separated list of CIDR notation IP ranges to assign pod IPs from.
    • The count and order of ranges in this list must match the value provided to --ip-families.
    • If no values are supplied, the default value 10.244.0.0/16,fd12:3456:789a::/64 is used.
  • --service-cidrs: Takes a comma-separated list of CIDR notation IP ranges to assign service IPs from.
    • The count and order of ranges in this list must match the value provided to --ip-families.
    • If no values are supplied, the default value 10.0.0.0/16,fd12:3456:789a:1::/108 is used.
    • The IPv6 subnet assigned to --service-cidrs can be no larger than a /108.

Deploy a dual-stack AKS cluster

  1. Create an Azure resource group for the cluster using the az group create command.

    az group create --location <region> --name <resourceGroupName>
    
  2. Create a dual-stack AKS cluster using the az aks create command with the --ip-families parameter set to ipv4,ipv6.

    az aks create \
        --location <region> \
        --resource-group <resourceGroupName> \
        --name <clusterName> \
        --ip-families ipv4,ipv6 \
        --generate-ssh-keys
    
  3. Once the cluster is created, get the cluster admin credentials using the az aks get-credentials command.

    az aks get-credentials --resource-group <resourceGroupName> --name <clusterName>
    

Inspect the nodes to see both IP families

  • Once the cluster is provisioned, confirm the nodes are provisioned with dual-stack networking using the kubectl get nodes command.

    kubectl get nodes -o=custom-columns="NAME:.metadata.name,ADDRESSES:.status.addresses[?(@.type=='InternalIP')].address,PODCIDRS:.spec.podCIDRs[*]"
    

    The output from the kubectl get nodes command shows the nodes have addresses and pod IP assignment space from both IPv4 and IPv6.

    NAME                                ADDRESSES                           PODCIDRS
    aks-nodepool1-14508455-vmss000000   10.240.0.4,2001:1234:5678:9abc::4   10.244.0.0/24,fd12:3456:789a::/80
    aks-nodepool1-14508455-vmss000001   10.240.0.5,2001:1234:5678:9abc::5   10.244.1.0/24,fd12:3456:789a:0:1::/80
    aks-nodepool1-14508455-vmss000002   10.240.0.6,2001:1234:5678:9abc::6   10.244.2.0/24,fd12:3456:789a:0:2::/80
    

Create an example workload

Once the cluster has been created, you can deploy your workloads. This article walks you through an example workload deployment of an NGINX web server.

Deploy an NGINX web server

  1. Create an NGINX web server using the kubectl create deployment nginx command.

    kubectl create deployment nginx --image=nginx:latest --replicas=3
    
  2. View the pod resources using the kubectl get pods command.

    kubectl get pods -o custom-columns="NAME:.metadata.name,IPs:.status.podIPs[*].ip,NODE:.spec.nodeName,READY:.status.conditions[?(@.type=='Ready')].status"
    

    The output shows the pods have both IPv4 and IPv6 addresses. The pods don't show IP addresses until they're ready.

    NAME                     IPs                                NODE                                READY
    nginx-55649fd747-9cr7h   10.244.2.2,fd12:3456:789a:0:2::2   aks-nodepool1-14508455-vmss000002   True
    nginx-55649fd747-p5lr9   10.244.0.7,fd12:3456:789a::7       aks-nodepool1-14508455-vmss000000   True
    nginx-55649fd747-r2rqh   10.244.1.2,fd12:3456:789a:0:1::2   aks-nodepool1-14508455-vmss000001   True
    

Expose the workload via a LoadBalancer type service

Important

Starting in AKS v1.27, you can create a dual-stack LoadBalancer service which will be provisioned with 1 IPv4 public IP and 1 IPv6 public IP. However, in older versions, only the first IP address for a service will be provisioned to the load balancer, so a dual-stack service only receives a public IP for its first-listed IP family. To provide a dual-stack service for a single deployment, please create two services targeting the same selector, one for IPv4 and one for IPv6.

AKS starting from v1.27

  1. Expose the NGINX deployment using the kubectl expose deployment nginx command.

    kubectl expose deployment nginx --name=nginx --port=80 --type=LoadBalancer --overrides='{"spec":{"ipFamilyPolicy": "PreferDualStack", "ipFamilies": ["IPv4", "IPv6"]}}'
    

    You receive an output that shows the services have been exposed.

    service/nginx exposed
    
  2. Once the deployment is exposed and the LoadBalancer services are fully provisioned, get the IP addresses of the services using the kubectl get services command.

    kubectl get services
    
    NAME         TYPE           CLUSTER-IP               EXTERNAL-IP         PORT(S)        AGE
    nginx        LoadBalancer   10.0.223.73   2603:1030:20c:9::22d,4.156.88.133   80:30664/TCP   2m11s
    
    kubectl get services nginx -ojsonpath='{.spec.clusterIPs}'
    
    ["10.0.223.73","fd17:d93e:db1f:f771::54e"]
    
  3. Verify functionality via a command-line web request from an IPv6 capable host. Azure Cloud Shell isn't IPv6 capable.

    SERVICE_IP=$(kubectl get services nginx -o jsonpath='{.status.loadBalancer.ingress[1].ip}')
    curl -s "http://[${SERVICE_IP}]" | head -n5
    
    <!DOCTYPE html>
    <html>
    <head>
    <title>Welcome to nginx!</title>
    <style>
    

AKS older than v1.27

  1. Expose the NGINX deployment using the kubectl expose deployment nginx command.

    kubectl expose deployment nginx --name=nginx-ipv4 --port=80 --type=LoadBalancer'
    kubectl expose deployment nginx --name=nginx-ipv6 --port=80 --type=LoadBalancer --overrides='{"spec":{"ipFamilies": ["IPv6"]}}'
    

    You receive an output that shows the services have been exposed.

    service/nginx-ipv4 exposed
    service/nginx-ipv6 exposed
    
  2. Once the deployment is exposed and the LoadBalancer services are fully provisioned, get the IP addresses of the services using the kubectl get services command.

    kubectl get services
    
    NAME         TYPE           CLUSTER-IP               EXTERNAL-IP         PORT(S)        AGE
    nginx-ipv4   LoadBalancer   10.0.88.78               20.46.24.24         80:30652/TCP   97s
    nginx-ipv6   LoadBalancer   fd12:3456:789a:1::981a   2603:1030:8:5::2d   80:32002/TCP   63s
    
  3. Verify functionality via a command-line web request from an IPv6 capable host. Azure Cloud Shell isn't IPv6 capable.

    SERVICE_IP=$(kubectl get services nginx-ipv6 -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
    curl -s "http://[${SERVICE_IP}]" | head -n5
    
    <!DOCTYPE html>
    <html>
    <head>
    <title>Welcome to nginx!</title>
    <style>