Use dual-stack kubenet networking in Azure Kubernetes Service (AKS) (Preview)

AKS clusters can now be deployed in a dual-stack (using both IPv4 and IPv6 addresses) mode when using kubenet networking and a dual-stack Azure virtual network. In this configuration, nodes receive both an IPv4 and IPv6 address from the Azure virtual network subnet. Pods receive both an IPv4 and IPv6 address from a logically different address space to the Azure virtual network subnet of the nodes. Network address translation (NAT) is then configured so that the pods can reach resources on the Azure virtual network. The source IP address of the traffic is NAT'd to the node's primary IP address of the same family (IPv4 to IPv4 and IPv6 to IPv6).

This article shows you how to use dual-stack networking with an AKS cluster. For more information on network options and considerations, see Network concepts for Kubernetes and AKS.

Limitations

Note

Dual-stack kubenet networking is currently not available in sovereign clouds. This note will be removed when rollout is complete.

• Azure Route Tables have a hard limit of 400 routes per table. Because each node in a dual-stack cluster requires two routes, one for each IP address family, dual-stack clusters are limited to 200 nodes.
• During preview, service objects are only supported with externalTrafficPolicy: Local.
• Dual-stack networking is required for the Azure Virtual Network and the pod CIDR - single stack IPv6-only isn't supported for node or pod IP addresses. Services can be provisioned on IPv4 or IPv6.
• Features not supported on dual-stack kubenet include:
  • Azure network policies
  • Calico network policies
  • NAT Gateway
  • Virtual nodes add-on
  • Windows node pools

Prerequisites

• All prerequisites from configure kubenet networking apply.
• AKS dual-stack clusters require Kubernetes version v1.21.2 or greater. v1.22.2 or greater is recommended to take advantage of the out-of-tree cloud controller manager, which is the default on v1.22 and up.
• Azure CLI with the aks-preview extension 0.5.48 or newer.
• If using Azure Resource Manager templates, schema version 2021-10-01 is required.

Install the aks-preview Azure CLI extension

Important

AKS preview features are available on a self-service, opt-in basis. Previews are provided "as is" and "as available," and they're excluded from the service-level agreements and limited warranty. AKS previews are partially covered by customer support on a best-effort basis. As such, these features aren't meant for production use. For more information, see the following support articles:

• AKS support policies
• Azure support FAQ

To install the aks-preview extension, run the following command:

az extension add --name aks-preview

Run the following command to update to the latest version of the extension released:

az extension update --name aks-preview

Register the 'AKS-EnableDualStack' feature flag

Register the AKS-EnableDualStack feature flag by using the az feature register command, as shown in the following example:

az feature register --namespace "Microsoft.ContainerService" --name "AKS-EnableDualStack"

It takes a few minutes for the status to show Registered. Verify the registration status by using the az feature show command:

az feature show --namespace "Microsoft.ContainerService" --name "AKS-EnableDualStack"

When the status reflects Registered, refresh the registration of the Microsoft.ContainerService resource provider by using the az provider register command:

az provider register --namespace Microsoft.ContainerService

Overview of dual-stack networking in Kubernetes

Kubernetes v1.23 brings stable upstream support for IPv4/IPv6 dual-stack clusters, including pod and service networking. Nodes and pods are always assigned both an IPv4 and an IPv6 address, while services can be single-stack on either address family or dual-stack.

AKS configures the required supporting services for dual-stack networking. This configuration includes:

• Dual-stack virtual network configuration (if managed Virtual Network is used)
• IPv4 and IPv6 node and pod addresses
• Outbound rules for both IPv4 and IPv6 traffic
• Load balancer setup for IPv4 and IPv6 services

Deploying a dual-stack cluster

Three new attributes are provided to support dual-stack clusters:

• --ip-families - takes a comma-separated list of IP families to enable on the cluster.
  • Currently only ipv4 or ipv4,ipv6 are supported.
• --pod-cidrs - takes a comma-separated list of CIDR notation IP ranges to assign pod IPs from.
  • The count and order of ranges in this list must match the value provided to --ip-families.
  • If no values are supplied, the default values of 10.244.0.0/16,fd12:3456:789a::/64 will be used.
• --service-cidrs - takes a comma-separated list of CIDR notation IP ranges to assign service IPs from.
  • The count and order of ranges in this list must match the value provided to --ip-families.
  • If no values are supplied, the default values of 10.0.0.0/16,fd12:3456:789a:1::/108 will be used.
  • The IPv6 subnet assigned to --service-cidrs can be no larger than a /108.

Deploy the cluster

Azure CLI

Deploying a dual-stack cluster requires passing the --ip-families parameter with the parameter value of ipv4,ipv6 to indicate that a dual-stack cluster should be created.

1. First, create a resource group to create the cluster in:

   az group create -l <Region> -n <ResourceGroupName>
   

2. Then create the cluster itself:

   az aks create -l <Region> -g <ResourceGroupName> -n <ClusterName> --ip-families ipv4,ipv6
   

Azure Resource Manager

When using an Azure Resource Manager template to deploy, pass ["IPv4", "IPv6"] to the ipFamilies parameter to the networkProfile object. See the Azure Resource Manager template documentation for help with deploying this template, if needed.

{
  "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "clusterName": {
      "type": "string",
      "defaultValue": "aksdualstack"
    },
    "location": {
      "type": "string",
      "defaultValue": "[resourceGroup().location]"
    },
    "kubernetesVersion": {
      "type": "string",
      "defaultValue": "1.22.2"
    },
    "nodeCount": {
      "type": "int",
      "defaultValue": 3
    },
    "nodeSize": {
      "type": "string",
      "defaultValue": "Standard_B2ms"
    }
  },
  "resources": [
    {
      "type": "Microsoft.ContainerService/managedClusters",
      "apiVersion": "2021-10-01",
      "name": "[parameters('clusterName')]",
      "location": "[parameters('location')]",
      "identity": {
        "type": "SystemAssigned"
      },
      "properties": {
        "agentPoolProfiles": [
          {
            "name": "nodepool1",
            "count": "[parameters('nodeCount')]",
            "mode": "System",
            "vmSize": "[parameters('nodeSize')]"
          }
        ],
        "dnsPrefix": "[parameters('clusterName')]",
        "kubernetesVersion": "[parameters('kubernetesVersion')]",
        "networkProfile": {
          "ipFamilies": [
            "IPv4",
            "IPv6"
          ]
        }
      }
    }
  ]
}

Bicep

When using a Bicep template to deploy, pass ["IPv4", "IPv6"] to the ipFamilies parameter to the networkProfile object. See the Bicep template documentation for help with deploying this template, if needed.

param clusterName string = 'aksdualstack'
param location string = resourceGroup().location
param kubernetesVersion string = '1.22.2'
param nodeCount int = 3
param nodeSize string = 'Standard_B2ms'

resource aksCluster 'Microsoft.ContainerService/managedClusters@2021-10-01' = {
  name: clusterName
  location: location
  identity: {
    type: 'SystemAssigned'
  }
  properties: {
    agentPoolProfiles: [
      {
        name: 'nodepool1'
        count: nodeCount
        mode: 'System'
        vmSize: nodeSize
      }
    ]
    dnsPrefix: clusterName
    kubernetesVersion: kubernetesVersion
    networkProfile: {
      ipFamilies: [
        'IPv4'
        'IPv6'
      ]
    }
  }
}

Finally, after the cluster has been created, get the admin credentials:

az aks get-credentials -g <ResourceGroupName> -n <ClusterName> -a

Inspect the nodes to see both IP families

Once the cluster is provisioned, confirm that the nodes are provisioned with dual-stack networking:

kubectl get nodes -o=custom-columns="NAME:.metadata.name,ADDRESSES:.status.addresses[?(@.type=='InternalIP')].address,PODCIDRS:.spec.podCIDRs[*]"

The output from the kubectl get nodes command will show that the nodes have addresses and pod IP assignment space from both IPv4 and IPv6.

NAME                                ADDRESSES                           PODCIDRS
aks-nodepool1-14508455-vmss000000   10.240.0.4,2001:1234:5678:9abc::4   10.244.0.0/24,fd12:3456:789a::/80
aks-nodepool1-14508455-vmss000001   10.240.0.5,2001:1234:5678:9abc::5   10.244.1.0/24,fd12:3456:789a:0:1::/80
aks-nodepool1-14508455-vmss000002   10.240.0.6,2001:1234:5678:9abc::6   10.244.2.0/24,fd12:3456:789a:0:2::/80

Create an example workload

Deploy an nginx web server

Once the cluster has been created, workloads can be deployed as usual. A simple example webserver can be created using the following command:

kubectl create

kubectl create deployment nginx --image=nginx:latest --replicas=3

YAML

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx:latest
        name: nginx

Using the following kubectl get pods command will show that the pods have both IPv4 and IPv6 addresses (note that the pods will not show IP addresses until they are ready):

kubectl get pods -o custom-columns="NAME:.metadata.name,IPs:.status.podIPs[*].ip,NODE:.spec.nodeName,READY:.status.conditions[?(@.type=='Ready')].status"

NAME                     IPs                                NODE                                READY
nginx-55649fd747-9cr7h   10.244.2.2,fd12:3456:789a:0:2::2   aks-nodepool1-14508455-vmss000002   True
nginx-55649fd747-p5lr9   10.244.0.7,fd12:3456:789a::7       aks-nodepool1-14508455-vmss000000   True
nginx-55649fd747-r2rqh   10.244.1.2,fd12:3456:789a:0:1::2   aks-nodepool1-14508455-vmss000001   True

Expose the workload via a LoadBalancer-type service

Important

There are currently two limitations pertaining to IPv6 services in AKS. These are both preview limitations and work is underway to remove them.

• Azure Load Balancer sends health probes to IPv6 destinations from a link-local address. This traffic cannot be routed to a pod and thus traffic flowing to IPv6 services deployed with externalTrafficPolicy: Cluster will fail. During preview, IPv6 services MUST be deployed with externalTrafficPolicy: Local, which causes kube-proxy to respond to the probe on the node, in order to function.
• Only the first IP address for a service will be provisioned to the load balancer, so a dual-stack service will only receive a public IP for its first listed IP family. In order to provide a dual-stack service for a single deployment, please create two services targeting the same selector, one for IPv4 and one for IPv6.

IPv6 services in Kubernetes can be exposed publicly similarly to an IPv4 service.

kubectl expose

kubectl expose deployment nginx --name=nginx-ipv4 --port=80 --type=LoadBalancer --overrides='{"spec":{"externalTrafficPolicy":"Local"}}'
kubectl expose deployment nginx --name=nginx-ipv6 --port=80 --type=LoadBalancer --overrides='{"spec":{"externalTrafficPolicy":"Local", "ipFamilies": ["IPv6"]}}'

service/nginx-ipv4 exposed
service/nginx-ipv6 exposed

YAML

---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: nginx
  name: nginx-ipv4
spec:
  externalTrafficPolicy: Local
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: nginx
  name: nginx-ipv6
spec:
  externalTrafficPolicy: Local
  ipFamilies:
  - IPv6
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer

Once the deployment has been exposed and the LoadBalancer services have been fully provisioned, kubectl get services will show the IP addresses of the services:

kubectl get services

NAME         TYPE           CLUSTER-IP               EXTERNAL-IP         PORT(S)        AGE
nginx-ipv4   LoadBalancer   10.0.88.78               20.46.24.24         80:30652/TCP   97s
nginx-ipv6   LoadBalancer   fd12:3456:789a:1::981a   2603:1030:8:5::2d   80:32002/TCP   63s

Next, we can verify functionality via a command-line web request from an IPv6 capable host (note that Azure Cloud Shell is not IPv6 capable):

SERVICE_IP=$(kubectl get services nginx-ipv6 -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
curl -s "http://[${SERVICE_IP}]" | head -n5

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>