Edit

Use Virtual Nodes v2 in Azure Kubernetes Service (AKS) (preview)

Virtual Nodes v2 is an improved Virtual Nodes stack that extends AKS beyond VM-based nodes by running pods on serverless compute in Azure Container Instances (ACI). Workloads burst to ACI in seconds without waiting for new nodes to be provisioned, and you pay only for the compute you use while the pods are running. Virtual Nodes v2 is delivered as an AKS cluster extension.

Important

AKS preview features are available on a self-service, opt-in basis. Previews are provided "as is" and "as available," and they're excluded from the service-level agreements and limited warranty. AKS previews are partially covered by customer support on a best-effort basis. As such, these features aren't meant for production use. For more information, see the following support articles:

Supported

  • Init containers
  • Host aliases
  • Container lifecycle hooks (postStart and preStop)
  • kubectl exec with arguments
  • Persistent volumes and PVCs (Azure Files only)
  • VNet peering
  • NSG-controlled egress
  • NAT Gateway for egress
  • Confidential containers
  • Up to 200 pods per virtual node

Limitations

The following features aren't supported:

  • Windows containers
  • DaemonSets
  • Port forwarding
  • IPv6
  • API server authorized IP ranges
  • Azure CNI Overlay
  • Cilium or NPM as the network policy
  • Bring your own (BYO) virtual network
  • Multiple extension instances per cluster
  • Standby pools

Resource availability and quota limits

Virtual Nodes v2 is available in all Azure public cloud regions where Azure Container Instances supports VNet SKUs. For more information, see Resource availability & quota limits for ACI.

Prerequisites

  • Azure CNI in node-subnet mode
  • Calico or none as the network policy
  • Workload identity enabled
  • At least one node pool with a VM size of 4+ vCPUs and 16+ GiB of memory. Each virtual node reserves about 3 vCPUs and 12 GiB of memory on an AKS node in your cluster, so ensure your nodes have enough resources.
  • k8s-extension version 1.8 or higher
  • kubectl 1.30 or later

Register the ACI resource provider:

az provider register --namespace Microsoft.ContainerInstance

Note

Run the following command if you encounter an error at any stage of the extension enablement process.

az provider register --namespace Microsoft.KubernetesConfig

Enable Virtual Nodes v2

Install the extension:

az k8s-extension create \
    --name <extension-name> \
    --extension-type Microsoft.virtualnodes \
    --cluster-name <cluster-name> \
    --resource-group <resource-group> \
    --cluster-type managedClusters \
    --configuration-settings replicaCount=1   # specifies number of virtual nodes - optional

Configure extension settings

Create and update operations support the following configuration settings.

Configuration setting Description
replicaCount Number of virtual nodes.
admissionControllerReplicaCount Number of admission controller pods.
podAnnotations Annotations added to infrastructure pods as a key-value map.
nodeSelector Labels that select the AKS nodes that host infrastructure and admission controller pods.
tolerations Tolerations applied to infrastructure and admission controller pods.
affinity Additional affinity settings merged with the required AKS affinity rules.
zones Semicolon-delimited availability zones where pods are deployed.
nodeLabels Labels added to virtual nodes as comma-separated key=value pairs.

The allowed kubernetes.io labels are:

  • beta.kubernetes.io/arch
  • beta.kubernetes.io/instance-type
  • beta.kubernetes.io/os
  • failure-domain.beta.kubernetes.io/region
  • failure-domain.beta.kubernetes.io/zone
  • kubernetes.io/arch
  • kubernetes.io/hostname
  • kubernetes.io/os
  • node.kubernetes.io/instance-type
  • topology.kubernetes.io/region
  • topology.kubernetes.io/zone

Verify the extension:

az k8s-extension show \
    --name <extension-name> \
    --cluster-type managedClusters \
    --cluster-name <cluster-name> \
    --resource-group <resource-group>

Confirm provisioningState is Succeeded. It can stay Pending for a few minutes.

Verify the extension pods are running in the vn-system namespace :

kubectl get pods --namespace vn-system

Update an extension

az k8s-extension update --name <extension-name> --cluster-name <aks-cluster-name> --resource-group <aks-resource-group> --cluster-type managedClusters

Note

The --auto-upgrade-mode parameter supports none, patch, or compatible. To pin a specific extension version by using --version, set --auto-upgrade-mode to none.

Azure Container Registry access

To pull images from a private Azure Container Registry (ACR), use the extension managed identity or a Kubernetes image pull secret.

Use the extension managed identity

Grant the AcrPull role to the managed identity generated for the extension. This role grants all pods running on the extension's virtual nodes access to pull images from the registry without extra pod configuration. You must have permission to create role assignments on the registry.

Get the principal ID of the extension managed identity:

$extensionPrincipalId = az k8s-extension show --name <extension-name> --cluster-name <aks-cluster-name> --resource-group <aks-resource-group> --cluster-type managedClusters --query aksAssignedIdentity.principalId --output tsv

Get the resource ID of the container registry:

$acrId = az acr show --name <acr-name> --resource-group <acr-resource-group> --query id --output tsv

Assign the AcrPull role to the extension managed identity at the registry scope:

az role assignment create --assignee-object-id $extensionPrincipalId --assignee-principal-type ServicePrincipal --role AcrPull --scope $acrId

Wait several minutes for the role assignment to propagate before deploying workloads that pull images from the registry.

Use an image pull secret

For instructions, see Kubernetes Pull Secret for ACR Authentication.

Deploy a workload to Virtual Nodes v2

Schedule pods on virtual nodes using the virtualization: virtualnode2 node selector and the virtual-kubelet.io/provider toleration:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo
spec:
  replicas: 3
  selector:
    matchLabels:
      app: demo
  template:
    metadata:
      labels:
        app: demo
    spec:
      containers:
        - name: hello
          image: mcr.microsoft.com/azuredocs/aci-helloworld
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              cpu: "1"
              memory: 1Gi
      nodeSelector:
        virtualization: virtualnode2
        kubernetes.io/os: linux
      tolerations:
        - key: virtual-kubelet.io/provider
          operator: Exists
          effect: NoSchedule

Save the manifest as demo.yaml, and then apply it.

kubectl apply -f demo.yaml

Verify the pods are running.

kubectl get pods -l app=demo -o wide

Disable Virtual Nodes v2

Warning

When you disable the virtual node extension, the AKS cluster disconnects from the backing Azure Container Instances container groups. To avoid orphaned instances that continue running and incurring charges, delete any workloads running on the virtual node before you disable the extension.

Delete the extension and clean up virtual node resources.

az k8s-extension delete \
    --name <extension-name> \
    --cluster-name <cluster-name> \
    --resource-group <resource-group> \
    --cluster-type managedClusters

kubectl delete node <virtual-node-name>

Next steps

  • Last updated on