Use Virtual Machines node pools in Azure Kubernetes Service (AKS)

In this article, you'll learn about the new Virtual Machines node pool type for AKS.

With Virtual Machines node pools, AKS directly manages the provisioning and bootstrapping of every single node. For Virtual Machine Scale Sets node pools, AKS manages the model of the Virtual Machine Scale Sets and uses it to achieve consistency across all nodes in the node pool. Virtual Machines node pools enable you to orchestrate your cluster with virtual machines that best fit your individual workloads.

By using virtual machines node pools, AKS directly manages the provisioning and bootstrapping of every single node. For virtual machine scale sets node pools, AKS manages the model of the virtual machine scale sets and uses it to achieve consistency across all nodes in the node pool. Virtual machines node pools enable you to orchestrate your cluster with virtual machines that best fit your individual workloads.

Virtual machines node pool overview

How it works

A node pool consists of a set of virtual machines (VMs), where different virtual machine sizes support different types of workloads. These virtual machine sizes, referred to as SKUs, are categorized into different families that are optimized for specific purposes. For more information, see VM SKUs documentation. By using virtual machines node pools, you can perform multi-SKU manual scaling and multi-SKU autoscaling.

To enable scaling of multiple virtual machine sizes, the Virtual Machines node pool type uses a ScaleProfile that contains configurations indicating how the node pool can scale, specifically the desired list of virtual machine size and the count of each size. A ManualScaleProfile is a scale profile that specifies one desired virtual machine size and the total count of that type in the node pool. Only one virtual machine size is allowed in a ManualScaleProfile. You need to create a separate ManualScaleProfile for each virtual machine size in your node pool. When creating a new Virtual Machines node pool, you add an initial manual scale profile for a virtual machine size using the vm-size field and including a node-count. You can also add more manual scale profiles following the instructions for adding manual scale profiles.

To enable scaling of multiple virtual machine sizes, the virtual machines node pool type uses a ScaleProfile that contains configurations indicating how the node pool can scale, specifically the desired list of virtual machine size and the count of each size. A ManualScaleProfile is a scale profile that specifies one desired virtual machine size and the total count of that type in the node pool. A ManualScaleProfile supports only one virtual machine size. You need to create a separate ManualScaleProfile for each virtual machine size in your node pool. When creating a new virtual machines node pool, add an initial manual scale profile for a virtual machine size by using the vm-size field and including a node-count. You can also add more manual scale profiles by following the instructions for adding manual scale profiles.

Virtual machines node pools also support Auto mode, which means the node pool can use cluster autoscaler. A virtual machines node pool supports up to five scale profiles total across manual and autoscale modes. Each profile specifies one same-family VM SKU. Each AutoScaleProfile can have its own minimum and maximum node count in the node pool.

Note

When creating a new Virtual Machines node pool, you can have multiple scale profiles, and you need at least one manual or auto scale profile in your node pool.

Advantages of virtual machines node pools

Advantages of the Virtual Machines node pool type include:

• Flexibility: Node specifications can be updated to adapt to your current workload and needs.
• Fine-tuned control: Single node-level controls allow specifying and mixing nodes of different specs to lift restrictions from a single model and improve consistency.
• Efficiency: You can reduce the node footprint for your cluster, simplifying your operational requirements.

Virtual machines node pools support up to five scale profiles per node pool, so workloads can use multiple same-family VM SKUs without requiring a separate node pool for each SKU.

Compare AKS compute scaling experiences

The following table highlights how Virtual Machines node pools compare with standard Scale Set node pools.

Node pool type	Scaling and node management capabilities
Virtual Machines node pool	You can add, remove, or update nodes in a node pool. Virtual machine types can be any virtual machine of the same family type (for example, D-series, A-Series, and so on). Virtual Machines node pools also allow for multiversion manual and auto scaling.
Virtual Machine Scale Set based node pool	You can add or remove nodes of the same size and type in a node pool. If you add a new virtual machine size to the cluster, you need to create a new node pool.

Which compute scaling experience should I choose on AKS?

Depending on your workload needs, there are multiple compute scaling experiences to consider. See the use cases for each:

• Node auto provisioning: best for multiversion autoscaling, and more intelligent, flexible VM version selection (including multiple version families).
• Virtual Machines node pools: best for multiversion manual scaling and supports multiversion autoscaling. Requires specific version selection of up to five sizes per node pool.
• Virtual Machine Scale Sets: supports single-version manual scaling and single-version autoscaling. Requires specific version selection of one size per node pool.

Virtual Machines node pool limitations

• VM Sizes specified in the pool must be of the same type. For example, GPU and non-GPU or x86 and ARM64 virtual machines cannot be in the same node pool.
• InifiniBand isn't available.
• Node pool snapshot isn't supported.
• All VM sizes selected in a node pool need to be from a similar virtual machine family. For example, you can't mix an N-Series virtual machine type with a D-Series virtual machine type in the same node pool.
• Virtual Machines node pools allow up to five scale profiles total per node pool. Each manual or autoscale profile specifies one VM size from the same VM family.
• Windows node pools aren't supported.
• Availability zones aren't supported. If your workload requires zone resiliency, use Virtual Machine Scale Sets node pools.

Prerequisites

• An Azure subscription. If you don't have one, you can create a free account.
• Azure CLI version 2.85.0 or later installed and configured. To find the version, run az --version. For more information about installing or upgrading the Azure CLI, see Install Azure CLI.
• This feature requires Kubernetes version 1.27 or greater. To upgrade your Kubernetes version, see Upgrade AKS cluster.

Important

Custom virtual network requirement: If you deploy a Virtual Machines node pool into a custom virtual network, the cluster must use a user-assigned managed identity with at least Network Contributor permissions on the target subnet. Unlike Virtual Machine Scale Set node pools, Virtual Machines node pools rely solely on the cluster identity for subnet join operations and don't use first-party tokens. Clusters that use a system-assigned managed identity fail preflight validation when creating or updating a Virtual Machines node pool on a custom virtual network, returning an InvalidParameter error. For more information on configuring a user-assigned managed identity for your cluster, see Use a managed identity in AKS.

Create an AKS cluster with Virtual Machines node pools

Note

Only one VM size is allowed in a scale profile, and the maximum limit is five VM scale profiles overall for a Virtual Machines node pool.

Create an AKS cluster with Virtual Machines node pools using the az aks create command with the --vm-set-type flag set to "VirtualMachines".

The following example creates a cluster named myAKSCluster with a Virtual Machines node pool containing two nodes:

az aks create \
    --resource-group myResourceGroup \
    --name myAKSCluster \
    --vm-set-type "VirtualMachines" \
    --vm-sizes "Standard_D4s_v3" \
    --node-count 2

Create an AKS cluster with Virtual Machines node pools in a custom virtual network

When you deploy Virtual Machines node pools into a custom virtual network, you must create a user-assigned managed identity and grant it Network Contributor permissions on the target subnet before you create the cluster.

1. Create a virtual network and subnet.

   az network vnet create \
       --resource-group myResourceGroup \
       --name myVnet \
       --address-prefixes 10.1.0.0/16 \
       --subnet-name mySubnet \
       --subnet-prefix 10.1.0.0/24
   

2. Get the subnet resource ID.

   SUBNET_ID=$(az network vnet subnet show \
       --resource-group myResourceGroup \
       --vnet-name myVnet \
       --name mySubnet \
       --query id \
       --output tsv)
   

3. Create a user-assigned managed identity.

   az identity create \
       --name myAKSIdentity \
       --resource-group myResourceGroup
   

4. Get the principal ID and resource ID of the managed identity.

   IDENTITY_PRINCIPAL_ID=$(az identity show \
       --name myAKSIdentity \
       --resource-group myResourceGroup \
       --query principalId \
       --output tsv)
   
   IDENTITY_RESOURCE_ID=$(az identity show \
       --name myAKSIdentity \
       --resource-group myResourceGroup \
       --query id \
       --output tsv)
   

5. Assign the Network Contributor role to the managed identity on the target subnet.

   az role assignment create \
       --assignee $IDENTITY_PRINCIPAL_ID \
       --role "Network Contributor" \
       --scope $SUBNET_ID
   

   It can take up to 60 minutes to propagate the permissions granted to your cluster's managed identity.

6. Verify that the role assignment exists on the target subnet before you create the cluster.

   az role assignment list \
       --assignee $IDENTITY_PRINCIPAL_ID \
       --role "Network Contributor" \
       --scope $SUBNET_ID \
       --query "[].{Role:roleDefinitionName, Scope:scope}" \
       --output table
   

   Continue when the output shows the Network Contributor role at the target subnet scope.

7. Create the AKS cluster with Virtual Machines node pools in your custom virtual network.

   az aks create \
       --resource-group myResourceGroup \
       --name myAKSCluster \
       --vm-set-type "VirtualMachines" \
       --vm-sizes "Standard_D4s_v3" \
       --node-count 2 \
       --vnet-subnet-id $SUBNET_ID \
       --assign-identity $IDENTITY_RESOURCE_ID
   

Add a Virtual Machines node pool to an existing cluster

Add a Virtual Machines node pool to an existing cluster using the az aks nodepool add command with the --vm-set-type flag set to "VirtualMachines".

The following example adds a Virtual Machines node pool named myvmpool to the myAKSCluster cluster. The node pool creates a ManualScaleProfile with --vm-sizes set to Standard_D4s_v3 and a --node-count of 3:

az aks nodepool add \
    --resource-group myResourceGroup \
    --cluster-name myAKSCluster \
    --name myvmpool \
    --vm-set-type "VirtualMachines" \
    --vm-sizes "Standard_D4s_v3" \
    --node-count 3

Add a manual scale profile to a node pool

Add a manual scale profile to a node pool using the az aks nodepool manual-scale add with the --vm-sizes flag set to "Standard_D2s_v3" and the node-count set to 2.

The following example adds a manual scale profile to node pool myvmpool in cluster myAKSCluster. The node pool includes two nodes with a VM SKU of Standard_D2s_v3:

az aks nodepool manual-scale add \
    --resource-group myResourceGroup \
    --cluster-name myAKSCluster \
    --name myvmpool \
    --vm-sizes "Standard_D2s_v3" \
    --node-count 2

Update an existing manual scale profile

Update an existing manual scale profile in a node pool using the az aks nodepool manual-scale update command with the --vm-sizes flag set to "Standard_D2s_v3".

Note

Use the --current-vm-sizes parameter to specify the size of the existing node pool that you want to update. You can update --vm-sizes and/or --node-count. When using other tools or REST APIs, you need to pass in a full agentPoolProfiles.virtualMachinesProfile.scale field when updating the node pool scale profile.

The following example updates a manual scale profile to the myvmpool node pool in the myAKSCluster cluster. The command updates the number of nodes to five and changes the VM SKU from Standard_D4s_v3 to Standard_D8s_v3:

az aks nodepool manual-scale update \
    --resource-group myResourceGroup \
    --cluster-name myAKSCluster \
    --name myvmpool \
    --current-vm-sizes "Standard_D4s_v3" \
    --vm-sizes "Standard_D8s_v3" \
    --node-count 5

Delete a manual scale profile

Delete an existing manual scale profile using the az aks nodepool manual-scale delete command.

Note

The --current-vm-sizes parameter specifies the size of the existing node pool to be deleted. When using other tools or REST APIs to update the node pool scale profile, pass in a full agentPoolProfiles.virtualMachinesProfile.scale field.

The following example deletes the manual scale profile for the Standard_D8s_v3 VM SKU in the myvmpool node pool.

az aks nodepool manual-scale delete \
    --resource-group myResourceGroup \
    --cluster-name myAKSCluster \
    --name myvmpool \
    --current-vm-sizes "Standard_D8s_v3"

Cluster autoscaler with Virtual Machines node pools (preview)

Virtual Machines node pools support cluster autoscaler. This preview feature requires Azure CLI version 2.85.0 or later, the aks-preview extension, and registration of the VMsAgentPoolAutoscalePreview feature flag. After you meet these prerequisites, you can use --enable-cluster-autoscaler during cluster creation, while adding a new node pool, or when updating an existing manual node pool.

When you use cluster autoscaler with Virtual Machines node pools, the behavior is as follows:

Autoscaler action	Behavior
Scale up	The autoscaler responds to pending pod pressure and can increase the node count for multiple VM sizes in the node pool.
Scale down	The autoscaler selects a node based on utilization. You can configure scale-down-utilization-threshold to adjust when the autoscaler triggers a scale-down action. For more information, see cluster autoscaler.

Limitations

• This feature is only available in public cloud.
• GPU Nodes are not currently supported.
• This feature requires Azure CLI version 2.85.0 or later and the aks-preview extension.

Install the aks-preview extension

Important

AKS preview features are available on a self-service, opt-in basis. Previews are provided "as is" and "as available," and they're excluded from the service-level agreements and limited warranty. AKS previews are partially covered by customer support on a best-effort basis. As such, these features aren't meant for production use. For more information, see the following support articles:

• AKS support policies
• Azure support FAQ

• Install or update the aks-preview Azure CLI extension by using the az extension add or az extension update command:

    # Install the aks-preview extension
    az extension add --name aks-preview
    
    # Update the aks-preview extension
    az extension update --name aks-preview

Register feature flag

Register the preview feature flag VMsAgentPoolAutoscalePreview using the az feature register command:

az feature register \
    --namespace Microsoft.ContainerService \
    --name VMsAgentPoolAutoscalePreview

It takes a few minutes for the status to become Registered. Verify the registration status using the az feature show command:

az feature show \
    --namespace Microsoft.ContainerService \
    --name VMsAgentPoolAutoscalePreview \
    --query properties.state \
    --output tsv

When the status is Registered, refresh the Microsoft.ContainerService resource provider registration using the az provider register command:

az provider register --namespace Microsoft.ContainerService

Create an AKS cluster with Virtual Machines node pools and cluster autoscaler enabled

• Create an AKS cluster with Virtual Machines node pools using the az aks create command with the --vm-set-type flag set to "VirtualMachines" and with the flag --enable-cluster-autoscaler.

The following example creates a cluster named myAKSCluster with a Virtual Machines node pool that uses the node size Standard_D4s_v3, a minimum node count of 2, and a maximum node count of 5:

az aks create \
    --resource-group myResourceGroup \
    --name myAKSCluster \
    --vm-set-type "VirtualMachines" \
    --node-vm-size "Standard_D4s_v3" \
    --enable-cluster-autoscaler \
    --min-count 2 \
    --max-count 5

Add a Virtual Machines node pool with cluster autoscaler enabled to an existing cluster

• Create a Virtual Machines node pool using the az aks nodepool add command with the --vm-set-type flag set to "VirtualMachines" and with the flag --enable-cluster-autoscaler.

The following example adds Virtual Machines node pool myvmpool with cluster autoscaler enabled to a cluster named myAKSCluster using virtual machine size of "Standard_D4s_v3", and a minimum node count of 2 and max count of 5:

az aks nodepool add \
    --resource-group myResourceGroup \
    --cluster-name myAKSCluster \
    --name myvmpool \
    --vm-set-type "VirtualMachines" \
    --node-vm-size "Standard_D4s_v3" \
    --enable-cluster-autoscaler \
    --min-count 2 \
    --max-count 5

Update cluster autoscaler settings for a Virtual Machines node pool

• Update the cluster autoscaler node count settings for a Virtual Machines node pool using the az aks nodepool auto-scale update command.

The following example updates settings for Virtual Machines node pool myvmpool in cluster named myAKSCluster using virtual machine size of "Standard_D4s_v3":

az aks nodepool auto-scale update \
    --resource-group myResourceGroup \
    --cluster-name myAKSCluster \
    --name myvmpool \
    --current-node-vm-size "Standard_D4s_v3" \
    --min-count 3 \
    --max-count 7

Switch Virtual Machines node pool scale profiles from manual to autoscale mode

The following example updates Virtual Machines node pool myvmpool in the cluster named myAKSCluster, converting all manual scale profiles to autoscale profiles with the same minimum and maximum node count:

az aks nodepool update \
    --resource-group myResourceGroup \
    --cluster-name myAKSCluster \
    --name myvmpool \
    --enable-cluster-autoscaler \
    --min-count 2 \
    --max-count 5

Disable cluster autoscaler in a Virtual Machines node pool

You can disable cluster autoscaler, or switch all the scale profiles in a virtual machine node pool from Auto mode to Manual mode.

The following example disables the cluster autoscaler for the Virtual Machines node pool myvmpool in the cluster named myAKSCluster. It switches all scale profiles from Auto mode to Manual mode:

az aks nodepool update \
    --resource-group myResourceGroup \
    --cluster-name myAKSCluster \
    --name myvmpool \
    --disable-cluster-autoscaler

Next steps

In this article, you learned how to use Virtual Machines node pools in AKS. To learn more about node pools in AKS, see Create node pools.