Training
Learning path
Use advance techniques in canvas apps to perform custom updates and optimization - Training
Use advance techniques in canvas apps to perform custom updates and optimization
This browser is no longer supported.
Upgrade to Microsoft Edge to take advantage of the latest features, security updates, and technical support.
Note
Many of the steps listed in this document apply to Virtual Machine Scale Sets using Uniform Orchestration mode. We recommend using Flexible Orchestration for new workloads. For more information, see Orchestration modes for Virtual Machine Scale Sets in Azure.
Enabling automatic OS image upgrades on your scale set helps ease update management by safely and automatically upgrading the OS disk for all instances in the scale set.
Automatic OS upgrade has the following characteristics:
Note
Before enabling automatic OS image upgrades, check requirements section of this documentation.
An upgrade works by replacing the OS disk of a virtual machine with a new disk created using the image version. Any configured extensions and custom data scripts are run on the OS disk, while data disks are retained. To minimize the application downtime, upgrades take place in batches, with no more than 20% of the scale set upgrading at any time.
You must integrate an Azure Load Balancer application health probe or Application Health extension to track the health of the application after an upgrade. This allows the platform to validate the virtual machine health to ensure updates are applied in a safe manner. We recommend incorporating an application heartbeat to validate upgrade success.
The availability-first model for platform orchestrated updates described below ensures that availability configurations in Azure are respected across multiple availability levels.
Across regions:
Within a region:
Within a 'set':
The platform orchestrated updates process is followed for rolling out supported OS platform image upgrades every month. For custom images through Azure Compute Gallery, an image upgrade is only kicked off for a particular Azure region when the new image is published and replicated to the region of that scale set.
The region of a scale set becomes eligible to get image upgrades either through the availability-first process for platform images or replicating new custom image versions for Share Image Gallery. The image upgrade is then applied to an individual scale set in a batched manner as follows:
The scale set OS upgrade orchestrator checks for the overall scale set health before upgrading every batch. While you're upgrading a batch, there could be other concurrent planned or unplanned maintenance activities that could impact the health of your scale set instances. In such cases if more than 20% of the scale set's instances become unhealthy, then the scale set upgrade stops at the end of current batch.
To modify the default settings associated with Rolling Upgrades, review Azure's Rolling Upgrade Policy.
Note
Automatic OS upgrade does not upgrade the reference image Sku on the scale set. To change the Sku (such as Ubuntu 18.04-LTS to 20.04-LTS), you must update the scale set model directly with the desired image Sku. Image publisher and offer can't be changed for an existing scale set.
Both OS Image Upgrade and Reimage are methods used to update virtual machines within a scale set, but they serve different purposes and have distinct impacts.
OS image upgrade involves updating the underlying operating system image that is used to create new instances in a scale set. When you perform an OS image upgrade, Azure creates new virtual machines with the updated OS image and gradually replace the old virtual machines in the scale set with the new ones. This process is typically performed in stages to ensure high availability. OS image upgrades are a non-disruptive way to apply updates or changes to the underlying OS of the virtual machines in a scale set. Existing virtual machines are not affected until they are replaced with the new instances.
Reimaging a virtual machine in a scale set is a more immediate and disruptive action. When you choose to reimage a virtual machine , Azure stops the selected virtual machine, perform the reimage operation, and then restart the virtual machine using the same OS image. This effectively reinstalls the OS on that specific virtual machine. Reimaging is typically used when you need to troubleshoot or reset a specific virtual machine due to issues with that instance.
Key differences:
When to use each method:
It's essential to carefully plan and choose the appropriate method based on your specific requirements to minimize any disruption to your applications and services running in a Virtual Machine Scale Set.
Only certain OS platform images are currently supported. Custom images are supported if the scale set uses custom images through Azure Compute Gallery.
The following platform SKUs are currently supported (and more are added periodically):
Publisher | OS Offer | Sku |
---|---|---|
Canonical | UbuntuServer | 18.04-LTS |
Canonical | UbuntuServer | 18_04-LTS-Gen2 |
Canonical | 0001-com-ubuntu-server-focal | 20_04-LTS |
Canonical | 0001-com-ubuntu-server-focal | 20_04-LTS-Gen2 |
Canonical | 0001-com-ubuntu-server-jammy | 22_04-LTS |
Canonical | 0001-com-ubuntu-server-jammy | 22_04-LTS-Gen2 |
MicrosoftCblMariner | Cbl-Mariner | cbl-mariner-1 |
MicrosoftCblMariner | Cbl-Mariner | 1-Gen2 |
MicrosoftCblMariner | Cbl-Mariner | cbl-mariner-2 |
MicrosoftCblMariner | Cbl-Mariner | cbl-mariner-2-Gen2 |
MicrosoftSqlServer | Sql2017-ws2019 | enterprise |
MicrosoftWindowsServer | WindowsServer | 2012-R2-Datacenter |
MicrosoftWindowsServer | WindowsServer | 2016-Datacenter |
MicrosoftWindowsServer | WindowsServer | 2016-Datacenter-gensecond |
MicrosoftWindowsServer | WindowsServer | 2016-Datacenter-gs |
MicrosoftWindowsServer | WindowsServer | 2016-Datacenter-smalldisk |
MicrosoftWindowsServer | WindowsServer | 2016-Datacenter-with-Containers |
MicrosoftWindowsServer | WindowsServer | 2016-Datacenter-with-containers-gs |
MicrosoftWindowsServer | WindowsServer | 2019-Datacenter |
MicrosoftWindowsServer | WindowsServer | 2019-Datacenter-Core |
MicrosoftWindowsServer | WindowsServer | 2019-Datacenter-Core-with-Containers |
MicrosoftWindowsServer | WindowsServer | 2019-Datacenter-gensecond |
MicrosoftWindowsServer | WindowsServer | 2019-Datacenter-gs |
MicrosoftWindowsServer | WindowsServer | 2019-Datacenter-smalldisk |
MicrosoftWindowsServer | WindowsServer | 2019-Datacenter-with-Containers |
MicrosoftWindowsServer | WindowsServer | 2019-Datacenter-with-Containers-gs |
MicrosoftWindowsServer | WindowsServer | 2022-Datacenter |
MicrosoftWindowsServer | WindowsServer | 2022-Datacenter-smalldisk |
MicrosoftWindowsServer | WindowsServer | 2022-Datacenter-smalldisk-g2 |
MicrosoftWindowsServer | WindowsServer | 2022-Datacenter-core |
MicrosoftWindowsServer | WindowsServer | 2022-Datacenter-core-smalldisk |
MicrosoftWindowsServer | WindowsServer | 2022-Datacenter-g2 |
MicrosoftWindowsServer | WindowsServer | Datacenter-core-20h2-with-containers-smalldisk-gs |
MicrosoftWindowsServer | WindowsServer | 2022-Datacenter-azure-edition |
MicrosoftWindowsServer | WindowsServer | 2022-Datacenter-azure-edition-smalldisk |
Mirantis | Windows_with_Mirantis_Container_Runtime_2019 | win_2019_mcr_23_0 |
Mirantis | Windows_with_Mirantis_Container_Runtime_2019 | win_2019_mcr_23_0_gen2 |
AutomaticByPlatform
in the scale set model definition. With automatic OS image upgrades enabled on your scale set, a platform orchestration patching process is not required.Note
After an OS disk is replaced through reimage or upgrade, the attached data disks may have their drive letters reassigned. To retain the same drive letters for attached disks, it is suggested to use a custom boot script.
If you are using Service Fabric, ensure the following conditions are met:
Ensure that durability settings are not mismatched on the Service Fabric cluster and Service Fabric extension, as a mismatch results in upgrade errors. Durability levels can be modified per the guidelines outlined on this page.
Automatic OS image upgrade is supported for custom images deployed through Azure Compute Gallery. Other custom images are not supported for automatic OS image upgrades.
Note
It can take up to 3 hours for a scale set to trigger the first image upgrade rollout after the scale set is first configured for automatic OS upgrades due to certain factors such as Maintenance Windows or other restrictions. Customers on the latest image may not get an upgrade until a new image is available.
To configure automatic OS image upgrade, ensure that the automaticOSUpgradePolicy.enableAutomaticOSUpgrade property is set to true in the scale set model definition.
Note
Upgrade Policy mode and Automatic OS Upgrade Policy are separate settings and control different aspects of the scale set. When there are changes in the scale set template, the Upgrade Policy mode
determines what happens to existing instances in the scale set. However, Automatic OS Upgrade Policy enableAutomaticOSUpgrade
is specific to the OS image and tracks changes the image publisher has made and determines what happens when there is an update to the image.
Note
If enableAutomaticOSUpgrade
is set to true, enableAutomaticUpdates
is automatically set to false and cannot be set to true.
The following example describes how to set automatic OS upgrades on a scale set model:
PUT or PATCH on `/subscriptions/subscription_id/resourceGroups/myResourceGroup/providers/Microsoft.Compute/virtualMachineScaleSets/myScaleSet?api-version=2021-03-01`
{
"properties": {
"upgradePolicy": {
"automaticOSUpgradePolicy": {
"enableAutomaticOSUpgrade": true
}
}
}
}
Use the New-AzVmss cmdlet to configure automatic OS image upgrades for your scale set during provisioning. The following example configures automatic upgrades for the scale set named myScaleSet in the resource group named myResourceGroup:
New-AzVmss -ResourceGroupName "myResourceGroup" -VMScaleSetName "myScaleSet" -AutomaticOSUpgrade $true
Use the Update-AzVmss cmdlet to configure automatic OS image upgrades for your existing scale set. The following example configures automatic upgrades for the scale set named myScaleSet in the resource group named myResourceGroup:
Update-AzVmss -ResourceGroupName "myResourceGroup" -VMScaleSetName "myScaleSet" -AutomaticOSUpgrade $true
Use az vmss create to configure automatic OS image upgrades for your scale set during provisioning. Use Azure CLI 2.0.47 or above. The following example configures automatic upgrades for the scale set named myScaleSet in the resource group named myResourceGroup:
az vmss create --name myScaleSet --resource-group myResourceGroup --enable-auto-os-upgrade true --upgrade-policy-mode Rolling
Use az vmss update to configure automatic OS image upgrades for your existing scale set. Use Azure CLI 2.0.47 or above. The following example configures automatic upgrades for the scale set named myScaleSet in the resource group named myResourceGroup:
az vmss update --name myScaleSet --resource-group myResourceGroup --enable-auto-os-upgrade true --upgrade-policy-mode Rolling
Note
After configuring automatic OS image upgrades for your scale set, you must also bring the scale set virtual machines to the latest scale set model if your scale set uses the 'Manual' upgrade policy.
The following example describes how to set automatic OS upgrades on a scale set model via Azure Resource Manager templates (ARM templates):
"properties": {
"upgradePolicy": {
"mode": "Automatic",
"RollingUpgradePolicy": {
"BatchInstancePercent": 20,
"MaxUnhealthyInstancePercent": 25,
"MaxUnhealthyUpgradedInstancePercent": 25,
"PauseTimeBetweenBatches": "PT0S"
},
"automaticOSUpgradePolicy": {
"enableAutomaticOSUpgrade": true,
"useRollingUpgradePolicy": true,
"disableAutomaticRollback": false
}
},
},
"imagePublisher": {
"type": "string",
"defaultValue": "MicrosoftWindowsServer"
},
"imageOffer": {
"type": "string",
"defaultValue": "WindowsServer"
},
"imageSku": {
"type": "string",
"defaultValue": "2022-datacenter"
},
"imageOSVersion": {
"type": "string",
"defaultValue": "latest"
}
The following example describes how to set automatic OS upgrades on a scale set model via Bicep:
properties: {
overprovision: overProvision
upgradePolicy: {
mode: 'Automatic'
automaticOSUpgradePolicy: {
enableAutomaticOSUpgrade: true
}
}
}
During an OS Upgrade, virtual machines in a scale set are upgraded one batch at a time. The upgrade should continue only if the customer application is healthy on the upgraded virtual machines. We recommend that the application provides health signals to the scale set OS Upgrade engine. By default, during OS Upgrades the platform considers virtual machine power state and extension provisioning state to determine if a virtual machine is healthy after an upgrade. During the OS Upgrade of a virtual machine, the OS disk on a virtual machine is replaced with a new disk based on latest image version. After the OS Upgrade has completed, the configured extensions are run on these virtual machines. The application is considered healthy only when all the extensions on the instance are successfully provisioned.
A scale set can optionally be configured with Application Health Probes to provide the platform with accurate information on the ongoing state of the application. Application Health Probes are Custom Load Balancer Probes that are used as a health signal. The application running on a scale set virtual machine can respond to external HTTP or TCP requests indicating whether it's healthy. For more information on how Custom Load Balancer Probes work, see to Understand load balancer probes. Application Health Probes are not supported for Service Fabric scale sets. Non-Service Fabric scale sets require either Load Balancer application health probes or Application Health extension.
If the scale set is configured to use multiple placement groups, probes using a Standard Load Balancer need to be used.
Note
Only one source of health monitoring can be used for a Virtual Machine Scale Set, either an Application Health Extension or a Health Probe. If you have both options enabled, you need to remove one before using orchestration services like Instance Repairs or Automatic OS Upgrades.
As a best practice, create a load balancer probe explicitly for scale set health. The same endpoint for an existing HTTP probe or TCP probe can be used, but a health probe could require different behavior from a traditional load-balancer probe. For example, a traditional load balancer probe could return unhealthy if the load on the instance is too high, but that would not be appropriate for determining the instance health during an automatic OS upgrade. Configure the probe to have a high probing rate of less than two minutes.
The load-balancer probe can be referenced in the networkProfile of the scale set and can be associated with either an internal or public facing load-balancer as follows:
"networkProfile": {
"healthProbe" : {
"id": "[concat(variables('lbId'), '/probes/', variables('sshProbeName'))]"
},
"networkInterfaceConfigurations":
...
}
Note
When using Automatic OS Upgrades with Service Fabric, the new OS image is rolled out Update Domain by Update Domain to maintain high availability of the services running in Service Fabric. To utilize Automatic OS Upgrades in Service Fabric your cluster node type must be configured to use the Silver Durability Tier or higher. For Bronze Durability tier, automatic OS image upgrade is only supported for Stateless node types. For more information on the durability characteristics of Service Fabric clusters, see this documentation.
The Application Health extension is deployed inside a Virtual Machine Scale Set instance and reports on virtual machine health from inside the scale set instance. You can configure the extension to probe on an application endpoint and update the status of the application on that instance. This instance status is checked by Azure to determine whether an instance is eligible for upgrade operations.
As the extension reports health from within a virtual machine, the extension can be used in situations where external probes such as Application Health Probes (that utilize custom Azure Load Balancer probes) can’t be used.
There are multiple ways of deploying the Application Health extension to your scale sets as detailed in the examples in this article.
Note
Only one source of health monitoring can be used for a Virtual Machine Scale Set, either an Application Health Extension or a Health Probe. If you have both options enabled, you need to remove one before using orchestration services like Instance Repairs or Automatic OS Upgrades.
Note
Custom metrics for rolling upgrades on Virtual Machine Scale Sets is currently in preview. Previews are made available to you on the condition that you agree to the supplemental terms of use. Some aspects of these features may change prior to general availability (GA).
Custom metrics for rolling upgrades enables you to utilize the application health extension to emit custom metrics to your Virtual Machine Scale Set. These custom metrics can be used to tell the scale set the order in which virtual machines should be updated when a rolling upgrade is triggered. The custom metrics can also inform your scale set when an upgrade should be skipped on a specific instance. This allows you to have more control over the ordering and the update process itself.
Custom metrics can be used in combination with other rolling upgrade functionality such as automatic OS upgrades, automatic extension upgrades and MaxSurge rolling upgrades.
For more information, see custom metrics for rolling upgrades on Virtual Machine Scale Sets
You can check the history of the most recent OS upgrade performed on your scale set with Azure PowerShell, Azure CLI 2.0, or the REST APIs. You can get history for the last five OS upgrade attempts within the past two months.
If your scale set uses any credentials to access external resources, such as a virtual machine extension configured to use a SAS token for storage account, then ensure that the credentials are updated. If any credentials including certificates and tokens have expired, the upgrade fails and the first batch of virtual machines are left in a failed state.
The recommended steps to recover virtual machines and re-enable automatic OS upgrade if there's a resource authentication failure are:
The following example uses REST API to check the status for the scale set named myScaleSet in the resource group named myResourceGroup:
GET on `/subscriptions/subscription_id/resourceGroups/myResourceGroup/providers/Microsoft.Compute/virtualMachineScaleSets/myScaleSet/osUpgradeHistory?api-version=2021-03-01`
The GET call returns properties similar to the following example output:
{
"value": [
{
"properties": {
"runningStatus": {
"code": "RollingForward",
"startTime": "2018-07-24T17:46:06.1248429+00:00",
"completedTime": "2018-04-21T12:29:25.0511245+00:00"
},
"progress": {
"successfulInstanceCount": 16,
"failedInstanceCount": 0,
"inProgressInstanceCount": 4,
"pendingInstanceCount": 0
},
"startedBy": "Platform",
"targetImageReference": {
"publisher": "MicrosoftWindowsServer",
"offer": "WindowsServer",
"sku": "2016-Datacenter",
"version": "2016.127.20180613"
},
"rollbackInfo": {
"successfullyRolledbackInstanceCount": 0,
"failedRolledbackInstanceCount": 0
}
},
"type": "Microsoft.Compute/virtualMachineScaleSets/rollingUpgrades",
"location": "westeurope"
}
]
}
Use the Get-AzVmss cmdlet to check OS upgrade history for your scale set. The following example details how you review the OS upgrade status for a scale set named myScaleSet in the resource group named myResourceGroup:
Get-AzVmss -ResourceGroupName "myResourceGroup" -VMScaleSetName "myScaleSet" -OSUpgradeHistory
Use az vmss get-os-upgrade-history to check the OS upgrade history for your scale set. Use Azure CLI 2.0.47 or above. The following example details how you review the OS upgrade status for a scale set named myScaleSet in the resource group named myResourceGroup:
az vmss get-os-upgrade-history --resource-group myResourceGroup --name myScaleSet
You can get the available image versions for automatic OS upgrade supported SKUs using the below examples:
GET on `/subscriptions/subscription_id/providers/Microsoft.Compute/locations/{location}/publishers/{publisherName}/artifacttypes/vmimage/offers/{offer}/skus/{skus}/versions?api-version=2021-03-01`
Get-AzVmImage -Location "westus" -PublisherName "Canonical" -offer "0001-com-ubuntu-server-jammy" -sku "22_04-lts"
az vm image list --location "westus" --publisher "Canonical" --offer "0001-com-ubuntu-server-jammy" --sku "22_04-lts" --all
With automatic OS image upgrade enabled on your scale set, you don't need to manually trigger image updates on your scale set. The OS upgrade orchestrator automatically applies the latest available image version to your scale set instances without any manual intervention.
For specific cases where you don't want to wait for the orchestrator to apply the latest image, you can trigger an OS image upgrade manually using the below examples.
Note
Manual trigger of OS image upgrades does not provide automatic rollback capabilities. If an instance does not recover its health after an upgrade operation, its previous OS disk can't be restored.
Use the Start OS Upgrade API call to start a rolling upgrade to move all Virtual Machine Scale Set instances to the latest available image OS version. Instances that are already running the latest available OS version are not affected. The following example details how you can start a rolling OS upgrade on a scale set named myScaleSet in the resource group named myResourceGroup:
POST on `/subscriptions/subscription_id/resourceGroups/myResourceGroup/providers/Microsoft.Compute/virtualMachineScaleSets/myScaleSet/osRollingUpgrade?api-version=2021-03-01`
Use the Start-AzVmssRollingOSUpgrade cmdlet to check OS upgrade history for your scale set. The following example details how you can start a rolling OS upgrade on a scale set named myScaleSet in the resource group named myResourceGroup:
Start-AzVmssRollingOSUpgrade -ResourceGroupName "myResourceGroup" -VMScaleSetName "myScaleSet"
Use az vmss rolling-upgrade start to check the OS upgrade history for your scale set. Use Azure CLI 2.0.47 or above. The following example details how you can start a rolling OS upgrade on a scale set named myScaleSet in the resource group named myResourceGroup:
az vmss rolling-upgrade start --resource-group "myResourceGroup" --name "myScaleSet" --subscription "subscriptionId"
Activity Log is a subscription log that provides insight into subscription-level events that have occurred in Azure. Customers are able to:
Customers receive three types of notifications related to Automatic OS Upgrade operation:
An action group is a collection of notification preferences defined by the owner of an Azure subscription. Azure Monitor and Service Health alerts use action groups to notify users that an alert has been triggered.
Action groups can be created and managed using:
Customers can set up the following using action groups:
The platform can return errors on virtual machines while performing Automatic Image Upgrade with Rolling Upgrade policy. The Get Instance View of a virtual machine contains the detailed error message to investigate and resolve an error. The Rolling Upgrades - Get Latest can provide more details on rolling upgrade configuration and status. The Get OS Upgrade History provides details on the last image upgrade operation on the scale set. Below are the topmost errors that can result in Rolling Upgrades.
RollingUpgradeInProgressWithFailedUpgradedVMs
MaxUnhealthyUpgradedInstancePercentExceededInRollingUpgrade
MaxUnhealthyInstancePercentExceededInRollingUpgrade
MaxUnhealthyInstancePercentExceededBeforeRollingUpgrade
InternalExecutionError
RollingUpgradeTimeoutError
Training
Learning path
Use advance techniques in canvas apps to perform custom updates and optimization - Training
Use advance techniques in canvas apps to perform custom updates and optimization
Documentation
Modify an Azure Virtual Machine Scale Set - Azure Virtual Machine Scale Sets
Learn how to modify and update an Azure Virtual Machine Scale Set with the REST APIs, Azure PowerShell, and Azure CLI
Learn how to use the Application Health extension to monitor the health of your applications deployed on Virtual Machine Scale Sets.
Upgrade policy modes for Virtual Machine Scale Sets - Azure Virtual Machine Scale Sets
Learn about the different upgrade policies available for Virtual Machine Scale Sets.