Orchestrate updates across multiple clusters by using Azure Kubernetes Fleet Manager

Platform admins managing Kubernetes fleets with large number of clusters often have problems with staging their updates in a safe and predictable way across multiple clusters. To address this pain point, Kubernetes Fleet Manager (Fleet) allows you to orchestrate updates across multiple clusters using update runs, stages, groups, and strategies.

Prerequisites

• Read the conceptual overview of this feature, which provides an explanation of update strategies, runs, stages, and groups references in this document.

• You must have a fleet resource with one or more member clusters. If not, follow the quickstart to create a Fleet resource and join Azure Kubernetes Service (AKS) clusters as members. This walkthrough demonstrates a fleet resource with five AKS member clusters as an example.

• Set the following environment variables:

  export GROUP=<resource-group>
  export FLEET=<fleet-name>
  

• If you're following the Azure CLI instructions in this article, you need Azure CLI version 2.53.1 or later installed. To install or upgrade, see Install the Azure CLI.

• You also need the fleet Azure CLI extension, which you can install by running the following command:

  az extension add --name fleet
  

  Run the following command to update to the latest version of the extension released:

  az extension update --name fleet
  

Update all clusters one by one

Azure portal

1. On the page for your Azure Kubernetes Fleet Manager resource, go to the Multi-cluster update menu and select Create.

2. You can choose either One by one or Stages.

   

3. For upgrade scope, you can choose to either update both the Kubernetes version and the node image version or you can update only your Node image version only.

   

   For the node image, the following options are available:

   • Latest: Updates every AKS cluster in the update run to the latest image available for that cluster in its region.
   • Consistent: As it's possible for an update run to have AKS clusters across multiple regions where the latest available node images can be different (check release tracker for more information). The update run picks the latest common image across all these regions to achieve consistency.

Azure CLI

Run the following command to update the Kubernetes version and the node image version for all clusters of the fleet one by one:

az fleet updaterun create --resource-group $GROUP --fleet-name $FLEET --name run-1 --upgrade-type Full --kubernetes-version 1.26.0

Note

The --upgrade-type flag supports the values Full or NodeImageOnly. Full updates both the node images and the Kubernetes version. --node-image-selection supports the values Latest and Consistent.

• Latest: Updates every AKS cluster in the update run to the latest image available for that cluster in its region.
• Consistent: As it's possible for an update run to have AKS clusters across multiple regions where the latest available node images can be different (check release tracker for more information). The update run picks the latest common image across all these regions to achieve consistency.

Run the following command to update only the node image versions for all clusters of the fleet one by one:

az fleet updaterun create --resource-group $GROUP --fleet-name $FLEET --name run-2 --upgrade-type NodeImageOnly

Update clusters in a specific order

Update groups and stages provide more control over the sequence that update runs follow when you're updating the clusters. Within an update stage, updates are applied to all the different update groups in parallel; within an update group, member clusters update sequentially.

Assign a cluster to an update group

You can assign a member cluster to a specific update group in one of two ways.

• Assign to group when adding member cluster to the fleet. For example:

Azure portal

1. On the page for your Azure Kubernetes Fleet Manager resource, go to Member clusters.

   

2. Specify the update group that the member cluster should belong to.

   

Azure CLI

az fleet member create --resource-group $GROUP --fleet-name $FLEET --name member1 --member-cluster-id $AKS_CLUSTER_ID --update-group group-1a

• The second method is to assign an existing fleet member to an update group. For example:

Azure portal

1. On the page for your Azure Kubernetes Fleet Manager resource, navigate to Member clusters. Choose the member clusters that you want, and then select Assign update group.

   

2. Specify the group name, and then select Assign.

   

Azure CLI

az fleet member update --resource-group $GROUP --fleet-name $FLEET --name member1 --update-group group-1a

Note

Any fleet member can only be a part of one update group, but an update group can have multiple fleet members inside it. An update group itself is not a separate resource type. Update groups are only strings representing references from the fleet members. So, if all fleet members with references to a common update group are deleted, that specific update group will cease to exist as well.

Define an update run and stages

You can define an update run using update stages in order to sequentially order the application of updates to different update groups. For example, a first update stage might update test environment member clusters, and a second update stage would then subsequently update production environment member clusters. You can also specify a wait time between the update stages.

Azure portal

1. On the page for your Azure Kubernetes Fleet Manager resource, navigate to Multi-cluster update and select Create.

2. Select Stages, and then choose either Node image (latest) + Kubernetes version or Node image (latest), depending on your desired upgrade scope.

3. Under Stages, select Create Stage. You can now specify the stage name and the duration to wait after each stage.

   

4. Choose the update groups that you want to include in this stage.

   

5. After you define all your stages and order them by using the Move up and Move down controls, proceed with creating the update run.

6. In the Multi-cluster update menu, choose the update run and select Start.

Azure CLI

1. Run the following command to create the update run:

   az fleet updaterun create --resource-group $GROUP --fleet-name $FLEET --name run-3 --upgrade-type Full --kubernetes-version 1.26.0 --stages example-stages.json
   

   Here's an example of input from the stages file (example-stages.json):

   {
       "stages": [
           {
               "name": "stage1",
               "groups": [
                   {
                       "name": "group-1a"
                   },
                   {
                       "name": "group-1b"
                   },
                   {
                       "name": "group-1c"
                   }
               ],
               "afterStageWaitInSeconds": 3600
           },
           {
               "name": "stage2",
               "groups": [
                   {
                       "name": "group-2a"
                   },
                   {
                       "name": "group-2b"
                   },
                   {
                       "name": "group-2c"
                   }
               ]
           }
       ]
   }
   

2. Run the following command to start this update run:

   az fleet updaterun start --resource-group $GROUP --fleet-name $FLEET --name run-3
   

Create an update run using update strategies

In the previous section, creating an update run required the stages, groups, and their order to be specified each time. Update strategies simplify this by allowing you to store templates for update runs.

Note

It is possible to create multiple update runs with unique names from the same update strategy.

Azure portal

When creating your update runs, you are given an option to create an update strategy at the same time, effectively saving the run as a template for subsequent update runs.

1. Save an update strategy while creating an update run:

   

2. The update strategy you created can later be referenced when creating new subsequent update runs:

   

Azure CLI

1. Run the following command to create a new update strategy:

   az fleet updatestrategy create --resource-group $GROUP --fleet-name $FLEET --name strategy-1 --stages example-stages.json
   

2. Run the following command to create an update run referencing this strategy:

   az fleet updaterun create --resource-group $GROUP --fleet-name $FLEET --name run-4 --update-strategy-name strategy-1 --upgrade-type NodeImageOnly --node-image-selection Consistent