Edit

Release notes for the AKS engine on Azure Stack Hub

  • Article
  • 34 minutes to read

Applies to version v0.75.3 of the AKS engine.

This article describes the contents of the Azure Kubernetes Service (AKS) engine on Azure Stack Hub update. The update includes improvements and fixes for the latest release of AKS engine targeted to the Azure Stack Hub platform. Notice that this isn't intended to document the release information for the AKS engine for global Azure.

Upgrade considerations

  • Are you using the correct marketplace items, AKS Base Ubuntu 20.04 Image Distro or AKS Base Windows Server for your version of the AKS engine? You can find the versions in the section Download new images and AKS engine.
  • Are you using the correct cluster specification (apimodel.json) and resource group for the target cluster? When you originally deployed the cluster, this file was generated in your output directory. See the deploy command parameters Deploy a Kubernetes cluster.
  • Are you using a reliable machine to run the AKS engine and from which you're performing upgrade operations?
  • If you're updating an operational cluster with active workloads, you can apply the upgrade without affecting them, assuming the cluster is under normal load. However, you should have a backup cluster in case there's a need to redirect users to it. A backup cluster is highly recommended.
  • If possible, run the command from a VM within the Azure Stack Hub environment to decrease the network hops and potential connectivity failures.
  • Make sure that your subscription has enough quota for the entire process. The process allocates new VMs during the process. The resulting number of VMs would be the same as the original, but plan for a couple more VMs to be created during the process.
  • No system updates or scheduled tasks are planned.
  • Set up a staged upgrade on a cluster that's configured with the same values as the production cluster and test the upgrade there before doing so in your production cluster.

Use the upgrade command

You'll be required to use the AKS engine upgrade command as described in the following article Upgrade a Kubernetes cluster on Azure Stack Hub.

Upgrade interruptions

Sometimes unexpected factors interrupt the upgrade of the cluster. An interruption can occur when the AKS engine reports an error or something happens to the AKS engine execution process. Examine the cause of the interruption, address it, and submit again the same upgrade command to continue the upgrade process. The upgrade command is idempotent and should resume the upgrade of the cluster once resubmitted the command. Normally, interruptions increase the time to complete the update, but shouldn't affect the completion of it.

Estimated upgrade time

The estimated time is between 12 to 15 minutes per VM in the cluster. For example, a 20-node cluster may take approximately to five (5) hours to upgrade.

Download new image and AKS engine

Download the new versions of the AKS base Ubuntu Image and AKS engine.

As explained in the AKS engine for Azure Stack Hub documentation, deploying a Kubernetes cluster requires:

  • The aks-engine binary (required)
  • AKS Base Ubuntu 16.04-LTS Image Distro (deprecated - no longer use, change in API Model to use 20.04 instead)
  • AKS Base Ubuntu 18.04-LTS Image Distro (deprecated - no longer use, change in API Model to use 20.04 instead)
  • AKS Base Ubuntu 20.04-LTS Image Distro (required for Linux agents)
  • AKS Base Windows Server Image (one of the following is required for Windows agents)
    • AKS Base Windows Server Image Containerd
    • AKS Base Windows Server Image Docker

New versions of these are available with this update:

Upgrading Kubernetes clusters created with the Ubuntu 16.04 distro

Starting with AKS Engine v0.63.0, the Ubuntu 16.04 distro is not longer a supported option as the OS reached its end-of-life. For AKS Engine v0.67.0 or later versions, aks-engine upgrade will automatically overwrite the unsupported aks-ubuntu-16.04 distro value with with aks-ubuntu-18.04. For AKS Engine v0.75.3 or later versions, if using Kubernetes v1.24 or above, aks-engine-azurestack upgrade will automatically overwrite the unsupported aks-ubuntu-16.04 distro value with aks-ubuntu-20.04.

Upgrading Kubernetes clusters created with the Ubuntu 18.04 distro

Starting with AKS Engine v0.75.3, the Ubuntu 18.04 distro is not longer a supported option as the OS reached its end-of-life. For AKS Engine v0.75.3 or later versions, aks-engine-azurestack upgrade will automatically overwrite the unsupported aks-ubuntu-18.04 distro value with aks-ubuntu-20.04.

Upgrading Kubernetes clusters created with docker container runtime

In Kubernetes v1.24, the dockershim component was removed from kubelet. As a result, the docker container runtime is no longer a supported option. See Kubernetes v1.24 release notes for more information. For AKS Engine v0.75.3 or later versions, aks-engine-azurestack upgrade will automatically overwrite the unsupported docker containerRuntime value with containerd.

For AKS Engine release v0.75.3, clusters with windows nodes on Kubernetes v1.23 can use the Windows base image with Docker runtime. Clusters with windows nodes on Kubernetes v1.24 can use the Windows base image with Containerd runtime.

AKS engine and Azure Stack version mapping

Azure Stack Hub version AKS engine version
1910 0.43.0, 0.43.1
2002 0.48.0, 0.51.0
2005 0.48.0, 0.51.0, 0.55.0, 0.55.4
2008 0.55.4, 0.60.1
2102 0.60.1, 0.63.0, 0.67.0, 0.67.3
2108 0.63.0, 0.67.0, 0.67.3, 0.70.0, 0.71.0, 0.73.0*, 0.75.3*
2206 0.70.0, 0.71.0, 0.73.0*, 0.75.3*

Note

*Supported. See AKS Engine Version Support policy for more information.

AKS engine and corresponding image mapping

You can find the supported Kubernetes versions for AKS Engine on Azure Stack Hub in the table below. Don't follow the aks-engine get-versions command since the command one also includes the versions supported in global Azure.

AKS engine AKS base image Kubernetes versions API model samples
v0.43.1 AKS Base Ubuntu 16.04-LTS Image Distro, October 2019 (2019.10.24) 1.15.5, 1.15.4, 1.14.8, 1.14.7
v0.48.0 AKS Base Ubuntu 16.04-LTS Image Distro, March 2020 (2020.03.19) 1.15.10, 1.14.7
v0.51.0 AKS Base Ubuntu 16.04-LTS Image Distro, May 2020 (2020.05.13), AKS Base Windows Image (17763.1217.200513) 1.15.12, 1.16.8, 1.16.9 Linux, Windows
v0.55.0 AKS Base Ubuntu 16.04-LTS Image Distro, August 2020 (2020.08.24), AKS Base Windows Image (17763.1397.200820) 1.15.12, 1.16.14, 1.17.11 Linux, Windows
v0.55.4 AKS Base Ubuntu 16.04-LTS Image Distro, September 2020 (2020.09.14), AKS Base Windows Image (17763.1397.200820) 1.15.12, 1.16.14, 1.17.11 Linux, Windows
V0.60.1 AKS Base Ubuntu 16.04-LTS Image Distro, January 2021 (2021.01.28),
AKS Base Ubuntu 18.04-LTS Image Distro, 2021 Q1 (2021.01.28),
AKS Base Windows Image (17763.1697.210129)		 1.16.14, 1.16.15, 1.17.17, 1.18.15 Linux, Windows
v0.63.0 AKS Base Ubuntu 18.04-LTS Image Distro, 2021 Q2 (2021.05.24), AKS Base Windows Image (17763.1935.210520) 1.18.18, 1.19.10, 1.20.6 API Model Samples (Linux, Windows)
v0.67.0 AKS Base Ubuntu 18.04-LTS Image Distro, 2021 Q3 (2021.09.27), AKS Base Windows Image (17763.2213.210927) 1.19.15, 1.20.11 API Model Samples (Linux, Windows)
v0.67.3 AKS Base Ubuntu 18.04-LTS Image Distro, 2021 Q3 (2021.09.27), AKS Base Windows Image (17763.2213.210927) 1.19.15, 1.20.11 API Model Samples (Linux, Windows)
v0.70.0 AKS Base Ubuntu 18.04-LTS Image Distro, 2022 Q2 (2022.04.07), AKS Base Windows Image (17763.2565.220408) 1.21.10*, 1.22.7* API Model Samples (Linux, Windows)
v0.71.0 AKS Base Ubuntu 18.04-LTS Image Distro, 2022 Q3 (2022.08.12), AKS Base Windows Image (17763.3232.220805) 1.22.7*, 1.23.6* API Model Samples (Linux, Windows)
v0.73.0 AKS Base Ubuntu 18.04-LTS Image Distro, 2022 Q4 (2022.11.02), AKS Base Windows Image (17763.3532.221102) 1.22.15*, 1.23.13* API Model Samples (Linux, Windows)
v0.75.3 AKS Base Ubuntu 20.04-LTS Image Distro (2023.032.2), AKS Base Windows Server 2019 Image Docker (17763.3887.20230332), AKS Base Windows Server 2019 Image Containerd (17763.3887.20230332) 1.23.15*, 1.24.9** API Model Samples (Linux, Windows)

Note

*Starting from Kubernetes v1.21, only Cloud Provider for Azure is supported on Azure Stack Hub.

Note

** Starting from Kubernetes v1.24, ONLY the containerd container runtime is supported. Please refer to the section Upgrading Kubernetes clusters created with docker container runtime for more details.

What's new with AKSe 0.75.3

AKS Engine release v0.75.3, and all future AKS Engine releases on Azure Stack Hub, will be from the new aks-engine-azurestack repo. As such, all aks-engine commands should be replaced with aks-engine-azurestack. Commands to get the latest AKS Engine release on Azure Stack Hub have also changed. You can see the new commands in the Create Linux Client and Create Windows Client directions. Please create an issue in the new repo if you find any. 

AKS Engine release v0.75.3 on Azure Stack Hub offers Ubuntu 20.04 LTS as its Linux base image. Starting from this release, Ubuntu 18.04 will no longer be supported. Please refer to the section Upgrading Kubernetes clusters created with the Ubuntu 18.04 Distro for more details. 

Starting from Kubernetes v1.24, only the containerd runtime is supported. Please refer to the section Upgrading Kubernetes clusters created with docker runtime for more details. For AKS Engine release v0.75.3, clusters with windows nodes on Kubernetes v1.23 can use the Windows base image with Docker runtime. Clusters with windows nodes on Kubernetes v1.24 can use the Windows base image with Containerd runtime.

You can find additional features at v0.75.3.

Update planning

The AKS engine upgrade command fully automates the upgrade process of your cluster, it takes care of virtual machines (VMs), networking, storage, Kubernetes, and orchestration tasks. Before applying the update, make sure to review the release note information.

Instructions to use AKS engine 0.70.0 and above

Microsoft upgraded the Azure Cloud Provider in version 0.70.0. The Azure Cloud Provider is a core component shared between AKS Azure and AKS engine on Azure Stack Hub.

To use AKS engine 0.70.0 and above:

  • If you're attempting to create a new Kubernetes cluster for the first time:
    Use the sample API model provided for appropriate version in the AKS engine and corresponding image mapping table.

  • If you're creating a new cluster, but want to use your existing API model:
    Modify your API model by following the Cloud Provider for Azure instructions. Failure to include the new setting will result in a deployment error.

  • If you're using storage volumes:
    Make sure that you're using the AzureDiskCSI driver. Version 0.70.0 and above only support CSI drivers, not the legacy in-tree storage provider. To upgrade, follow the instructions to upgrade while using storage volumes.

Known issues

  • Deploying multiple Kubernetes services in parallel inside a single cluster may lead to an error in the basic load balancer configuration. We recommend deploying one service at the time.
  • Since the aks-engine tool is a share source code repository across Azure and Azure Stack Hub. Examining the many release notes and Pull Requests will lead you to believe that the tool supports other versions of Kubernetes and OS platform beyond the listed above, ignore them and use the version table above as the official guide for this update.
  • AKS Engine v0.67.0 uses the wrong Windows Image when deploying Windows clusters, users should use v0.70.0 to solve this issue.

Reference

This is the complete set of release notes for Azure and Azure Stack Hub combined:

Applies to version v0.67.0 of the AKS engine.

This article describes the contents of the Azure Kubernetes Service (AKS) engine on Azure Stack Hub update. The update includes improvements and fixes for the latest release of AKS engine targeted to the Azure Stack Hub platform. Notice that this isn't intended to document the release information for the AKS engine for global Azure.

Update planning

The AKS engine upgrade command fully automates the upgrade process of your cluster, it takes care of virtual machines (VMs), networking, storage, Kubernetes, and orchestration tasks. Before applying the update, make sure to review the release note information.

Upgrade considerations

  • Are you using the correct marketplace items, AKS Base Ubuntu 16.04-LTS or 18.04 Image Distro or AKS Base Windows Server for your version of the AKS engine? You can find the versions in the section "Download new images and AKS engine".
  • Are you using the correct cluster specification (apimodel.json) and resource group for the target cluster? When you originally deployed the cluster, this file was generated in your output directory. See the deploy command parameters Deploy a Kubernetes cluster.
  • Are you using a reliable machine to run the AKS engine and from which you're performing upgrade operations?
  • If you're updating an operational cluster with active workloads, you can apply the upgrade without affecting them, assuming the cluster is under normal load. However, you should have a backup cluster in case there's a need to redirect users to it. A backup cluster is highly recommended.
  • If possible, run the command from a VM within the Azure Stack Hub environment to decrease the network hops and potential connectivity failures.
  • Make sure that your subscription has enough quota for the entire process. The process allocates new VMs during the process. The resulting number of VMs would be the same as the original, but plan for a couple more VMs to be created during the process.
  • No system updates or scheduled tasks are planned.
  • Set up a staged upgrade on a cluster that's configured with the same values as the production cluster and test the upgrade there before doing so in your production cluster.

Use the upgrade command

You will be required to use the AKS engine upgrade command as described in the following article Upgrade a Kubernetes cluster on Azure Stack Hub.

Upgrade interruptions

Sometimes unexpected factors interrupt the upgrade of the cluster. An interruption can occur when the AKS engine reports an error or something happens to the AKS engine execution process. Examine the cause of the interruption, address it, and submit again the same upgrade command to continue the upgrade process. The upgrade command is idempotent and should resume the upgrade of the cluster once resubmitted the command. Normally, interruptions increase the time to complete the update, but shouldn't affect the completion of it.

Estimated upgrade time

The estimated time is between 12 to 15 minutes per VM in the cluster. For example, a 20-node cluster may take approximately to five (5) hours to upgrade.

Download new image and AKS engine

Download the new versions of the AKS base Ubuntu Image and AKS engine.

As explained in the AKS engine for Azure Stack Hub documentation, deploying a Kubernetes cluster requires:

  • The aks-engine binary (required)
  • AKS Base Ubuntu 16.04-LTS Image Distro (deprecated - no longer use, change in API Model to use 18.04 instead)
  • AKS Base Ubuntu 18.04-LTS Image Distro (required for Linux agents)
  • AKS Base Windows Server Image Distro (required for Windows agents)

New versions of these are available with this update:

Upgrading Kubernetes clusters created with the Ubuntu 16.04 distro

Upgrading Kubernetes clusters created with the Ubuntu 16.04 distro Starting with AKS Engine v0.67.0, the Ubuntu 16.04 distro is not longer a supported option as the OS reached its end-of-life. In order to upgrade a cluster, make sure to set the OS distro to aks-ubuntu-18.04 in your input API model, the one generated by aks-engine deploy and passed as input to aks-engine upgrade.

    "masterProfile": {
        "distro": "aks-ubuntu-18.04"
    },

    "agentPoolProfiles": [{
        "distro": "aks-ubuntu-18.04"
    }]

AKS engine and Azure Stack version mapping

Azure Stack Hub version AKS engine version
1910 0.43.0, 0.43.1
2002 0.48.0, 0.51.0
2005 0.48.0, 0.51.0, 0.55.0, 0.55.4
2008 0.55.4, 0.60.1
2102 0.60.1, 0.63.0, 0.67.0
2108 0.63.0, 0.67.0

Kubernetes version upgrade path in AKS engine v0.67.0

You can find the current version and upgrade version in the following table for Azure Stack Hub. Don't follow the aks-engine get-versions command since the command one also includes the versions supported in global Azure. The following version and upgrade table applies to the AKS engine cluster in Azure Stack Hub.

Current version Upgrade available
1.15.12 1.16.14, 1.16.15
1.16.14 1.16.15, 1.17.17
1.17.11, 1.17.17 1.18.18
1.18.15, 1.18.18 1.19.10
1.19.10 1.19.15, 1.20.11
1.20.6 1.20.11

In the API Model json file, please specify the release and version values under the orchestratorProfile section, for example, if you're planning to deploy Kubernetes 1.17.17, the following two values must be set, (see example kubernetes-azurestack.json):

    -   "orchestratorRelease": "1.17",
    -   "orchestratorVersion": "1.17.17"

AKS engine and corresponding image mapping

AKS engine AKS base image Kubernetes versions API model samples
v0.43.1 AKS Base Ubuntu 16.04-LTS Image Distro, October 2019 (2019.10.24) 1.15.5, 1.15.4, 1.14.8, 1.14.7
v0.48.0 AKS Base Ubuntu 16.04-LTS Image Distro, March 2020 (2020.03.19) 1.15.10, 1.14.7
v0.51.0 AKS Base Ubuntu 16.04-LTS Image Distro, May 2020 (2020.05.13), AKS Base Windows Image (17763.1217.200513) 1.15.12, 1.16.8, 1.16.9 Linux, Windows
v0.55.0 AKS Base Ubuntu 16.04-LTS Image Distro, August 2020 (2020.08.24), AKS Base Windows Image (17763.1397.200820) 1.15.12, 1.16.14, 1.17.11 Linux, Windows
v0.55.4 AKS Base Ubuntu 16.04-LTS Image Distro, September 2020 (2020.09.14), AKS Base Windows Image (17763.1397.200820) 1.15.12, 1.16.14, 1.17.11 Linux, Windows
V0.60.1 AKS Base Ubuntu 16.04-LTS Image Distro, January 2021 (2021.01.28),
AKS Base Ubuntu 18.04-LTS Image Distro, 2021 Q1 (2021.01.28),
AKS Base Windows Image (17763.1697.210129)		 1.16.14, 1.16.15, 1.17.17, 1.18.15 Linux, Windows
v0.63.0 AKS Base Ubuntu 18.04-LTS Image Distro, 2021 Q2 (2021.05.24), AKS Base Windows Image (17763.1935.210520) 1.18.18, 1.19.10, 1.20.6 API Model Samples (Linux, Windows)
v0.67.0 AKS Base Ubuntu 18.04-LTS Image Distro, 2021 Q3 (2021.09.27), AKS Base Windows Image (17763.2213.210927) 1.19.15, 1.20.11 API Model Samples (Linux, Windows)

What's new

If you're interested in participating in a private preview, you can request preview access.

New features include:

  • Support for Kubernetes 1.19.15 and 1.20.11

Known issues

  • Deploying multiple Kubernetes services in parallel inside a single cluster may lead to an error in the basic load balancer configuration. We recommend deploying one service at the time.
  • Since the aks-engine tool is a share source code repository across Azure and Azure Stack Hub. Examining the many release notes and Pull Requests will lead you to believe that the tool supports other versions of Kubernetes and OS platform beyond the listed above, ignore them and use the version table above as the official guide for this update.

Reference

This is the complete set of release notes for Azure and Azure Stack Hub combined:

Applies to version v0.60.1 of the AKS engine.

This article describes the contents of the Azure Kubernetes Service (AKS) engine on Azure Stack Hub update. The update includes improvements and fixes for the latest release of AKS engine targeted to the Azure Stack Hub platform. Notice that this isn't intended to document the release information for the AKS engine for global Azure.

Update planning

The AKS engine upgrade command fully automates the upgrade process of your cluster, it takes care of virtual machines (VMs), networking, storage, Kubernetes, and orchestration tasks. Before applying the update, make sure to review the release note information.

Upgrade considerations

  • Are you using the correct marketplace items, AKS Base Ubuntu 16.04-LTS or 18.04 Image Distro or AKS Base Windows Server for your version of the AKS engine? You can find the versions in the section "Download new images and AKS engine".
  • Are you using the correct cluster specification (apimodel.json) and resource group for the target cluster? When you originally deployed the cluster, this file was generated in your output directory. See the deploy command parameters Deploy a Kubernetes cluster.
  • Are you using a reliable machine to run the AKS engine and from which you're performing upgrade operations?
  • If you're updating an operational cluster with active workloads, you can apply the upgrade without affecting them, assuming the cluster is under normal load. However, you should have a backup cluster in case there's a need to redirect users to it. A backup cluster is highly recommended.
  • If possible, run the command from a VM within the Azure Stack Hub environment to decrease the network hops and potential connectivity failures.
  • Make sure that your subscription has enough quota for the entire process. The process allocates new VMs during the process. The resulting number of VMs would be the same as the original, but plan for a couple more VMs to be created during the process.
  • No system updates or scheduled tasks are planned.
  • Set up a staged upgrade on a cluster that's configured with the same values as the production cluster and test the upgrade there before doing so in your production cluster.

Use the upgrade command

You will be required to use the AKS engine upgrade command as described in the following article Upgrade a Kubernetes cluster on Azure Stack Hub.

Upgrade interruptions

Sometimes unexpected factors interrupt the upgrade of the cluster. An interruption can occur when the AKS engine reports an error or something happens to the AKS engine execution process. Examine the cause of the interruption, address it, and submit again the same upgrade command to continue the upgrade process. The upgrade command is idempotent and should resume the upgrade of the cluster once resubmitted the command. Normally, interruptions increase the time to complete the update, but shouldn't affect the completion of it.

Estimated upgrade time

The estimated time is between 12 to 15 minutes per VM in the cluster. For example, a 20-node cluster may take approximately to five (5) hours to upgrade.

Download new image and AKS engine

Download the new versions of the AKS base Ubuntu Image and AKS engine.

As explained in the AKS engine for Azure Stack Hub documentation, deploying a Kubernetes cluster requires:

  • The aks-engine binary (required)
  • AKS Base Ubuntu 16.04-LTS Image Distro (required)
  • AKS Base Ubuntu 18.04-LTS Image Distro (optional)
  • AKS Base Windows Server Image Distro (optional)

New versions of these are available with this update:

AKS engine and Azure Stack version mapping

Azure Stack Hub version AKS engine version
1910 0.43.0, 0.43.1
2002 0.48.0, 0.51.0
2005 0.48.0, 0.51.0, 0.55.0, 0.55.4
2008 0.55.4, 0.60.1

Kubernetes version upgrade path in AKS engine v0.60.1

You can find the current version and upgrade version in the following table for Azure Stack Hub. Don't follow the aks-engine get-versions command since the command one also includes the versions supported in global Azure. The following version and upgrade table applies to the AKS engine cluster in Azure Stack Hub.

Current version Upgrade available
1.15.12 1.16.14, 1.16.15
1.16.14 1.16.15, 1.17.17
1.17.11 1.17.17, 1.18.15
1.17.17 1.18.15

In the API Model json file, please specify the release and version values under the orchestratorProfile section, for example, if you're planning to deploy Kubernetes 1.17.17, the following two values must be set, (see example kubernetes-azurestack.json):

    -   "orchestratorRelease": "1.17",
    -   "orchestratorVersion": "1.17.17"

AKS engine and corresponding image mapping

AKS engine AKS base image Kubernetes versions API model samples
v0.43.1 AKS Base Ubuntu 16.04-LTS Image Distro, October 2019 (2019.10.24) 1.15.5, 1.15.4, 1.14.8, 1.14.7
v0.48.0 AKS Base Ubuntu 16.04-LTS Image Distro, March 2020 (2020.03.19) 1.15.10, 1.14.7
v0.51.0 AKS Base Ubuntu 16.04-LTS Image Distro, May 2020 (2020.05.13), AKS Base Windows Image (17763.1217.200513) 1.15.12, 1.16.8, 1.16.9 Linux, Windows
v0.55.0 AKS Base Ubuntu 16.04-LTS Image Distro, August 2020 (2020.08.24), AKS Base Windows Image (17763.1397.200820) 1.15.12, 1.16.14, 1.17.11 Linux, Windows
v0.55.4 AKS Base Ubuntu 16.04-LTS Image Distro, September 2020 (2020.09.14), AKS Base Windows Image (17763.1397.200820) 1.15.12, 1.16.14, 1.17.11 Linux, Windows
V0.60.1 AKS Base Ubuntu 16.04-LTS Image Distro, January 2021 (2021.01.28),
AKS Base Ubuntu 18.04-LTS Image Distro, 2021 Q1 (2021.01.28),
AKS Base Windows Image (17763.1697.210129)		 1.16.14, 1.16.15, 1.17.17, 1.18.15 Linux, Windows

What's new

If you're interested in participating in a private preview, you can request preview access.

New features include:

  • General Availability of Ubuntu 18.04
  • Certificate Rotation Public Preview #4214
  • T4 Nvidia GPU Private Preview #4259
  • Azure Active Directory integration private preview
  • CSI Driver for Azure Blobs Private Preview #712
  • CSI Driver Azure Disks Public Preview #712
  • CSI Driver NFS Public Preview #712
  • Support for Kubernetes 1. 17.17 #4188 and 1.18.15 #4187

Known issues

  • Deploying multiple Kubernetes services in parallel inside a single cluster may lead to an error in the basic load balancer configuration. We recommend deploying one service at the time.
  • Since the aks-engine tool is a share source code repository across Azure and Azure Stack Hub. Examining the many release notes and Pull Requests will lead you to believe that the tool supports other versions of Kubernetes and OS platform beyond the listed above, ignore them and use the version table above as the official guide for this update.

Reference

This is the complete set of release notes for Azure and Azure Stack Hub combined:

Applies to version v0.55.4 of the AKS engine.

This article describes the contents of the Azure Kubernetes Service (AKS) engine on Azure Stack Hub update. The update includes improvements and fixes for the latest release of AKS engine targeted to the Azure Stack Hub platform. Notice that this isn't intended to document the release information for the AKS engine for global Azure.

Update planning

The AKS engine upgrade command fully automates the upgrade process of your cluster, it takes care of virtual machines (VMs), networking, storage, Kubernetes, and orchestration tasks. Before applying the update, make sure to review the release note information.

Upgrade considerations

  • Are you using the correct marketplace item, AKS Base Ubuntu 16.04-LTS Image Distro for your version of the AKS engine? You can find the versions in the section "Download new image and AKS engine".

  • Are you using the correct cluster specification (apimodel.json) and resource group for the target cluster? When you originally deployed the cluster, this file was generated in your output directory. See the deploy command parameters Deploy a Kubernetes cluster.

  • Are you using a reliable machine to run the AKS engine and from which you're performing upgrade operations?

  • If you're updating an operational cluster with active workloads you can apply the upgrade without affecting them, assuming the cluster is under normal load. However, you should have a backup cluster in case there's a need to redirect users to it. A backup cluster is highly recommended.

  • If possible, run the command from a VM within the Azure Stack Hub environment to decrease the network hops and potential connectivity failures.

  • Make sure that your subscription has enough quota for the entire process. The process allocates new VMs during the process. The resulting number of VMs would be the same as the original, but plan for a couple more VMs to be created during the process.

  • No system updates or scheduled tasks are planned.

  • Set up a staged upgrade on a cluster that's configured with the same values as the production cluster and test the upgrade there before doing so in your production cluster.

Use the upgrade command

You will be required to use the AKS engine upgrade command as described in the following article Upgrade a Kubernetes cluster on Azure Stack Hub.

Upgrade interruptions

Sometimes unexpected factors interrupt the upgrade of the cluster. An interruption can occur when the AKS engine reports an error or something happens to the AKS engine execution process. Examine the cause of the interruption, address it, and submit again the same upgrade command to continue the upgrade process. The upgrade command is idempotent and should resume the upgrade of the cluster once resubmitted the command. Normally, interruptions increase the time to complete the update, but shouldn't affect the completion of it.

Estimated upgrade time

The estimated time is between 12 to 15 minutes per VM in the cluster. For example, a 20-node cluster may take approximately to five (5) hours to upgrade.

Download new image and AKS engine

Download the new versions of the AKS base Ubuntu Image and AKS engine.

As explained in the AKS engine for Azure Stack Hub documentation, deploying a Kubernetes cluster requires two main components:

  • The aks-engine binary

  • AKS Base Ubuntu 16.04-LTS Image Distro

New versions of these are available with this update:

AKS engine and Azure Stack version mapping

Azure Stack Hub version AKS engine version
1910 0.43.0, 0.43.1
2002 0.48.0, 0.51.0
2005 0.48.0, 0.51.0, 0.55.0, 0.55.4

Kubernetes version upgrade path in AKS engine v0.55.4

You can find the current version and upgrade version in the following table for Azure Stack Hub. Don't follow the aks-engine get-versions command since the command one also includes the versions supported in global Azure. The following version and upgrade table applies to the AKS engine cluster in Azure Stack Hub.

Current version Upgrade available
1.15.10 1.15.12
1.15.12, 1.16.8, 1.16.9 1.16.14
1.16.8, 1.16.9, 1.16.14 1.17.11

In the API Model json file, please specify the release and version values under the orchestratorProfile section, for example, if you're planning to deploy Kubernetes 1.16.14, the following two values must be set, (see example kubernetes-azurestack.json):

    -   "orchestratorRelease": "1.16",
    -   "orchestratorVersion": "1.16.14"

AKS engine and corresponding image mapping

Kubernetes versions Notes
1.15.5, 1.15.4, 1.14.8, 1.14.7
1.15.10, 1.14.7
1.15.12, 1.16.8, 1.16.9 API Model Samples (Linux, Windows)
1.15.12, 1.16.14, 1.17.11 API Model Samples (Linux, Windows)
1.15.12, 1.16.14, 1.17.11 API Model Samples (Linux, Windows)

What's new

  • Update Azure Stack's Linux VHD to 2020.09.14 #3828
  • Adds support for K8s v1.17.11 on Azure Stack #3702
  • Adds support for K8s v1.16.14 on Azure Stack #3704
  • Linux VHD update to 2020.09.14 #3750
  • Windows VHD update to August #3730
  • Upgrades Kubernetes metrics-server to v0.3.7 #3669
  • Upgrades docker version to fix log rotation issue #3693
  • Upgrades CoreDNS to v1.7.0 #3608
  • Use moby 19.03.x packages #3549
  • Fixes to azure-cni update strategy #3571

Known issues

  • Deploying multiple Kubernetes services in parallel inside a single cluster may lead to an error in the basic load balancer configuration. Deploying one service at the time if possible.
  • Running aks-engine get-versions will produce information applicable to Azure and Azure Stack Hub, however, there's not explicit way to discern what corresponds to Azure Stack Hub. Don't use this command to figure out what versions are available to upgrade. Use the upgrade reference table described above.
  • Since aks-engine tool is a share source code repository across Azure and Azure Stack Hub. Examining the many release notes and Pull Requests will lead you to believe that the tool supports other versions of Kubernetes and OS platform beyond the listed above, ignore them and use the version table above as the official guide for this update.

Reference

This is the complete set of release notes for Azure and Azure Stack Hub combined:

Applies to version 0.48.0 or earlier of the AKS engine.

This article describes the contents of the Azure Kubernetes Service (AKS) engine on Azure Stack Hub update. The update includes improvements and fixes for the latest release of AKS engine targeted to the Azure Stack Hub platform. Notice that this isn't intended to document the release information for the AKS engine for global Azure.

Update planning

The AKS engine upgrade command fully automates the upgrade process of your cluster, it takes care of virtual machines (VMs), networking, storage, Kubernetes, and orchestration tasks. Before applying the update, make sure to review the information in this article.

Upgrade considerations

  • Are you using the correct marketplace item, AKS Base Ubuntu 16.04-LTS Image Distro for your version of the AKS engine? You can find the versions in the section Download the new AKS base Ubuntu Image and AKS engine versions.

  • Are you using the correct cluster specification (apimodel.json) and resource group for the target cluster? When you originally deployed the cluster, this file was generated in your output directory. See the "deploy" command parameters Deploy a Kubernetes cluster.

  • Are you using a reliable machine to run the AKS engine and from which you're performing upgrade operations?

  • If you're updating an operational cluster with active workloads you can apply the upgrade without affecting them, assuming the cluster is under normal load. However, you should have a backup cluster in case there's a need to redirect users to it.

  • If possible, run the command from a VM within the Azure Stack Hub environment to decrease the network hops and potential connectivity failures.

  • Make sure that your subscription has enough quota for the entire process. The process allocates new VMs during the process. The resulting number of VMs would be the same as the original, but plan for a couple more VMs to be created during the process.

  • No system updates or scheduled tasks are planned.

  • Set up a staged upgrade on a cluster that's configured with the same values as the production cluster and test the upgrade there before doing so in your production cluster.

Use the upgrade command

You will be required to use the AKS engine "upgrade" command as described in the following article Upgrade a Kubernetes cluster on Azure Stack Hub.

Upgrade interruptions

Sometimes unexpected factors interrupt the upgrade of the cluster. An interruption can occur when the AKS engine reports an error or something happens to the AKS engine execution process. Examine the cause of the interruption, address it, and submit again the same upgrade command to continue the upgrade process. The upgrade command is idempotent and should resume the upgrade of the cluster once resubmitted the command. Normally, interruptions increase the time to complete the update, but shouldn't affect the completion of it.

Estimated upgrade time

The estimated time is between 12 to 15 minutes per VM in the cluster. For example, a 20-node cluster may take approximately to five (5) hours to upgrade.

Download new image and AKS engine

Download the new versions of the AKS base Ubuntu Image and AKS engine.

As explained in the AKS engine for Azure Stack Hub documentation, deploying a Kubernetes cluster requires two main components:

  • The aks-engine binary
  • AKS Base Ubuntu 16.04-LTS Image Distro

New versions of these are available with this update:

Kubernetes version upgrade path

You can find the current version and upgrade version in the following table for Azure Stack Hub. Don't follow the aks-engine get-versions command since the command one also includes the versions supported in global Azure. The following version and upgrade table applies to the AKS engine cluster in Azure Stack Hub.

Current version Upgrade available
1.14.7 1.15.10
1.14.8 1.15.10
1.15.4 1.15.10
1.15.5 1.15.10

What's new

  • Support for Kubernetes version 1.15.10 (#2834). When deploying a new cluster make sure to specify in your api model json file (a.k.s. cluster definition file) the release version number as well as the minor version number. You can find an example: kubernetes-azurestack.json:

    • "orchestratorRelease": "1.15,

    • "orchestratorVersion": "1.15.10"

    Note

    If the Kubernetes version is not explicitly provided in the API model json file, version 1.15 will be used (#2932) and the orchestratorVersion will default to 1.15.11, which will result in an error during deployment of the cluster.

  • With aks-engine v0.43.1, the default frequency settings for the cloud provider to perform its control loop and other tasks don't work well with Azure Stack Hub Resource Manager threshold limits for incoming requests. This update changes defaults for Azure Stack Hub to reduce the retry load to Azure Stack Hub Resource Manager (#2861).

  • New verification step in aks-engine will result in either execution stopping or displaying warnings if api model json file contains properties not supported by Azure Stack Hub (#2717).

  • With a new verification check-in, the aks-engine will validate availability of the version of the AKS base image needed for the version of aks-engine executing (#2342). This will occur after parsing the api model file and before calling the Azure Stack Hub Resource Manager.

  • New aks-engine option "--control-plane-only" in the "upgrade" command allows the user to upgrade operations to target only the master Virtual Machines (#2635).

  • Updates to Linux Kernel version 4.15.0-1071-azure for Ubuntu 16.04-LTS. See "Package: linux-image-4.15.0-1071-azure (4.15.0-1071.76) [security]" for details.

  • New hyperkube updates to support Kubernetes versions 1.14.8 and 1.15.10.

  • Update kubectl to match the version of Kubernetes for the cluster.. This component is available in the Kubernetes cluster control plane nodes, you can run it by SSH into a master.

  • Updates for the Azure Container Monitor add-on with latest February 2020 release (#2850).

  • Upgrade of coredns to version v1.6.6 (#2555).

  • Upgrade etcd to version 3.3.18 (#2462).

  • Upgrade moby to version 3.0.11 (#2887).

  • With this release AKS Engine cuts dependency from k8s.gcr.io to now use the official Kubernetes MCR registry @ mcr.microsoft.com when building its images (#2722).

Known issues

  • Deploying multiple Kubernetes services in parallel inside a single cluster may lead to an error in the basic load balancer configuration. Deploying one service at the time if possible.

  • Running aks-engine get-versions will produce information applicable to Azure and Azure Stack Hub, however, there's not explicit way to discern what corresponds to Azure Stack Hub. Don't use this command to figure out what versions area available to upgrade. Use the upgrade reference table described above.

  • Since aks-engine tool is a share source code repository across Azure and Azure Stack Hub. Examining the many release notes and Pull Requests will lead you to believe that the tool supports other versions of Kubernetes and OS platform beyond the listed above, please ignore them and use the version table above as the official guide for this update.

Reference

Following is a list of some of the bugs fixed as well as the complete set of release notes from 0.44.0 to 0.48.0, notice that the latter list will include Azure and Azure Stack Hub items

Bug fixes

  • userAssignedIdentityId in windows azure.json missing quotes (#2327)

  • Addons update config is upgrade-only (#2282)

  • Bumping timeout for getting management IP on windows nodes (#2284)

  • Add 1.0.28 Azure CNI zip file into windows VHD (#2268)

  • Correct defaults order for setting IPAddressCount (#2358)

  • Update to use single omsagent yaml for all k8s versions to avoid any manual errors and easy maintainability (#2692)

Release notes

This is the complete set of release notes for Azure and Azure Stack Hub combined

Next steps