Tutorial: Install Azure Container Storage Preview for use with Azure Kubernetes Service

Azure Container Storage is a cloud-based volume management, deployment, and orchestration service built natively for containers. In this tutorial, you'll create an Azure Kubernetes Service (AKS) cluster and install Azure Container Storage on the cluster. If you already have an AKS cluster deployed, we recommend installing Azure Container Storage using this QuickStart instead of following the manual steps in this tutorial.

  • Create a resource group
  • Choose a data storage option and VM type
  • Create an AKS cluster
  • Connect to the cluster
  • Label the node pool
  • Assign Azure Container Storage Operator role to AKS managed identity
  • Install Azure Container Storage

Prerequisites

  • If you don't have an Azure subscription, create a free account before you begin.

  • This article requires the latest version (2.35.0 or later) of the Azure CLI. See How to install the Azure CLI. If you're using the Bash environment in Azure Cloud Shell, the latest version is already installed. If you plan to run the commands locally instead of in Azure Cloud Shell, be sure to run them with administrative privileges. For more information, see Get started with Azure Cloud Shell.

  • You'll need the Kubernetes command-line client, kubectl. It's already installed if you're using Azure Cloud Shell, or you can install it locally by running the az aks install-cli command.

  • Check if your target region is supported in Azure Container Storage regions.

Getting started

  • Take note of your Azure subscription ID. If you want to use Azure Elastic SAN as data storage, you'll need either an Azure Container Storage Owner role or Azure Container Storage Contributor role assigned to the Azure subscription. Owner-level access allows you to install the Azure Container Storage extension, grants access to its storage resources, and gives you permission to configure your Azure Elastic SAN resource. Contributor-level access allows you to install the extension and grants access to its storage resources. If you're planning on using Azure Disks or Ephemeral Disk as data storage, you don't need special permissions on your subscription.

  • Launch Azure Cloud Shell, or if you're using a local installation, sign in to the Azure CLI by using the az login command.

  • If you're using Azure Cloud Shell, you might be prompted to mount storage. Select the Azure subscription where you want to create the storage account and select Create.

Set subscription context

Set your Azure subscription context using the az account set command. You can view the subscription IDs for all the subscriptions you have access to by running the az account list --output table command. Remember to replace <subscription-id> with your subscription ID.

az account set --subscription <subscription-id>

Register resource providers

The Microsoft.ContainerService and Microsoft.KubernetesConfiguration resource providers must be registered on your Azure subscription. To register these providers, run the following command:

az provider register --namespace Microsoft.ContainerService --wait 
az provider register --namespace Microsoft.KubernetesConfiguration --wait 

To check if these providers are registered successfully, run the following command:

az provider list --query "[?namespace=='Microsoft.ContainerService'].registrationState"
az provider list --query "[?namespace=='Microsoft.KubernetesConfiguration'].registrationState"

Create a resource group

An Azure resource group is a logical group that holds your Azure resources that you want to manage as a group. When you create a resource group, you're prompted to specify a location. This location is:

  • The storage location of your resource group metadata.
  • Where your resources will run in Azure if you don't specify another region during resource creation.

Create a resource group using the az group create command. Replace <resource-group-name> with the name of the resource group you want to create, and replace <location> with an Azure region such as eastus, westus2, westus3, or westeurope.

az group create --name <resource-group-name> --location <location>

If the resource group was created successfully, you'll see output similar to this:

{
  "id": "/subscriptions/<guid>/resourceGroups/myContainerStorageRG",
  "location": "eastus",
  "managedBy": null,
  "name": "myContainerStorageRG",
  "properties": {
    "provisioningState": "Succeeded"
  },
  "tags": null
}

Choose a data storage option and virtual machine type

Before you create your cluster, you should understand which back-end storage option you'll ultimately choose to create your storage pool. This is because different storage services work best with different virtual machine (VM) types as cluster nodes, and you'll deploy your cluster before you create the storage pool.

Data storage options

  • Azure Elastic SAN: Azure Elastic SAN is a good fit for general purpose databases, streaming and messaging services, CD/CI environments, and other tier 1/tier 2 workloads. Storage is provisioned on demand per created volume and volume snapshot. Multiple clusters can access a single SAN concurrently, however persistent volumes can only be attached by one consumer at a time.

  • Azure Disks: Azure Disks are a good fit for databases such as MySQL, MongoDB, and PostgreSQL. Storage is provisioned per target container storage pool size and maximum volume size.

  • Ephemeral Disk: This option uses local NVMe or temp SSD drives on the AKS nodes and is extremely latency sensitive (low sub-ms latency), so it's best for applications with no data durability requirement or with built-in data replication support such as Cassandra. AKS discovers the available ephemeral storage on AKS nodes and acquires the drives for volume deployment.

VM types

To use Azure Container Storage, you'll need a node pool of at least three Linux VMs. Each VM should have a minimum of four virtual CPUs (vCPUs). Azure Container Storage will consume one core for I/O processing on every VM the extension is deployed to.

If you intend to use Azure Elastic SAN or Azure Disks with Azure Container Storage, then you should choose a general purpose VM type such as standard_d4s_v5 for the cluster nodes.

If you intend to use Ephemeral Disk with local NVMe, choose a storage optimized VM type such as standard_l8s_v3. If you intend to use Ephemeral Disk with temp SSD, a Ev3 and Esv3-series VM is required.

Important

You must choose a VM type that supports Azure premium storage.

Create AKS cluster

Run the following command to create a Linux-based AKS cluster and enable a system-assigned managed identity. If you already have an AKS cluster you want to use, you can skip this step.

Replace <resource-group> with the name of the resource group you created, <cluster-name> with the name of the cluster you want to create, and <vm-type> with the VM type you selected in the previous step. In this example, we'll create a cluster with three nodes. Increase the --node-count if you want a larger cluster.

az aks create -g <resource-group> -n <cluster-name> --node-count 3 -s <vm-type> --generate-ssh-keys

The deployment will take a few minutes to complete.

Note

When you create an AKS cluster, AKS automatically creates a second resource group to store the AKS resources. This second resource group follows the naming convention MC_YourResourceGroup_YourAKSClusterName_Region. For more information, see Why are two resource groups created with AKS?.

Connect to the cluster

To connect to the cluster, use the Kubernetes command-line client, kubectl. It's already installed if you're using Azure Cloud Shell, or you can install it locally by running the az aks install-cli command.

  1. Configure kubectl to connect to your cluster using the az aks get-credentials command. The following command:

    • Downloads credentials and configures the Kubernetes CLI to use them.
    • Uses ~/.kube/config, the default location for the Kubernetes configuration file. You can specify a different location for your Kubernetes configuration file using the --file argument.
    az aks get-credentials --resource-group <resource-group> --name <cluster-name>
    
  2. Verify the connection to your cluster using the kubectl get command. This command returns a list of the cluster nodes.

    kubectl get nodes
    
  3. The following output example shows the nodes in your cluster. Make sure the status for all nodes shows Ready:

    NAME                                STATUS   ROLES   AGE   VERSION
    aks-nodepool1-34832848-vmss000000   Ready    agent   80m   v1.25.6
    aks-nodepool1-34832848-vmss000001   Ready    agent   80m   v1.25.6
    aks-nodepool1-34832848-vmss000002   Ready    agent   80m   v1.25.6
    

    Take note of the name of your node pool. In this example, it would be nodepool1.

Label the node pool

Next, you must update your node pool label to associate the node pool with the correct IO engine for Azure Container Storage.

Important

If you created your AKS cluster using the Azure portal: The cluster will likely have a user node pool and a system/agent node pool. Before you can install Azure Container Storage, you must update the user node pool label as described in this section. However, if your cluster consists of only a system node pool, which is the case with test/dev clusters created with the Azure portal, you'll need to first add a new user node pool and then label it. This is because when you create an AKS cluster using the Azure portal, a taint CriticalAddOnsOnly is added to the agent/system nodepool, which blocks installation of Azure Container Storage on the system node pool. This taint isn't added when an AKS cluster is created using Azure CLI.

Run the following command to update the node pool label. Remember to replace <resource-group> and <cluster-name> with your own values, and replace <nodepool-name> with the name of your node pool.

az aks nodepool update --resource-group <resource-group> --cluster-name <cluster-name> --name <nodepool-name> --labels acstor.azure.com/io-engine=acstor

You can verify that the node pool is correctly labeled by signing into the Azure portal and navigating to your AKS cluster. Go to Settings > Node pools, select your node pool, and under Taints and labels you should see Labels: acstor.azure.com/io-engine:acstor.

Assign Azure Container Storage Operator role to AKS managed identity

You only need to perform this step if you plan to use Azure Elastic SAN as backing storage. In order to use Elastic SAN, you'll need to grant permissions to allow Azure Container Storage to provision storage for your cluster. Specifically, you must assign the Azure Container Storage Operator role to the AKS managed identity. You can do this using the Azure portal or Azure CLI. You'll need either an Azure Container Storage Owner role or Azure Container Storage Contributor role for your Azure subscription in order to do this. If you don't have sufficient permissions, ask your admin to perform these steps.

  1. Sign in to the Azure portal, and search for and select Kubernetes services.
  2. Locate and select your AKS cluster. Select Settings > Properties from the left navigation.
  3. Under Infrastructure resource group, you should see a link to the resource group that AKS created when you created the cluster. Select it.
  4. Select Access control (IAM) from the left pane.
  5. Select Add > Add role assignment.
  6. Under the Job function roles tab, select or search for Azure Container Storage Operator, then select Next. If you don't have an Azure Container Storage Owner or Azure Container Storage Contributor role on the subscription, you won't be able to add the Azure Container Storage Operator role.
  7. Under Assign access to, select Managed identity.
  8. Under Members, click + Select members. The Select managed identities menu will appear.
  9. Under Managed identity, select User-assigned managed identity.
  10. Under Select, search for and select the managed identity with your cluster name and -agentpool appended.
  11. Click Select, then Review + assign.

Install Azure Container Storage

The initial install uses Azure Arc CLI commands to download a new extension. Replace <cluster-name> and <resource-group> with your own values. The <extension-name> value can be whatever you want; it's just a label for the extension you're installing.

During installation, you might be asked to install the k8s-extension. Select Y.

az k8s-extension create --cluster-type managedClusters --cluster-name <cluster-name> --resource-group <resource-group> --name <extension-name> --extension-type microsoft.azurecontainerstorage --scope cluster --release-train stable --release-namespace acstor

Installation takes 10-15 minutes to complete. You can check if the installation completed correctly by running the following command and ensuring that provisioningState says Succeeded:

az k8s-extension list --cluster-name <cluster-name> --resource-group <resource-group> --cluster-type managedClusters

Congratulations, you've successfully installed Azure Container Storage. You now have new storage classes that you can use for your Kubernetes workloads.

Next step

Now you can create a storage pool and volumes. Follow the steps in the appropriate article depending on the backing storage type you want to use.