Quickstart: Use Azure Container Storage Preview with Azure Kubernetes Service

Article
05/29/2024

Azure Container Storage is a cloud-based volume management, deployment, and orchestration service built natively for containers. This Quickstart shows you how to create a Linux-based Azure Kubernetes Service (AKS) cluster, install Azure Container Storage, and create a storage pool using Azure CLI.

Prerequisites

If you don't have an Azure subscription, create a free account before you begin.
This article requires the latest version (2.35.0 or later) of the Azure CLI. See How to install the Azure CLI. If you're using the Bash environment in Azure Cloud Shell, the latest version is already installed. If you plan to run the commands locally instead of in Azure Cloud Shell, be sure to run them with administrative privileges. For more information, see Get started with Azure Cloud Shell.
You'll need the Kubernetes command-line client, kubectl. It's already installed if you're using Azure Cloud Shell, or you can install it locally by running the az aks install-cli command.
If you haven't already installed Azure Container Storage, follow the instructions in Install Azure Container Storage.
Check if your target region is supported in Azure Container Storage regions.

If you haven't already created an AKS cluster, follow the instructions for Installing an AKS Cluster.

Getting started

Take note of your Azure subscription ID. If you want to use Azure Elastic SAN as data storage, you'll need either an Azure Container Storage Owner role or Azure Container Storage Contributor role assigned to the Azure subscription. Owner-level access allows you to install the Azure Container Storage extension, grants access to its storage resources, and gives you permission to configure your Azure Elastic SAN resource. Contributor-level access allows you to install the extension and grants access to its storage resources. If you're planning on using Azure Disks or Ephemeral Disk as data storage, you don't need special permissions on your subscription.
Launch Azure Cloud Shell, or if you're using a local installation, sign in to Azure by using the az login command.
If you're using Azure Cloud Shell, you might be prompted to mount storage. Select the Azure subscription where you want to create the storage account and select Create.

Install the required extension

Add or upgrade to the latest version of k8s-extension by running the following command.

az extension add --upgrade --name k8s-extension

Set subscription context

Set your Azure subscription context using the az account set command. You can view the subscription IDs for all the subscriptions you have access to by running the az account list --output table command. Remember to replace <subscription-id> with your subscription ID.

az account set --subscription <subscription-id>

Connect to the cluster

To connect to the cluster, use the Kubernetes command-line client, kubectl. It's already installed if you're using Azure Cloud Shell, or you can install it locally by running the az aks install-cli command.

Configure kubectl to connect to your cluster using the az aks get-credentials command. The following command:
- Downloads credentials and configures the Kubernetes CLI to use them.
- Uses ~/.kube/config, the default location for the Kubernetes configuration file. You can specify a different location for your Kubernetes configuration file using the --file argument.
```
az aks get-credentials --resource-group <resource-group> --name <cluster-name>
```
Verify the connection to your cluster using the kubectl get command. This command returns a list of the cluster nodes.
```
kubectl get nodes
```

The following output example shows the nodes in your cluster. Make sure the status for all nodes shows Ready:

NAME                                STATUS   ROLES   AGE   VERSION
aks-nodepool1-34832848-vmss000000   Ready    agent   80m   v1.25.6
aks-nodepool1-34832848-vmss000001   Ready    agent   80m   v1.25.6
aks-nodepool1-34832848-vmss000002   Ready    agent   80m   v1.25.6

Take note of the name of your node pool. In this example, it would be nodepool1.

Choose a data storage option for your storage pool

Before deploying Azure Container Storage, you'll need to decide which back-end storage option you want to use to create your storage pool and persistent volumes. Three options are currently available:

Azure Elastic SAN: Azure Elastic SAN is a good fit for general purpose databases, streaming and messaging services, CI/CD environments, and other tier 1/tier 2 workloads. Storage is provisioned on demand per created volume and volume snapshot. Multiple clusters can access a single SAN concurrently, however persistent volumes can only be attached by one consumer at a time.
Azure Disks: Azure Disks are a good fit for databases such as MySQL, MongoDB, and PostgreSQL. Storage is provisioned per target container storage pool size and maximum volume size.
Ephemeral Disk: This option uses local NVMe drives or temp SSD on the AKS cluster nodes. It's extremely latency sensitive (low sub-ms latency), so it's best for applications with no data durability requirement or with built-in data replication support such as Cassandra. AKS discovers the available ephemeral storage on AKS nodes and acquires the drives for volume deployment.

Note

For Azure Elastic SAN and Azure Disks, Azure Container Storage will deploy the backing storage for you as part of the installation. You don't need to create your own Elastic SAN or Azure Disk. In order to use Elastic SAN, you'll need either an Azure Container Storage Owner role or Azure Container Storage Contributor role on the Azure subscription.

Resource consumption

Azure Container Storage requires certain node resources to run components for the service. Based on your storage pool type selection, which you'll specify when you install Azure Container Storage, these are the resources that will be consumed:

Storage pool type	CPU cores	RAM
Azure Elastic SAN	None	None
Azure Disks	1	1 GiB
Ephemeral Disk - Temp SSD	1	1 GiB
Ephemeral Disk - Local NVMe	25% of cores (depending on node size)*	2 GiB

The resources consumed are per node, and will be consumed for each node in the node pool where Azure Container Storage will be installed. If your nodes don't have enough resources, Azure Container Storage will fail to run. Kubernetes will automatically re-try to initialize these failed pods, so if resources get liberated, these pods can be initialized again.

*In a storage pool type Ephemeral Disk - Local NVMe, if you're using multiple VM SKU types for your cluster nodes, the 25% of CPU cores consumed applies to the smallest SKU used. For example, if you're using a mix of 8-core and 16-core VM types, resource consumption is 2 cores.

Ensure VM type for your cluster meets the following criteria

To use Azure Container Storage, you'll need a node pool of at least three Linux VMs. Each VM should have a minimum of four virtual CPUs (vCPUs). Azure Container Storage will consume one core for I/O processing on every VM the extension is deployed to.

Follow these guidelines when choosing a VM type for the cluster nodes. You must choose a VM type that supports Azure premium storage.

If you intend to use Azure Elastic SAN or Azure Disks as backing storage, choose a general purpose VM type such as standard_d4s_v5.
If you intend to use Ephemeral Disk with local NVMe, choose a storage optimized VM type such as standard_l8s_v3.
If you intend to use Ephemeral Disk with temp SSD, choose a VM that has a temp SSD disk such as Ev3 and Esv3-series.

Install Azure Container Storage on your AKS cluster

If your AKS cluster meets the VM requirements, run the following command to install Azure Container Storage on the cluster and create a storage pool. Replace <cluster-name> and <resource-group> with your own values. Replace <storage-pool-type> with azureDisk, ephemeraldisk, or elasticSan. If you select ephemeralDisk, you can also specify --storage-pool-option, and the values can be NVMe or Temp.

Running this command will enable Azure Container Storage on the system node pool, which by default is named nodepool1*. If you want to enable it on other node pools, see Install Azure Container Storage on specific node pools. If you want to specify additional storage pool parameters, see this table.

*If there are any existing node pools with the acstor.azure.com/io-engine:acstor label then Azure Container Storage will be installed there by default. Otherwise, it's installed on the system node pool.

Important

If you created your AKS cluster using the Azure portal: The cluster will likely have a user node pool and a system/agent node pool. However, if your cluster consists of only a system node pool, which is the case with test/dev clusters created with the Azure portal, you'll need to first add a new user node pool and then label it. This is because when you create an AKS cluster using the Azure portal, a taint CriticalAddOnsOnly is added to the system/agent node pool, which blocks installation of Azure Container Storage on the system node pool. This taint isn't added when an AKS cluster is created using Azure CLI.

az aks update -n <cluster-name> -g <resource-group> --enable-azure-container-storage <storage-pool-type>

The deployment will take 10-15 minutes. When it completes, you'll have an AKS cluster with Azure Container Storage installed, the components for your chosen storage pool type enabled, and a default storage pool. If you want to enable additional storage pool types to create additional storage pools, see Enable additional storage pool types.

Important

If you specified Azure Elastic SAN as backing storage for your storage pool and you don't have either Azure Container Storage Owner role or Azure Container Storage Contributor role assigned to the Azure subscription, only Azure Container Storage will be installed and a storage pool won't be created. In this case, you'll have to create an Elastic SAN storage pool manually.

Install Azure Container Storage on specific node pools

If you want to install Azure Container Storage on specific node pools, follow these instructions. The node pools must contain at least three Linux VMs each.

Run the following command to view the list of available node pools. Replace <resource-group> and <cluster-name> with your own values.
```
az aks nodepool list --resource-group <resource-group> --cluster-name <cluster-name>
```
Run the following command to install Azure Container Storage on specific node pools. Replace <cluster-name> and <resource-group> with your own values. Replace <storage-pool-type> with azureDisk, ephemeraldisk, or elasticSan. If you select ephemeralDisk, you can also specify --storage-pool-option, and the values can be NVMe or Temp.
```
az aks update -n <cluster-name> -g <resource-group> --enable-azure-container-storage <storage-pool-type> --azure-container-storage-nodepools <comma separated values of nodepool names>
```

Enable additional storage pool types

If you want to enable a storage pool type that wasn't originally enabled during installation of Azure Container Storage, run the following command. Replace <cluster-name> and <resource-group> with your own values. For <storage-pool-type>, specify azureDisk, ephemeralDisk, or elasticSan.

If you want to specify additional storage pool parameters with this command, see this table.

az aks update -n <cluster-name> -g <resource-group> --enable-azure-container-storage <storage-pool-type>

If the new storage pool type that you've enabled takes up more resources than the storage pool type that's already enabled, the resource consumption will change to the maximum amount.

Tip

If you've added a new node pool to your cluster and want to run Azure Container Storage on that node pool, you can specify the node pool with --azure-container-storage-nodepools <nodepool-name> when running the az aks update command.

Display available storage pools

To get the list of available storage pools, run the following command:

kubectl get sp -n acstor

To check the status of a storage pool, run the following command:

kubectl describe sp <storage-pool-name> -n acstor

If the Message doesn't say StoragePool is ready, then your storage pool is still creating or ran into a problem. See Troubleshoot Azure Container Storage.

Disable storage pool types

If you're no longer using a specific storage pool type and want to disable it to free up resources in your node pool, run the following command. Replace <cluster-name> and <resource-group> with your own values. For <storage-pool-type>, specify azureDisk, ephemeralDisk, or elasticSan.

az aks update -n <cluster-name> -g <resource-group> --disable-azure-container-storage <storage-pool-type>