Provision Azure NetApp Files volumes on Azure Kubernetes Service

Article
03/09/2023
11 minutes to read

A persistent volume represents a piece of storage that has been provisioned for use with Kubernetes pods. A persistent volume can be used by one or many pods, and can be dynamically or statically provisioned. This article shows you how to create Azure NetApp Files volumes to be used by pods on an Azure Kubernetes Service (AKS) cluster.

Azure NetApp Files is an enterprise-class, high-performance, metered file storage service running on Azure. Kubernetes users have two options for using Azure NetApp Files volumes for Kubernetes workloads:

Create Azure NetApp Files volumes statically. In this scenario, the creation of volumes is external to AKS. Volumes are created using the Azure CLI or from the Azure portal, and are then exposed to Kubernetes by the creation of a PersistentVolume. Statically created Azure NetApp Files volumes have many limitations (for example, inability to be expanded, needing to be over-provisioned, and so on). Statically created volumes are not recommended for most use cases.
Create Azure NetApp Files volumes on-demand, orchestrating through Kubernetes. This method is the preferred way to create multiple volumes directly through Kubernetes, and is achieved using Astra Trident. Astra Trident is a CSI-compliant dynamic storage orchestrator that helps provision volumes natively through Kubernetes.

Using a CSI driver to directly consume Azure NetApp Files volumes from AKS workloads is the recommended configuration for most use cases. This requirement is accomplished using Astra Trident, an open-source dynamic storage orchestrator for Kubernetes. Astra Trident is an enterprise-grade storage orchestrator purpose-built for Kubernetes, and fully supported by NetApp. It simplifies access to storage from Kubernetes clusters by automating storage provisioning.

You can take advantage of Astra Trident's Container Storage Interface (CSI) driver for Azure NetApp Files to abstract underlying details and create, expand, and snapshot volumes on-demand. Also, using Astra Trident enables you to use Astra Control Service built on top of Astra Trident. Using the Astra Control Service, you can backup, recover, move, and manage the application-data lifecycle of your AKS workloads across clusters within and across Azure regions to meet your business and service continuity needs.

Before you begin

The following considerations apply when you use Azure NetApp Files:

Your AKS cluster must be in a region that supports Azure NetApp Files.
The Azure CLI version 2.0.59 or higher installed and configured. Run az --version to find the version. If you need to install or upgrade, see Install Azure CLI.
After the initial deployment of an AKS cluster, you can choose to provision Azure NetApp Files volumes statically or dynamically.
To use dynamic provisioning with Azure NetApp Files, install and configure Astra Trident version 19.07 or higher.

Configure Azure NetApp Files

Register the Microsoft.NetApp resource provider by running the following command:
```
az provider register --namespace Microsoft.NetApp --wait
```
Note

This operation can take several minutes to complete.
When you create an Azure NetApp account for use with AKS, you can create the account in an existing resource group or create a new one in the same region as the AKS cluster. The following command creates an account named myaccount1 in the myResourceGroup resource group and eastus region:
```
az netappfiles account create \
    --resource-group myResourceGroup \
    --location eastus \
    --account-name myaccount1
```

Create a new capacity pool by using az netappfiles pool create. The following example creates a new capacity pool named mypool1 with 4 TB in size and Premium service level:

az netappfiles pool create \
    --resource-group myResourceGroup \
    --location eastus \
    --account-name myaccount1 \
    --pool-name mypool1 \
    --size 4 \
    --service-level Premium

Create a subnet to delegate to Azure NetApp Files using az network vnet subnet create. Specify the resource group hosting the existing virtual network for your AKS cluster.

Note

This subnet must be in the same virtual network as your AKS cluster.

RESOURCE_GROUP=myResourceGroup
VNET_NAME=$(az network vnet list --resource-group $RESOURCE_GROUP --query [].name -o tsv)
VNET_ID=$(az network vnet show --resource-group $RESOURCE_GROUP --name $VNET_NAME --query "id" -o tsv)
SUBNET_NAME=MyNetAppSubnet
az network vnet subnet create \
    --resource-group $RESOURCE_GROUP \
    --vnet-name $VNET_NAME \
    --name $SUBNET_NAME \
    --delegations "Microsoft.NetApp/volumes" \
    --address-prefixes 10.0.0.0/28

Volumes can either be provisioned statically or dynamically. Both options are covered further in the next sections.

Provision Azure NetApp Files volumes statically

Create a volume using the az netappfiles volume create command. Update RESOURCE_GROUP, LOCATION, ANF_ACCOUNT_NAME (Azure NetApp account name), POOL_NAME, and SERVICE_LEVEL with the correct values.

RESOURCE_GROUP=myResourceGroup
LOCATION=eastus
ANF_ACCOUNT_NAME=myaccount1
POOL_NAME=mypool1
SERVICE_LEVEL=Premium
VNET_NAME=$(az network vnet list --resource-group $RESOURCE_GROUP --query [].name -o tsv)
VNET_ID=$(az network vnet show --resource-group $RESOURCE_GROUP --name $VNET_NAME --query "id" -o tsv)
SUBNET_NAME=MyNetAppSubnet
SUBNET_ID=$(az network vnet subnet show --resource-group $RESOURCE_GROUP --vnet-name $VNET_NAME --name $SUBNET_NAME --query "id" -o tsv)
VOLUME_SIZE_GiB=100 # 100 GiB
UNIQUE_FILE_PATH="myfilepath2" # Note that file path needs to be unique within all ANF Accounts

az netappfiles volume create \
    --resource-group $RESOURCE_GROUP \
    --location $LOCATION \
    --account-name $ANF_ACCOUNT_NAME \
    --pool-name $POOL_NAME \
    --name "myvol1" \
    --service-level $SERVICE_LEVEL \
    --vnet $VNET_ID \
    --subnet $SUBNET_ID \
    --usage-threshold $VOLUME_SIZE_GiB \
    --file-path $UNIQUE_FILE_PATH \
    --protocol-types "NFSv3"

Create the persistent volume

List the details of your volume using az netappfiles volume show

az netappfiles volume show \
    --resource-group $RESOURCE_GROUP \
    --account-name $ANF_ACCOUNT_NAME \
    --pool-name $POOL_NAME \
    --volume-name "myvol1" -o JSON

The following output resembles the output of the previous command:

{
  ...
  "creationToken": "myfilepath2",
  ...
  "mountTargets": [
    {
      ...
      "ipAddress": "10.0.0.4",
      ...
    }
  ],
  ...
}

Create a pv-nfs.yaml defining a persistent volume by copying the following manifest. Replace path with the creationToken and server with ipAddress from the previous step.

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-nfs
spec:
  capacity:
    storage: 100Gi
  accessModes:
    - ReadWriteMany
  mountOptions:
    - vers=3
  nfs:
    server: 10.0.0.4
    path: /myfilepath2

Create the persistent volume using the kubectl apply command:
```
kubectl apply -f pv-nfs.yaml
```
Verify the Status of the PersistentVolume is Available using the kubectl describe command:
```
kubectl describe pv pv-nfs
```

Create a persistent volume claim

Create a pvc-nfs.yaml defining a PersistentVolume by copying the following manifest:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-nfs
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: ""
  resources:
    requests:
      storage: 1Gi

Create the persistent volume claim using the kubectl apply command:
```
kubectl apply -f pvc-nfs.yaml
```
Verify the Status of the persistent volume claim is Bound using the kubectl describe command:
```
kubectl describe pvc pvc-nfs
```

Mount with a pod

Create a nginx-nfs.yaml defining a pod that uses the persistent volume claim by using the following manifest:

kind: Pod
apiVersion: v1
metadata:
  name: nginx-nfs
spec:
  containers:
  - image: mcr.microsoft.com/oss/nginx/nginx:1.15.5-alpine
    name: nginx-nfs
    command:
    - "/bin/sh"
    - "-c"
    - while true; do echo $(date) >> /mnt/azure/outfile; sleep 1; done
    volumeMounts:
    - name: disk01
      mountPath: /mnt/azure
  volumes:
  - name: disk01
    persistentVolumeClaim:
      claimName: pvc-nfs

Create the pod using the kubectl apply command:
```
kubectl apply -f nginx-nfs.yaml
```
Verify the pod is Running using the kubectl describe command:
```
kubectl describe pod nginx-nfs
```

Verify your volume has been mounted on the pod by using kubectl exec to connect to the pod, and then use df -h to check if the volume is mounted.

kubectl exec -it nginx-nfs -- sh

/ # df -h
Filesystem             Size  Used Avail Use% Mounted on
...
10.0.0.4:/myfilepath2  100T  384K  100T   1% /mnt/azure
...

Provision Azure NetApp Files volumes dynamically

Install and configure Astra Trident

To dynamically provision volumes, you need to install Astra Trident. Astra Trident is NetApp's dynamic storage provisioner that is purpose-built for Kubernetes. Simplify the consumption of storage for Kubernetes applications using Astra Trident's industry-standard Container Storage Interface (CSI) driver. Astra Trident deploys on Kubernetes clusters as pods and provides dynamic storage orchestration services for your Kubernetes workloads.

Before proceeding to the next section, you need to:

Install Astra Trident. Trident can be installed using the Trident operator (manually or using Helm) or tridentctl. The instructions provided later in this article explain how Astra Trident can be installed using the operator. To learn more about these installation methods and how they work, see the Install Guide.
Create a backend. To instruct Astra Trident about the Azure NetApp Files subscription and where it needs to create volumes, a backend is created. This step requires details about the account that was created in the previous step.

Install Astra Trident using the operator

This section walks you through the installation of Astra Trident using the operator.

Download Astra Trident from its GitHub repository. Choose from the desired version and download the installer bundle.

wget https://github.com/NetApp/trident/releases/download/v21.07.1/trident-installer-21.07.1.tar.gz
tar xzvf trident-installer-21.07.1.tar.gz

Run the [kubectl create][kubectl-create] command to create the trident namespace:
```
kubectl create ns trident
```
The output of the command resembles the following example:
```
namespace/trident created
```

Run the kubectl apply command to deploy the Trident operator using the bundle file:

kubectl apply -f trident-installer/deploy/bundle.yaml -n trident

The output of the command resembles the following example:

serviceaccount/trident-operator created
clusterrole.rbac.authorization.k8s.io/trident-operator created
clusterrolebinding.rbac.authorization.k8s.io/trident-operator created
deployment.apps/trident-operator created
podsecuritypolicy.policy/tridentoperatorpods created

Run the following command to create a TridentOrchestrator to install Astra Trident.
```
kubectl apply -f trident-installer/deploy/crds/tridentorchestrator_cr.yaml
```
The output of the command resembles the following example:
```
tridentorchestrator.trident.netapp.io/trident created 
```
The operator installs by using the parameters provided in the TridentOrchestrator spec. You can learn about the configuration parameters and example backends from the Trident install guide and backend guide.

To confirm Astra Trident was installed successfully, run the following kubectl describe command:

kubectl describe torc trident

The output of the command resembles the following example:

Name:         trident
Namespace:
Labels:       <none>
Annotations:  <none>
API Version:  trident.netapp.io/v1
Kind:         TridentOrchestrator
...
Spec:
  Debug:      true
  Namespace:  trident
Status:
  Current Installation Params:
    IPv6:                       false
    Autosupport Hostname:
    Autosupport Image:          netapp/trident-autosupport:21.01
    Autosupport Proxy:
    Autosupport Serial Number:
    Debug:                      true
    Enable Node Prep:           false
    Image Pull Secrets:
    Image Registry:
    k8sTimeout:           30
    Kubelet Dir:          /var/lib/kubelet
    Log Format:           text
    Silence Autosupport:  false
    Trident Image:        netapp/trident:21.07.1
  Message:                Trident installed
  Namespace:              trident
  Status:                 Installed
  Version:                v21.07.1
Events:
  Type    Reason      Age   From                        Message
  ----    ------      ----  ----                        -------
  Normal  Installing  74s   trident-operator.netapp.io  Installing Trident
  Normal  Installed   67s   trident-operator.netapp.io  Trident installed

Create a backend

Before creating a backend, you need to update backend-anf.yaml to include details about the Azure NetApp Files subscription, such as:
- subscriptionID for the Azure subscription where Azure NetApp Files will be enabled.
- tenantID, clientID, and clientSecret from an App Registration in Azure Active Directory (AD) with sufficient permissions for the Azure NetApp Files service. The App Registration include the Owner or Contributor role that's predefined by Azure.
- An Azure location that contains at least one delegated subnet.
In addition, you can choose to provide a different service level. Azure NetApp Files provides three service levels: Standard, Premium, and Ultra.

After Astra Trident is installed, create a backend that points to your Azure NetApp Files subscription by running the following command.

kubectl apply -f trident-installer/sample-input/backends-samples/azure-netapp-files/backend-anf.yaml -n trident

The output of the command resembles the following example:

secret/backend-tbc-anf-secret created
tridentbackendconfig.trident.netapp.io/backend-tbc-anf created

Create a StorageClass

A storage class is used to define how a unit of storage is dynamically created with a persistent volume. To consume Azure NetApp Files volumes, a storage class must be created.

Create a file named anf-storageclass.yaml and copy in the following manifest:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: azure-netapp-files
provisioner: csi.trident.netapp.io
parameters:
  backendType: "azure-netapp-files"
  fsType: "nfs"

Create the storage class using the kubectl apply command:
```
kubectl apply -f anf-storageclass.yaml
```
The output of the command resembles the following example::
```
storageclass/azure-netapp-files created
```

Run the kubectl get command to view the status of the storage class:

kubectl get sc
NAME                 PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
azure-netapp-files   csi.trident.netapp.io   Delete          Immediate           false                  3s

Create a persistent volume claim

A persistent volume claim (PVC) is a request for storage by a user. Upon the creation of a persistent volume claim, Astra Trident automatically creates an Azure NetApp Files volume and makes it available for Kubernetes workloads to consume.

Create a file named anf-pvc.yaml and copy the following manifest. In this example, a 1-TiB volume is created that with ReadWriteMany access.

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: anf-pvc
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Ti
  storageClassName: azure-netapp-files

Create the persistent volume claim with the kubectl apply command:
```
kubectl apply -f anf-pvc.yaml
```
The output of the command resembles the following example:
```
persistentvolumeclaim/anf-pvc created
```

To view information about the persistent volume claim, run the kubectl get command:

kubectl get pvc

The output of the command resembles the following example:

kubectl get pvc -n trident
NAME      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS         AGE
anf-pvc   Bound    pvc-bffa315d-3f44-4770-86eb-c922f567a075   1Ti        RWO            azure-netapp-files   62s

Use the persistent volume

After the PVC is created, a pod can be spun up to access the Azure NetApp Files volume. The following manifest can be used to define an NGINX pod that mounts the Azure NetApp Files volume created in the previous step. In this example, the volume is mounted at /mnt/data.

Create a file named anf-nginx-pod.yaml and copy the following manifest:

kind: Pod
apiVersion: v1
metadata:
  name: nginx-pod
spec:
  containers:
  - name: nginx
    image: mcr.microsoft.com/oss/nginx/nginx:latest1.15.5-alpine
    resources:
      requests:
        cpu: 100m
        memory: 128Mi
      limits:
        cpu: 250m
        memory: 256Mi
    volumeMounts:
    - mountPath: "/mnt/data"
      name: volume
  volumes:
    - name: volume
      persistentVolumeClaim:
        claimName: anf-pvc

Create the pod using the kubectl apply command:

kubectl apply -f anf-nginx-pod.yaml

The output of the command resembles the following example:

pod/nginx-pod created

Kubernetes has created a pod with the volume mounted and accessible within the nginx container at /mnt/data. You can confirm by checking the event logs for the pod using kubectl describe command:

kubectl describe pod nginx-pod

The output of the command resembles the following example:

[...]
Volumes:
  volume:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  anf-pvc
    ReadOnly:   false
  default-token-k7952:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-k7952
    Optional:    false
[...]
Events:
  Type    Reason                  Age   From                     Message
  ----    ------                  ----  ----                     -------
  Normal  Scheduled               15s   default-scheduler        Successfully assigned trident/nginx-pod to brameshb-non-root-test
  Normal  SuccessfulAttachVolume  15s   attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-bffa315d-3f44-4770-86eb-c922f567a075"
  Normal  Pulled                  12s   kubelet                  Container image "mcr.microsoft.com/oss/nginx/nginx:1.15.5-alpine" already present on machine
  Normal  Created                 11s   kubelet                  Created container nginx
  Normal  Started                 10s   kubelet                  Started container nginx

Using Azure tags

For more details on using Azure tags, see Use Azure tags in Azure Kubernetes Service (AKS).

Next steps

Astra Trident supports many features with Azure NetApp Files. For more information, see:

Provision Azure NetApp Files volumes on Azure Kubernetes Service

Before you begin

Configure Azure NetApp Files

Provision Azure NetApp Files volumes statically

Create the persistent volume

Create a persistent volume claim

Mount with a pod

Provision Azure NetApp Files volumes dynamically

Install and configure Astra Trident

Install Astra Trident using the operator

Create a backend

Create a StorageClass

Create a persistent volume claim

Use the persistent volume

Using Azure tags

Next steps

Feedback

Additional resources

Additional resources