Provision Azure NetApp Files volumes on Azure Kubernetes Service
A persistent volume represents a piece of storage that has been provisioned for use with Kubernetes pods. A persistent volume can be used by one or many pods, and can be dynamically or statically provisioned. This article shows you how to create Azure NetApp Files volumes to be used by pods on an Azure Kubernetes Service (AKS) cluster.
Azure NetApp Files is an enterprise-class, high-performance, metered file storage service running on Azure. Kubernetes users have two options for using Azure NetApp Files volumes for Kubernetes workloads:
- Create Azure NetApp Files volumes statically. In this scenario, the creation of volumes is external to AKS. Volumes are created using the Azure CLI or from the Azure portal, and are then exposed to Kubernetes by the creation of a
PersistentVolume
. Statically created Azure NetApp Files volumes have many limitations (for example, inability to be expanded, needing to be over-provisioned, and so on). Statically created volumes are not recommended for most use cases. - Create Azure NetApp Files volumes on-demand, orchestrating through Kubernetes. This method is the preferred way to create multiple volumes directly through Kubernetes, and is achieved using Astra Trident. Astra Trident is a CSI-compliant dynamic storage orchestrator that helps provision volumes natively through Kubernetes.
Using a CSI driver to directly consume Azure NetApp Files volumes from AKS workloads is the recommended configuration for most use cases. This requirement is accomplished using Astra Trident, an open-source dynamic storage orchestrator for Kubernetes. Astra Trident is an enterprise-grade storage orchestrator purpose-built for Kubernetes, and fully supported by NetApp. It simplifies access to storage from Kubernetes clusters by automating storage provisioning.
You can take advantage of Astra Trident's Container Storage Interface (CSI) driver for Azure NetApp Files to abstract underlying details and create, expand, and snapshot volumes on-demand. Also, using Astra Trident enables you to use Astra Control Service built on top of Astra Trident. Using the Astra Control Service, you can backup, recover, move, and manage the application-data lifecycle of your AKS workloads across clusters within and across Azure regions to meet your business and service continuity needs.
Before you begin
The following considerations apply when you use Azure NetApp Files:
- Your AKS cluster must be in a region that supports Azure NetApp Files.
- The Azure CLI version 2.0.59 or higher installed and configured. Run
az --version
to find the version. If you need to install or upgrade, see Install Azure CLI. - After the initial deployment of an AKS cluster, you can choose to provision Azure NetApp Files volumes statically or dynamically.
- To use dynamic provisioning with Azure NetApp Files, install and configure Astra Trident version 19.07 or higher.
Configure Azure NetApp Files
Register the Microsoft.NetApp resource provider by running the following command:
az provider register --namespace Microsoft.NetApp --wait
Note
This operation can take several minutes to complete.
When you create an Azure NetApp account for use with AKS, you can create the account in an existing resource group or create a new one in the same region as the AKS cluster. The following command creates an account named myaccount1 in the myResourceGroup resource group and eastus region:
az netappfiles account create \ --resource-group myResourceGroup \ --location eastus \ --account-name myaccount1
Create a new capacity pool by using az netappfiles pool create. The following example creates a new capacity pool named mypool1 with 4 TB in size and Premium service level:
az netappfiles pool create \ --resource-group myResourceGroup \ --location eastus \ --account-name myaccount1 \ --pool-name mypool1 \ --size 4 \ --service-level Premium
Create a subnet to delegate to Azure NetApp Files using az network vnet subnet create. Specify the resource group hosting the existing virtual network for your AKS cluster.
Note
This subnet must be in the same virtual network as your AKS cluster.
RESOURCE_GROUP=myResourceGroup VNET_NAME=$(az network vnet list --resource-group $RESOURCE_GROUP --query [].name -o tsv) VNET_ID=$(az network vnet show --resource-group $RESOURCE_GROUP --name $VNET_NAME --query "id" -o tsv) SUBNET_NAME=MyNetAppSubnet az network vnet subnet create \ --resource-group $RESOURCE_GROUP \ --vnet-name $VNET_NAME \ --name $SUBNET_NAME \ --delegations "Microsoft.NetApp/volumes" \ --address-prefixes 10.0.0.0/28
Volumes can either be provisioned statically or dynamically. Both options are covered further in the next sections.
Provision Azure NetApp Files volumes statically
Create a volume using the az netappfiles volume create command. Update
RESOURCE_GROUP
,LOCATION
,ANF_ACCOUNT_NAME
(Azure NetApp account name),POOL_NAME
, andSERVICE_LEVEL
with the correct values.RESOURCE_GROUP=myResourceGroup LOCATION=eastus ANF_ACCOUNT_NAME=myaccount1 POOL_NAME=mypool1 SERVICE_LEVEL=Premium VNET_NAME=$(az network vnet list --resource-group $RESOURCE_GROUP --query [].name -o tsv) VNET_ID=$(az network vnet show --resource-group $RESOURCE_GROUP --name $VNET_NAME --query "id" -o tsv) SUBNET_NAME=MyNetAppSubnet SUBNET_ID=$(az network vnet subnet show --resource-group $RESOURCE_GROUP --vnet-name $VNET_NAME --name $SUBNET_NAME --query "id" -o tsv) VOLUME_SIZE_GiB=100 # 100 GiB UNIQUE_FILE_PATH="myfilepath2" # Note that file path needs to be unique within all ANF Accounts az netappfiles volume create \ --resource-group $RESOURCE_GROUP \ --location $LOCATION \ --account-name $ANF_ACCOUNT_NAME \ --pool-name $POOL_NAME \ --name "myvol1" \ --service-level $SERVICE_LEVEL \ --vnet $VNET_ID \ --subnet $SUBNET_ID \ --usage-threshold $VOLUME_SIZE_GiB \ --file-path $UNIQUE_FILE_PATH \ --protocol-types "NFSv3"
Create the persistent volume
List the details of your volume using az netappfiles volume show
az netappfiles volume show \ --resource-group $RESOURCE_GROUP \ --account-name $ANF_ACCOUNT_NAME \ --pool-name $POOL_NAME \ --volume-name "myvol1" -o JSON
The following output resembles the output of the previous command:
{ ... "creationToken": "myfilepath2", ... "mountTargets": [ { ... "ipAddress": "10.0.0.4", ... } ], ... }
Create a
pv-nfs.yaml
defining a persistent volume by copying the following manifest. Replacepath
with the creationToken andserver
with ipAddress from the previous step.--- apiVersion: v1 kind: PersistentVolume metadata: name: pv-nfs spec: capacity: storage: 100Gi accessModes: - ReadWriteMany mountOptions: - vers=3 nfs: server: 10.0.0.4 path: /myfilepath2
Create the persistent volume using the kubectl apply command:
kubectl apply -f pv-nfs.yaml
Verify the Status of the PersistentVolume is Available using the kubectl describe command:
kubectl describe pv pv-nfs
Create a persistent volume claim
Create a
pvc-nfs.yaml
defining a PersistentVolume by copying the following manifest:apiVersion: v1 kind: PersistentVolumeClaim metadata: name: pvc-nfs spec: accessModes: - ReadWriteMany storageClassName: "" resources: requests: storage: 1Gi
Create the persistent volume claim using the kubectl apply command:
kubectl apply -f pvc-nfs.yaml
Verify the Status of the persistent volume claim is Bound using the kubectl describe command:
kubectl describe pvc pvc-nfs
Mount with a pod
Create a
nginx-nfs.yaml
defining a pod that uses the persistent volume claim by using the following manifest:kind: Pod apiVersion: v1 metadata: name: nginx-nfs spec: containers: - image: mcr.microsoft.com/oss/nginx/nginx:1.15.5-alpine name: nginx-nfs command: - "/bin/sh" - "-c" - while true; do echo $(date) >> /mnt/azure/outfile; sleep 1; done volumeMounts: - name: disk01 mountPath: /mnt/azure volumes: - name: disk01 persistentVolumeClaim: claimName: pvc-nfs
Create the pod using the kubectl apply command:
kubectl apply -f nginx-nfs.yaml
Verify the pod is Running using the kubectl describe command:
kubectl describe pod nginx-nfs
Verify your volume has been mounted on the pod by using kubectl exec to connect to the pod, and then use
df -h
to check if the volume is mounted.kubectl exec -it nginx-nfs -- sh
/ # df -h Filesystem Size Used Avail Use% Mounted on ... 10.0.0.4:/myfilepath2 100T 384K 100T 1% /mnt/azure ...
Provision Azure NetApp Files volumes dynamically
Install and configure Astra Trident
To dynamically provision volumes, you need to install Astra Trident. Astra Trident is NetApp's dynamic storage provisioner that is purpose-built for Kubernetes. Simplify the consumption of storage for Kubernetes applications using Astra Trident's industry-standard Container Storage Interface (CSI) driver. Astra Trident deploys on Kubernetes clusters as pods and provides dynamic storage orchestration services for your Kubernetes workloads.
Before proceeding to the next section, you need to:
Install Astra Trident. Trident can be installed using the Trident operator (manually or using Helm) or
tridentctl
. The instructions provided later in this article explain how Astra Trident can be installed using the operator. To learn more about these installation methods and how they work, see the Install Guide.Create a backend. To instruct Astra Trident about the Azure NetApp Files subscription and where it needs to create volumes, a backend is created. This step requires details about the account that was created in the previous step.
Install Astra Trident using the operator
This section walks you through the installation of Astra Trident using the operator.
Download Astra Trident from its GitHub repository. Choose from the desired version and download the installer bundle.
wget https://github.com/NetApp/trident/releases/download/v21.07.1/trident-installer-21.07.1.tar.gz tar xzvf trident-installer-21.07.1.tar.gz
Run the [kubectl create][kubectl-create] command to create the trident namespace:
kubectl create ns trident
The output of the command resembles the following example:
namespace/trident created
Run the kubectl apply command to deploy the Trident operator using the bundle file:
kubectl apply -f trident-installer/deploy/bundle.yaml -n trident
The output of the command resembles the following example:
serviceaccount/trident-operator created clusterrole.rbac.authorization.k8s.io/trident-operator created clusterrolebinding.rbac.authorization.k8s.io/trident-operator created deployment.apps/trident-operator created podsecuritypolicy.policy/tridentoperatorpods created
Run the following command to create a
TridentOrchestrator
to install Astra Trident.kubectl apply -f trident-installer/deploy/crds/tridentorchestrator_cr.yaml
The output of the command resembles the following example:
tridentorchestrator.trident.netapp.io/trident created
The operator installs by using the parameters provided in the
TridentOrchestrator
spec. You can learn about the configuration parameters and example backends from the Trident install guide and backend guide.To confirm Astra Trident was installed successfully, run the following kubectl describe command:
kubectl describe torc trident
The output of the command resembles the following example:
Name: trident Namespace: Labels: <none> Annotations: <none> API Version: trident.netapp.io/v1 Kind: TridentOrchestrator ... Spec: Debug: true Namespace: trident Status: Current Installation Params: IPv6: false Autosupport Hostname: Autosupport Image: netapp/trident-autosupport:21.01 Autosupport Proxy: Autosupport Serial Number: Debug: true Enable Node Prep: false Image Pull Secrets: Image Registry: k8sTimeout: 30 Kubelet Dir: /var/lib/kubelet Log Format: text Silence Autosupport: false Trident Image: netapp/trident:21.07.1 Message: Trident installed Namespace: trident Status: Installed Version: v21.07.1 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Installing 74s trident-operator.netapp.io Installing Trident Normal Installed 67s trident-operator.netapp.io Trident installed
Create a backend
Before creating a backend, you need to update
backend-anf.yaml
to include details about the Azure NetApp Files subscription, such as:subscriptionID
for the Azure subscription where Azure NetApp Files will be enabled.tenantID
,clientID
, andclientSecret
from an App Registration in Azure Active Directory (AD) with sufficient permissions for the Azure NetApp Files service. The App Registration include theOwner
orContributor
role that's predefined by Azure.- An Azure location that contains at least one delegated subnet.
In addition, you can choose to provide a different service level. Azure NetApp Files provides three service levels: Standard, Premium, and Ultra.
After Astra Trident is installed, create a backend that points to your Azure NetApp Files subscription by running the following command.
kubectl apply -f trident-installer/sample-input/backends-samples/azure-netapp-files/backend-anf.yaml -n trident
The output of the command resembles the following example:
secret/backend-tbc-anf-secret created tridentbackendconfig.trident.netapp.io/backend-tbc-anf created
Create a StorageClass
A storage class is used to define how a unit of storage is dynamically created with a persistent volume. To consume Azure NetApp Files volumes, a storage class must be created.
Create a file named
anf-storageclass.yaml
and copy in the following manifest:apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: azure-netapp-files provisioner: csi.trident.netapp.io parameters: backendType: "azure-netapp-files" fsType: "nfs"
Create the storage class using the kubectl apply command:
kubectl apply -f anf-storageclass.yaml
The output of the command resembles the following example::
storageclass/azure-netapp-files created
Run the kubectl get command to view the status of the storage class:
kubectl get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE azure-netapp-files csi.trident.netapp.io Delete Immediate false 3s
Create a persistent volume claim
A persistent volume claim (PVC) is a request for storage by a user. Upon the creation of a persistent volume claim, Astra Trident automatically creates an Azure NetApp Files volume and makes it available for Kubernetes workloads to consume.
Create a file named
anf-pvc.yaml
and copy the following manifest. In this example, a 1-TiB volume is created that with ReadWriteMany access.kind: PersistentVolumeClaim apiVersion: v1 metadata: name: anf-pvc spec: accessModes: - ReadWriteMany resources: requests: storage: 1Ti storageClassName: azure-netapp-files
Create the persistent volume claim with the kubectl apply command:
kubectl apply -f anf-pvc.yaml
The output of the command resembles the following example:
persistentvolumeclaim/anf-pvc created
To view information about the persistent volume claim, run the kubectl get command:
kubectl get pvc
The output of the command resembles the following example:
kubectl get pvc -n trident NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE anf-pvc Bound pvc-bffa315d-3f44-4770-86eb-c922f567a075 1Ti RWO azure-netapp-files 62s
Use the persistent volume
After the PVC is created, a pod can be spun up to access the Azure NetApp Files volume. The following manifest can be used to define an NGINX pod that mounts the Azure NetApp Files volume created in the previous step. In this example, the volume is mounted at /mnt/data
.
Create a file named
anf-nginx-pod.yaml
and copy the following manifest:kind: Pod apiVersion: v1 metadata: name: nginx-pod spec: containers: - name: nginx image: mcr.microsoft.com/oss/nginx/nginx:latest1.15.5-alpine resources: requests: cpu: 100m memory: 128Mi limits: cpu: 250m memory: 256Mi volumeMounts: - mountPath: "/mnt/data" name: volume volumes: - name: volume persistentVolumeClaim: claimName: anf-pvc
Create the pod using the kubectl apply command:
kubectl apply -f anf-nginx-pod.yaml
The output of the command resembles the following example:
pod/nginx-pod created
Kubernetes has created a pod with the volume mounted and accessible within the
nginx
container at/mnt/data
. You can confirm by checking the event logs for the pod using kubectl describe command:kubectl describe pod nginx-pod
The output of the command resembles the following example:
[...] Volumes: volume: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: anf-pvc ReadOnly: false default-token-k7952: Type: Secret (a volume populated by a Secret) SecretName: default-token-k7952 Optional: false [...] Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 15s default-scheduler Successfully assigned trident/nginx-pod to brameshb-non-root-test Normal SuccessfulAttachVolume 15s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-bffa315d-3f44-4770-86eb-c922f567a075" Normal Pulled 12s kubelet Container image "mcr.microsoft.com/oss/nginx/nginx:1.15.5-alpine" already present on machine Normal Created 11s kubelet Created container nginx Normal Started 10s kubelet Started container nginx
Using Azure tags
For more details on using Azure tags, see Use Azure tags in Azure Kubernetes Service (AKS).
Next steps
Astra Trident supports many features with Azure NetApp Files. For more information, see:
Feedback
Submit and view feedback for