Prepare Linux for Edge Volumes using a multi-node cluster

This article describes how to prepare Linux using a multi-node cluster, and assumes you fulfilled the prerequisites.

Prepare Linux with AKS enabled by Azure Arc

Install and configure Open Service Mesh (OSM) using the following commands:

az k8s-extension create --resource-group "YOUR_RESOURCE_GROUP_NAME" --cluster-name "YOUR_CLUSTER_NAME" --cluster-type connectedClusters --extension-type Microsoft.openservicemesh --scope cluster --name osm \
--config "osm.osm.featureFlags.enableWASMStats=false" \
--config "osm.osm.enablePermissiveTrafficPolicy=false" \
--config "osm.osm.configResyncInterval=10s" \
--config "osm.osm.osmController.resource.requests.cpu=100m" \
--config "osm.osm.osmBootstrap.resource.requests.cpu=100m" \
--config "osm.osm.injector.resource.requests.cpu=100m"

Prepare Linux with AKS Edge Essentials

This section describes how to prepare Linux with AKS Edge Essentials if you run a multi-node cluster.

  1. On each node in your cluster, set the number of HugePages to 512 using the following command:

    Invoke-AksEdgeNodeCommand -NodeType "Linux" -Command 'echo 512 | sudo tee /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages'
    Invoke-AksEdgeNodeCommand -NodeType "Linux" -Command 'echo "vm.nr_hugepages=512" | sudo tee /etc/sysctl.d/99-hugepages.conf'
    
  2. On each node in your cluster, install the required NVME over TCP module for your kernel using:

    Invoke-AksEdgeNodeCommand -NodeType "Linux" -Command 'sudo apt install linux-modules-extra-`uname -r`'
    

    Note

    The minimum supported version is 5.1. At this time, there are known issues with 6.4 and 6.2.

  3. On each node in your cluster, increase the maximum number of files using the following command:

    Invoke-AksEdgeNodeCommand -NodeType "Linux" -Command 'echo -e "LimitNOFILE=1048576" | sudo tee -a /etc/systemd/system/containerd.service.d/override.conf'
    
  4. Install and configure Open Service Mesh (OSM) using the following commands:

    az k8s-extension create --resource-group "YOUR_RESOURCE_GROUP_NAME" --cluster-name "YOUR_CLUSTER_NAME" --cluster-type connectedClusters --extension-type Microsoft.openservicemesh --scope cluster --name osm \
    --config "osm.osm.featureFlags.enableWASMStats=false" \
    --config "osm.osm.enablePermissiveTrafficPolicy=false" \
    --config "osm.osm.configResyncInterval=10s" \
    --config "osm.osm.osmController.resource.requests.cpu=100m" \
    --config "osm.osm.osmBootstrap.resource.requests.cpu=100m" \
    --config "osm.osm.injector.resource.requests.cpu=100m"
    
    kubectl patch meshconfig osm-mesh-config -n "arc-osm-system" -p '{"spec":{"featureFlags":{"enableWASMStats": false }, "traffic":{"outboundPortExclusionList":[443,2379,2380], "inboundPortExclusionList":[443,2379,2380]}}}' --type=merge
    

Prepare Linux with Ubuntu

This section describes how to prepare Linux with Ubuntu if you run a multi-node cluster.

First, install and configure Open Service Mesh (OSM) using the following command:

az k8s-extension create --resource-group "YOUR_RESOURCE_GROUP_NAME" --cluster-name "YOUR_CLUSTER_NAME" --cluster-type connectedClusters --extension-type Microsoft.openservicemesh --scope cluster --name osm \
--config "osm.osm.featureFlags.enableWASMStats=false" \
--config "osm.osm.enablePermissiveTrafficPolicy=false" \
--config "osm.osm.configResyncInterval=10s" \
--config "osm.osm.osmController.resource.requests.cpu=100m" \
--config "osm.osm.osmBootstrap.resource.requests.cpu=100m" \
--config "osm.osm.injector.resource.requests.cpu=100m"

Then, perform the following steps in your Kubernetes cluster:

Important

You must complete the following steps for each node in your Kubernetes cluster.

  1. Run the following command to determine if you set fs.inotify.max_user_instances to 1024:

    sysctl fs.inotify.max_user_instances
    

    After you run this command, if it outputs less than 1024, run the following command to increase the maximum number of files and reload the sysctl settings:

    echo 'fs.inotify.max_user_instances = 1024' | sudo tee -a /etc/sysctl.conf
    sudo sysctl -p
    
  2. Install the required NVME over TCP module for your kernel using:

    sudo apt install linux-modules-extra-`uname -r`
    

    Note

    The minimum supported version is 5.1. At this time, there are known issues with 6.4 and 6.2.

  3. Set the number of HugePages to 512 using the following command:

    HUGEPAGES_NR=512
    echo $HUGEPAGES_NR | sudo tee /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
    echo "vm.nr_hugepages=$HUGEPAGES_NR" | sudo tee /etc/sysctl.d/99-hugepages.conf
    
  4. Restart K3s using the following command:

    sudo systemctl restart k3s || sudo systemctl restart k3s-agent
    

Prepare Linux with other platforms

The available platform options are production-like environments that Microsoft validated. These platforms aren't necessarily the only environments on which Azure Container Storage enabled by Azure Arc can run. Azure Container Storage enabled by Azure Arc can run on any Arc-enabled Kubernetes cluster that meets the Azure Arc-enabled Kubernetes system requirements. If you're running on an environment not listed, here are a few suggestions to increase the likelihood of a successful installation:

  1. Run the following commands to increase the user watch and instance limits:

    echo fs.inotify.max_user_instances=8192 | sudo tee -a /etc/sysctl.conf
    echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf
    sudo sysctl -p
    
  2. Run the following commands to increase the file descriptor limit for better performance:

    echo fs.file-max = 100000 | sudo tee -a /etc/sysctl.conf
    sudo sysctl -p
    
  3. Run the following command to install the required NVME over TCP module for your kernel:

    sudo apt install linux-modules-extra-`uname -r`
    
  4. Run the following command to set the number of HugePages to 512:

    HUGEPAGES_NR=512
    echo $HUGEPAGES_NR | sudo tee /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
    echo "vm.nr_hugepages=$HUGEPAGES_NR" | sudo tee /etc/sysctl.d/99-hugepages.conf
    

Next steps

Install Extension