Edit

Share via

Prepare Linux using a multi-node cluster

  • Article

This article describes how to prepare Linux using a multi-node cluster, and assumes you fulfilled the prerequisites.

Prepare Linux with AKS enabled by Azure Arc

Install and configure Open Service Mesh (OSM) using the following commands:

az k8s-extension create --resource-group "YOUR_RESOURCE_GROUP_NAME" --cluster-name "YOUR_CLUSTER_NAME" --cluster-type connectedClusters --extension-type Microsoft.openservicemesh --scope cluster --name osm \
--config osm.osm.featureFlags.enableWASMStats=false" \
--config osm.osm.enablePermissiveTrafficPolicy=false" \
--config osm.osm.configResyncInterval=10s" \
--config osm.osm.osmController.resource.requests.cpu=100m" \
--config osm.osm.osmBootstrap.resource.requests.cpu=100m" \
--config osm.osm.injector.resource.requests.cpu=100m

kubectl patch meshconfig osm-mesh-config -n "arc-osm-system" -p '{"spec":{"featureFlags":{"enableWASMStats": false }, "traffic":{"outboundPortExclusionList":[443,2379,2380], "inboundPortExclusionList":[443,2379,2380]}}}' --type=merge

Prepare Linux with AKS Edge Essentials

This section describes how to prepare Linux with AKS Edge Essentials if you run a multi-node cluster.

  1. On each node in your cluster, set the number of HugePages to 512 using the following command:

    Invoke-AksEdgeNodeCommand -NodeType "Linux" -Command 'echo 512 | sudo tee /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages'
Invoke-AksEdgeNodeCommand -NodeType "Linux" -Command 'echo "vm.nr_hugepages=512" | sudo tee /etc/sysctl.d/99-hugepages.conf'

  2. On each node in your cluster, install the required NVME over TCP module for your kernel using:

    Invoke-AksEdgeNodeCommand -NodeType "Linux" -Command 'sudo apt install linux-modules-extra-`uname -r`'

    Note

    The minimum supported version is 5.1. At this time, there are known issues with 6.4 and 6.2.

  3. On each node in your cluster, increase the maximum number of files using the following command:

    Invoke-AksEdgeNodeCommand -NodeType "Linux" -Command 'echo -e "LimitNOFILE=1048576" | sudo tee -a /etc/systemd/system/containerd.service.d/override.conf'

  4. Install and configure Open Service Mesh (OSM) using the following commands:

    az k8s-extension create --resource-group "YOUR_RESOURCE_GROUP_NAME" --cluster-name "YOUR_CLUSTER_NAME" --cluster-type connectedClusters --extension-type Microsoft.openservicemesh --scope cluster --name osm \
--config "osm.osm.featureFlags.enableWASMStats=false" \
--config "osm.osm.enablePermissiveTrafficPolicy=false" \
--config "osm.osm.configResyncInterval=10s" \
--config "osm.osm.osmController.resource.requests.cpu=100m" \
--config "osm.osm.osmBootstrap.resource.requests.cpu=100m" \
--config "osm.osm.injector.resource.requests.cpu=100m

kubectl patch meshconfig osm-mesh-config -n "arc-osm-system" -p '{"spec":{"featureFlags":{"enableWASMStats": false }, "traffic":{"outboundPortExclusionList":[443,2379,2380], "inboundPortExclusionList":[443,2379,2380]}}}' --type=merge

  1. Create a file named config.json with the following contents:

    {
    "acstor.capacityProvisioner.tempDiskMountPoint": /var
}

    Note

    The location/path of this file is referenced later, when you install the Cache Volumes Arc extension.

Prepare Linux with Ubuntu

This section describes how to prepare Linux with Ubuntu if you run a multi-node cluster.

  1. Install and configure Open Service Mesh (OSM) using the following command:

    az k8s-extension create --resource-group "YOUR_RESOURCE_GROUP_NAME" --cluster-name "YOUR_CLUSTER_NAME" --cluster-type connectedClusters --extension-type Microsoft.openservicemesh --scope cluster --name osm
kubectl patch meshconfig osm-mesh-config -n "arc-osm-system" -p '{"spec":{"featureFlags":{"enableWASMStats": false }, "traffic":{"outboundPortExclusionList":[443,2379,2380], "inboundPortExclusionList":[443,2379,2380]}}}' --type=merge

Then, perform the following steps in your Kubernetes cluster:

Important

You must complete the following steps for each node in your Kubernetes cluster.

  1. Run the following command to determine if you set fs.inotify.max_user_instances to 1024:

    sysctl fs.inotify.max_user_instances

    After you run this command, if it outputs less than 1024, run the following command to increase the maximum number of files and reload the sysctl settings:

    echo 'fs.inotify.max_user_instances = 1024' | sudo tee -a /etc/sysctl.conf
sudo sysctl -p

  2. Install the required NVME over TCP module for your kernel using:

    sudo apt install linux-modules-extra-`uname -r`

    Note

    The minimum supported version is 5.1. At this time, there are known issues with 6.4 and 6.2.

  3. Set the number of HugePages to 512 using the following command:

    HUGEPAGES_NR=512
echo $HUGEPAGES_NR | sudo tee /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
echo "vm.nr_hugepages=$HUGEPAGES_NR" | sudo tee /etc/sysctl.d/99-hugepages.conf

  4. Restart K3s using the following command:

    sudo systemctl restart k3s || sudo systemctl restart k3s-agent

Next steps

Install Azure Container Storage enabled by Azure Arc