Prepare your Azure Arc-enabled Kubernetes cluster

Important

Azure IoT Operations Preview – enabled by Azure Arc is currently in preview. You shouldn't use this preview software in production environments.

You'll need to deploy a new Azure IoT Operations installation when a generally available release is made available. You won't be able to upgrade a preview installation.

See the Supplemental Terms of Use for Microsoft Azure Previews for legal terms that apply to Azure features that are in beta, preview, or otherwise not yet released into general availability.

An Azure Arc-enabled Kubernetes cluster is a prerequisite for deploying Azure IoT Operations Preview. This article describes how to prepare a cluster before you Deploy Azure IoT Operations Preview to an Arc-enabled Kubernetes cluster. This article includes guidance for both Ubuntu and Windows.

Tip

The steps in this article prepare your cluster for a secure settings deployment, which is a longer but production-ready process. If you want to deploy Azure IoT Operations quickly and run a sample workload with only test settings, see the Quickstart: Run Azure IoT Operations Preview in Github Codespaces with K3s instead.

For more information about test settings and secure settings, see Deployment details > Choose your features.

Prerequisites

Azure IoT Operations should work on any Arc-enabled Kubernetes cluster that meets the Azure Arc-enabled Kubernetes system requirements. Currently Azure IoT Operations doesn't support Arm64 architectures.

Microsoft supports Azure Kubernetes Service (AKS) Edge Essentials for deployments on Windows and K3s for deployments on Ubuntu. For a list of specific hardware and software combinations that are tested and validated, see Validated environments.

If you want to deploy Azure IoT Operations to a multi-node solution, use K3s on Ubuntu.

To prepare your Azure Arc-enabled Kubernetes cluster, you need:

AKS Edge Essentials

• An Azure subscription. If you don't have an Azure subscription, create one for free before you begin.

• Azure CLI version 2.64.0 or newer installed on your development machine. Use az --version to check your version and az upgrade to update if necessary. For more information, see How to install the Azure CLI.

• The latest version of the Azure IoT Operations extension for Azure CLI. Use the following command to add the extension or update it to the latest version:

  az extension add --upgrade --name azure-iot-ops
  

• Hardware that meets the system requirements:

  • Ensure that your machine has a minimum of 16-GB available RAM, 8 available vCPUs, and 52-GB free disk space reserved for Azure IoT Operations.
  • Azure Arc-enabled Kubernetes system requirements.
  • AKS Edge Essentials requirements and support matrix.
  • AKS Edge Essentials networking guidance.

• If you're going to deploy Azure IoT Operations to a multi-node cluster with fault tolerance enabled, review the hardware and storage requirements in Prepare Linux for Edge Volumes.

Ubuntu

• An Azure subscription. If you don't have an Azure subscription, create one for free before you begin.

• Azure CLI version 2.64.0 or newer installed on your development machine. Use az --version to check your version and az upgrade to update if necessary. For more information, see How to install the Azure CLI.

• The latest version of the Azure IoT Operations extension for Azure CLI. Use the following command to add the extension or update it to the latest version:

  az extension add --upgrade --name azure-iot-ops
  

• Hardware that meets the system requirements:

  • Ensure that your machine has a minimum of 16-GB available RAM and 8 available vCPUs reserved for Azure IoT Operations.
  • Azure Arc-enabled Kubernetes system requirements.
  • K3s requirements.

Create a cluster

This section provides steps to create clusters in validated environments on Linux and Windows.

AKS Edge Essentials

Azure Kubernetes Service Edge Essentials is an on-premises Kubernetes implementation of Azure Kubernetes Service (AKS) that automates running containerized applications at scale. AKS Edge Essentials includes a Microsoft-supported Kubernetes platform that includes a lightweight Kubernetes distribution with a small footprint and simple installation experience that supports PC-class or "light" edge hardware.

The AksEdgeQuickStartForAio.ps1 script automates the process of creating and connecting a cluster, and is the recommended path for deploying Azure IoT Operations on AKS Edge Essentials.

1. Open an elevated PowerShell window and change the directory to a working folder.

2. Get the objectId of the Microsoft Entra ID application that the Azure Arc service uses in your tenant.

   az ad sp show --id bc313c14-388c-4e7d-a58e-70017303ee3b --query id -o tsv
   

3. Run the following commands, replacing the placeholder values with your information:

   Placeholder	Value
   SUBSCRIPTION_ID	The ID of your Azure subscription. If you don't know your subscription ID, see Find your Azure subscription.
   TENANT_ID	The ID of your Microsoft Entra tenant. If you don't know your tenant ID, see Find your Microsoft Entra tenant.
   RESOURCE_GROUP_NAME	The name of an existing resource group or a name for a new resource group to be created.
   LOCATION	An Azure region close to you. For the list of currently supported Azure regions, see Supported regions.
   CLUSTER_NAME	A name for the new cluster to be created.
   ARC_APP_OBJECT_ID	The object ID value that you retrieved in the previous step.

   $url = "https://raw.githubusercontent.com/Azure/AKS-Edge/main/tools/scripts/AksEdgeQuickStart/AksEdgeQuickStartForAio.ps1"
   Invoke-WebRequest -Uri $url -OutFile .\AksEdgeQuickStartForAio.ps1
   Unblock-File .\AksEdgeQuickStartForAio.ps1
   Set-ExecutionPolicy -ExecutionPolicy Bypass -Scope Process -Force
   .\AksEdgeQuickStartForAio.ps1 -SubscriptionId "<SUBSCRIPTION_ID>" -TenantId "<TENANT_ID>" -ResourceGroupName "<RESOURCE_GROUP_NAME>"  -Location "<LOCATION>"  -ClusterName "<CLUSTER_NAME>" -CustomLocationOid "<ARC_APP_OBJECT_ID>"
   

   If there are any issues during deployment, including if your machine reboots as part of this process, run the whole set of commands again.

4. Run the following commands to check that the deployment was successful:

   Import-Module AksEdge
   Get-AksEdgeDeploymentInfo
   

   In the output of the Get-AksEdgeDeploymentInfo command, you should see that the cluster's Arc status is Connected.

Ubuntu

To prepare a K3s Kubernetes cluster on Ubuntu:

1. Install K3s following the instructions in the K3s quick-start guide.

2. Check to see that kubectl was installed as part of K3s. If not, follow the instructions to Install kubectl on Linux.

   kubectl version --client
   

3. Follow the instructions to Install Helm.

4. Create a K3s configuration yaml file in .kube/config:

   mkdir ~/.kube
   sudo KUBECONFIG=~/.kube/config:/etc/rancher/k3s/k3s.yaml kubectl config view --flatten > ~/.kube/merged
   mv ~/.kube/merged ~/.kube/config
   chmod  0600 ~/.kube/config
   export KUBECONFIG=~/.kube/config
   #switch to k3s context
   kubectl config use-context default
   sudo chmod 644 /etc/rancher/k3s/k3s.yaml
   

5. Run the following command to increase the user watch/instance limits.

   echo fs.inotify.max_user_instances=8192 | sudo tee -a /etc/sysctl.conf
   echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf
   
   sudo sysctl -p
   

6. For better performance, increase the file descriptor limit:

   echo fs.file-max = 100000 | sudo tee -a /etc/sysctl.conf
   
   sudo sysctl -p
   

Configure multi-node clusters for Azure Container Storage

On multi-node clusters with at least three nodes, you have the option of enabling fault tolerance for storage with Azure Container Storage enabled by Azure Arc when you deploy Azure IoT Operations. If you want to enable that option, prepare your multi-node cluster with the following steps:

1. Install the required NVME over TCP module for your kernel using the following command:

   sudo apt install linux-modules-extra-`uname -r`
   

   Note

   The minimum supported Linux kernel version is 5.1. At this time, there are known issues with 6.4 and 6.2. For the latest information, refer to Azure Container Storage release notes

2. On each node in your cluster, set the number of HugePages to 512 using the following command:

   HUGEPAGES_NR=512
   echo $HUGEPAGES_NR | sudo tee /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
   echo "vm.nr_hugepages=$HUGEPAGES_NR" | sudo tee /etc/sysctl.d/99-hugepages.conf
   

Arc-enable your cluster

Connect your cluster to Azure Arc so that it can be managed remotely.

AKS Edge Essentials

The AksEdgeQuickStartForAio.ps1 script that you ran in the previous section handled the steps to connect your cluster. You don't need to take any extra steps to Arc-enable.

Ubuntu

To connect your cluster to Azure Arc:

1. On the machine where you deployed the Kubernetes cluster, sign in with Azure CLI:

   az login
   

   If at any point you get an error that says Your device is required to be managed to access your resource, run az login again and make sure that you sign in interactively with a browser.

2. Set environment variables for your Azure subscription, location, a new resource group, and the cluster name as you want it to show up in your resource group.

   For the list of currently supported Azure regions, see Supported regions.

   # Id of the subscription where your resource group and Arc-enabled cluster will be created
   export SUBSCRIPTION_ID=<SUBSCRIPTION_ID>
   
   # Azure region where the created resource group will be located
   export LOCATION=<REGION>
   
   # Name of a new resource group to create which will hold the Arc-enabled cluster and Azure IoT Operations resources
   export RESOURCE_GROUP=<NEW_RESOURCE_GROUP_NAME>
   
   # Name of the Arc-enabled cluster to create in your resource group
   export CLUSTER_NAME=<NEW_CLUSTER_NAME>
   

1. After signing in, Azure CLI displays all of your subscriptions and indicates your default subscription with an asterisk *. To continue with your default subscription, select Enter. Otherwise, type the number of the Azure subscription that you want to use.

2. Register the required resource providers in your subscription:

   Note

   This step only needs to be run once per subscription. To register resource providers, you need permission to do the /register/action operation, which is included in subscription Contributor and Owner roles. For more information, see Azure resource providers and types.

   az provider register -n "Microsoft.ExtendedLocation"
   az provider register -n "Microsoft.Kubernetes"
   az provider register -n "Microsoft.KubernetesConfiguration"
   az provider register -n "Microsoft.IoTOperations"
   az provider register -n "Microsoft.DeviceRegistry"
   

3. Use the az group create command to create a resource group in your Azure subscription to store all the resources:

   az group create --location $LOCATION --resource-group $RESOURCE_GROUP --subscription $SUBSCRIPTION_ID
   

4. Remove the existing connected k8s cli if any

   az extension remove --name connectedk8s 
   

5. Download and install a preview version of the connectedk8s extension for Azure CLI.

   curl -L -o connectedk8s-1.10.0-py2.py3-none-any.whl https://github.com/AzureArcForKubernetes/azure-cli-extensions/raw/refs/heads/connectedk8s/public/cli-extensions/connectedk8s-1.10.0-py2.py3-none-any.whl   
   az extension add --upgrade --source connectedk8s-1.10.0-py2.py3-none-any.whl
   

6. Use the az connectedk8s connect command to Arc-enable your Kubernetes cluster and manage it as part of your Azure resource group:

   az connectedk8s connect --name $CLUSTER_NAME -l $LOCATION --resource-group $RESOURCE_GROUP --subscription $SUBSCRIPTION_ID --enable-oidc-issuer --enable-workload-identity
   

7. Get the cluster's issuer URL.

   az connectedk8s show --resource-group $RESOURCE_GROUP --name $CLUSTER_NAME --query oidcIssuerProfile.issuerUrl --output tsv
   

   Save the output of this command to use in the next steps.

8. Create a k3s config file.

   sudo nano /etc/rancher/k3s/config.yaml
   

9. Add the following content to the config.yaml file, replacing the <SERVICE_ACCOUNT_ISSUER> placeholder with your cluster's issuer URL.

   kube-apiserver-arg:
    - service-account-issuer=<SERVICE_ACCOUNT_ISSUER>
    - service-account-max-token-expiration=24h
   

10. Save the file and exit the nano editor.

11. Get the objectId of the Microsoft Entra ID application that the Azure Arc service uses in your tenant and save it as an environment variable.

    export OBJECT_ID=$(az ad sp show --id bc313c14-388c-4e7d-a58e-70017303ee3b --query id -o tsv)
    

12. Use the az connectedk8s enable-features command to enable custom location support on your cluster. This command uses the objectId of the Microsoft Entra ID application that the Azure Arc service uses. Run this command on the machine where you deployed the Kubernetes cluster:

    az connectedk8s enable-features -n $CLUSTER_NAME -g $RESOURCE_GROUP --custom-locations-oid $OBJECT_ID --features cluster-connect custom-locations
    

13. Restart K3s.

    systemctl restart k3s
    

Verify your cluster

To verify that your cluster is ready for Azure IoT Operations deployment, you can use the verify-host helper command in the Azure IoT Operations extension for Azure CLI. When run on the cluster host, this helper command checks connectivity to Azure Resource Manager and Microsoft Container Registry endpoints.

az iot ops verify-host

To verify that your Kubernetes cluster is Azure Arc-enabled, run the following command:

kubectl get deployments,pods -n azure-arc

The output looks like the following example:

NAME                                         READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/clusterconnect-agent         1/1     1            1           10m
deployment.apps/extension-manager            1/1     1            1           10m
deployment.apps/clusteridentityoperator      1/1     1            1           10m
deployment.apps/controller-manager           1/1     1            1           10m
deployment.apps/flux-logs-agent              1/1     1            1           10m
deployment.apps/cluster-metadata-operator    1/1     1            1           10m
deployment.apps/extension-events-collector   1/1     1            1           10m
deployment.apps/config-agent                 1/1     1            1           10m
deployment.apps/kube-aad-proxy               1/1     1            1           10m
deployment.apps/resource-sync-agent          1/1     1            1           10m
deployment.apps/metrics-agent                1/1     1            1           10m

NAME                                              READY   STATUS    RESTARTS        AGE
pod/clusterconnect-agent-5948cdfb4c-vzfst         3/3     Running   0               10m
pod/extension-manager-65b8f7f4cb-tp7pp            3/3     Running   0               10m
pod/clusteridentityoperator-6d64fdb886-p5m25      2/2     Running   0               10m
pod/controller-manager-567c9647db-qkprs           2/2     Running   0               10m
pod/flux-logs-agent-7bf6f4bf8c-mr5df              1/1     Running   0               10m
pod/cluster-metadata-operator-7cc4c554d4-nck9z    2/2     Running   0               10m
pod/extension-events-collector-58dfb78cb5-vxbzq   2/2     Running   0               10m
pod/config-agent-7579f558d9-5jnwq                 2/2     Running   0               10m
pod/kube-aad-proxy-56d9f754d8-9gthm               2/2     Running   0               10m
pod/resource-sync-agent-769bb66b79-z9n46          2/2     Running   0               10m
pod/metrics-agent-6588f97dc-455j8                 2/2     Running   0               10m

Next steps

Now that you have an Azure Arc-enabled Kubernetes cluster, you can deploy Azure IoT Operations.