Edit

Share via

Configure Azure IoT Layered Network Management Preview on level 4 cluster

  • Article

Important

Azure IoT Operations Preview – enabled by Azure Arc is currently in PREVIEW. You shouldn't use this preview software in production environments.

See the Supplemental Terms of Use for Microsoft Azure Previews for legal terms that apply to Azure features that are in beta, preview, or otherwise not yet released into general availability.

Azure IoT Layered Network Management Preview is one of the Azure IoT Operations Preview components. However, it can be deployed individually to the top network layer for supporting the Azure IoT Operations in the lower layer. In the top level of your network layers (usually level 4 of the ISA-95 network architecture), the cluster and Layered Network Management service have direct internet access. Once the setup is completed, the Layered Network Management service is ready for receiving network traffic from the child layer and forwards it to Azure Arc.

Prerequisites

Meet the following minimum requirements for deploying the Layered Network Management individually on the system.

Set up Kubernetes cluster in Level 4

To set up only Layered Network Management, the prerequisites are simpler than an Azure IoT Operations deployment. It's optional to fulfill the general requirements for Azure IoT Operations in Prepare your Kubernetes cluster.

The following steps for setting up AKS Edge Essentials and K3S Kubernetes cluster are verified by Microsoft.

Prepare an Ubuntu machine

  1. Ubuntu 22.04 LTS is the recommended version for the host machine.

  2. Install Helm 3.8.0 or later.

  3. Install Kubectl.

  4. Install the Azure CLI. You can install the Azure CLI directly onto the level 4 machine or on another developer or jumpbox machine if you plan to access the level 3 cluster remotely. If you choose to access the Kubernetes cluster remotely to keep the cluster host clean, you run the kubectl and az" related commands from the developer machine for the rest of the steps in this article.

Create the K3S cluster

  1. Install K3S with the following command:

    curl -sfL https://get.k3s.io | sh -s - --disable=traefik --write-kubeconfig-mode 644

    Important

    Be sure to use the --disable=traefik parameter to disable treafik. Otherwise, you might have an issue when you try to allocate public IP for the Layered Network Management service in later steps.

  2. Copy the K3s configuration yaml file to .kube/config.

    mkdir ~/.kube
cp ~/.kube/config ~/.kube/config.back
sudo KUBECONFIG=~/.kube/config:/etc/rancher/k3s/k3s.yaml kubectl config view --flatten > ~/.kube/merged
mv ~/.kube/merged ~/.kube/config
chmod  0600 ~/.kube/config
export KUBECONFIG=~/.kube/config
#switch to k3s context
kubectl config use-context default

Arc enable the cluster

  1. Sign in with Azure CLI. To avoid permission issues later, it's important that you sign in interactively using a browser window:

    az login

  2. Set environment variables for the setup steps. Replace values in <> with valid values or names of your choice. The CLUSTER_NAME and RESOURCE_GROUP are created based on the names you provide:

    # Id of the subscription where your resource group and Arc-enabled cluster will be created
$SUBSCRIPTION_ID = "<subscription-id>"
# Azure region where the created resource group will be located
# Currently supported regions: : "westus3" or "eastus2"
$LOCATION = "WestUS3"
# Name of a new resource group to create which will hold the Arc-enabled cluster and Azure IoT Operations resources
$RESOURCE_GROUP = "<resource-group-name>"
# Name of the Arc-enabled cluster to create in your resource group
$CLUSTER_NAME = "<cluster-name>"

  3. Set the Azure subscription context for all commands:

    az account set -s $SUBSCRIPTION_ID

  4. Register the required resource providers in your subscription:

    Note

    This is a one-time configuration per subscription.

    az provider register -n "Microsoft.ExtendedLocation"
az provider register -n "Microsoft.Kubernetes"
az provider register -n "Microsoft.KubernetesConfiguration"
az provider register -n "Microsoft.IoTOperationsOrchestrator"
az provider register -n "Microsoft.IoTOperationsMQ"
az provider register -n "Microsoft.IoTOperationsDataProcessor"
az provider register -n "Microsoft.DeviceRegistry"

  5. Use the az group create command to create a resource group in your Azure subscription to store all the resources:

    az group create --location $LOCATION --resource-group $RESOURCE_GROUP --subscription $SUBSCRIPTION_ID

  6. Use the az connectedk8s connect command to Arc-enable your Kubernetes cluster and manage it in the resource group you created in the previous step:

    az connectedk8s connect -n $CLUSTER_NAME -l $LOCATION -g $RESOURCE_GROUP --subscription $SUBSCRIPTION_ID

    Tip

    If the connectedk8s commands fail, try using the cmdlets in Connect your AKS Edge Essentials cluster to Arc.

Deploy Layered Network Management Preview Service to the cluster

Once your Kubernetes cluster is Arc-enabled, you can deploy the Layered Network Management service to the cluster.

Install the Layered Network Management Preview operator

  1. Run the following command. Replace the placeholders <RESOURCE GROUP> and <CLUSTER NAME> with your Arc onboarding information from an earlier step.

    az login

az k8s-extension create --resource-group <RESOURCE GROUP> --name kind-lnm-extension --cluster-type connectedClusters --cluster-name <CLUSTER NAME> --auto-upgrade false --extension-type Microsoft.IoTOperations.LayeredNetworkManagement --version 0.1.0-preview --release-train preview

  2. Use the kubectl command to verify the Layered Network Management operator is running.

    kubectl get pods -n azure-iot-operations

    NAME                                   READY   STATUS    RESTARTS   AGE
azedge-lnm-operator-598cc495c-5428j   1/1     Running   0          28h

Configure Layered Network Management Preview Service

Create the Layered Network Management custom resource.

  1. Create a lnm-cr.yaml file as specified:

    apiVersion: layerednetworkmgmt.iotoperations.azure.com/v1beta1
kind: Lnm
metadata:
  name: level4
  namespace: azure-iot-operations
spec:
  image:
    pullPolicy: IfNotPresent
    repository: mcr.microsoft.com/oss/envoyproxy/envoy-distroless
    tag: v1.27.0
  replicas: 1
  logLevel: "debug"
  openTelemetryMetricsCollectorAddr: "http://aio-otel-collector.azure-iot-operations.svc.cluster.local:4317"
  level: 4
  allowList:
    enableArcDomains: true
    domains:
    - destinationUrl: "*.ods.opinsights.azure.com"
      destinationType: external
    - destinationUrl: "*.oms.opinsights.azure.com"
      destinationType: external
    - destinationUrl: "*.monitoring.azure.com"
      destinationType: external
    - destinationUrl: "*.handler.control.monitor.azure.com"
      destinationType: external
    - destinationUrl: "quay.io"
      destinationType: external
    - destinationUrl: "*.quay.io"
      destinationType: external
    - destinationUrl: "docker.io"
      destinationType: external
    - destinationUrl: "*.docker.io"
      destinationType: external
    - destinationUrl: "*.docker.com"
      destinationType: external
    - destinationUrl: "gcr.io"
      destinationType: external
    - destinationUrl: "*.googleapis.com"
      destinationType: external
    - destinationUrl: "login.windows.net"
      destinationType: external
    - destinationUrl: "graph.windows.net"
      destinationType: external
    - destinationUrl: "msit-onelake.pbidedicated.windows.net"
      destinationType: external
    - destinationUrl: "*.vault.azure.net"
      destinationType: external
    - destinationUrl: "*.k8s.io"
      destinationType: external
    - destinationUrl: "*.pkg.dev"
      destinationType: external
    - destinationUrl: "github.com"
      destinationType: external
    - destinationUrl: "raw.githubusercontent.com"
      destinationType: external
    sourceIpRange:
    - addressPrefix: "0.0.0.0"
      prefixLen: 0

    For debugging or experimentation, you can change the value of loglevel parameter to debug.

  2. Create the Custom Resource to create a Layered Network Management instance.

    kubectl apply -f lnm-cr.yaml

  3. View the Layered Network Management Kubernetes service:

    kubectl get services -n azure-iot-operations

    NAME           TYPE           CLUSTER-IP    EXTERNAL-IP   PORT(S)                                      AGE
lnm-level-4   LoadBalancer   10.43.91.54   192.168.0.4   80:30530/TCP,443:31117/TCP,10000:31914/TCP   95s

Add iptables configuration

Important

This step is for AKS Edge Essentials only.

The Layered Network Management deployment creates a Kubernetes service of type LoadBalancer. To ensure that the service is accessible from outside the Kubernetes cluster, you need to map the underlying Windows host's ports to the appropriate ports on the Layered Network Management service.

netsh interface portproxy add v4tov4 listenport=443 listenaddress=0.0.0.0 connectport=443 connectaddress=192.168.0.4
netsh interface portproxy add v4tov4 listenport=10000 listenaddress=0.0.0.0 connectport=10000 connectaddress=192.168.0.4

After these commands are run successfully, traffic received on ports 443 and 10000 on the Windows host is routed through to the Kubernetes service. When configuring customized DNS for the child level network layer, you direct the network traffic to the IP of this Windows host and then to the Layered Network Management service running on it.

Related content