Deploy Azure IoT Operations Preview to an Arc-enabled Kubernetes cluster

Important

Azure IoT Operations Preview – enabled by Azure Arc is currently in preview. You shouldn't use this preview software in production environments.

You'll need to deploy a new Azure IoT Operations installation when a generally available release is made available. You won't be able to upgrade a preview installation.

See the Supplemental Terms of Use for Microsoft Azure Previews for legal terms that apply to Azure features that are in beta, preview, or otherwise not yet released into general availability.

Learn how to deploy Azure IoT Operations Preview to a Kubernetes cluster using the Azure CLI or Azure portal.

In this article, we discuss Azure IoT Operations deployments and instances, which are two different concepts:

• An Azure IoT Operations deployment describes all of the components and resources that enable the Azure IoT Operations scenario. These components and resources include:

  • An Azure IoT Operations instance
  • Arc extensions
  • Custom locations
  • Resource sync rules
  • Resources that you can configure in your Azure IoT Operations solution, like assets and asset endpoints.

• An Azure IoT Operations instance is the parent resource that bundles the suite of services that are defined in What is Azure IoT Operations Preview? like MQTT broker, dataflows, and OPC UA connector.

When we talk about deploying Azure IoT Operations, we mean the full set of components that make up a deployment. Once the deployment exists, you can view, manage, and update the instance.

Prerequisites

Cloud resources:

• An Azure subscription.

• An Azure key vault. To create a new key vault, use the az keyvault create command:

  az keyvault create --enable-rbac-authorization --name "<NEW_KEYVAULT_NAME>" --resource-group "<RESOURCE_GROUP>"
  

• Azure access permissions. For more information, see Deployment details > Required permissions.

Development resources:

• Azure CLI installed on your development machine. This scenario requires Azure CLI version 2.64.0 or higher. Use az --version to check your version and az upgrade to update if necessary. For more information, see How to install the Azure CLI.

• The Azure IoT Operations extension for Azure CLI. Use the following command to add the extension or update it to the latest version:

  az extension add --upgrade --name azure-iot-ops
  

A cluster host:

• An Azure Arc-enabled Kubernetes cluster with the custom location and workload identity features enabled. If you don't have one, follow the steps in Prepare your Azure Arc-enabled Kubernetes cluster.

  If you deployed Azure IoT Operations to your cluster previously, uninstall those resources before continuing. For more information, see Update Azure IoT Operations.

• Verify that your cluster host is configured correctly for deployment by using the verify-host command on the cluster host:

  az iot ops verify-host
  

• (Optional) Prepare your cluster for observability before deploying Azure IoT Operations: Configure observability.

Deploy

Use the Azure portal or Azure CLI to deploy Azure IoT Operations to your Arc-enabled Kubernetes cluster.

The Azure portal deployment experience is a helper tool that generates a deployment command based on your resources and configuration. The final step is to run an Azure CLI command, so you still need the Azure CLI prerequisites described in the previous section.

Azure portal

1. In the Azure portal, search for and select Azure IoT Operations.

2. Select Create.

3. On the Basics tab, provide the following information:

   Parameter	Value
   Subscription	Select the subscription that contains your Arc-enabled cluster.
   Resource group	Select the resource group that contains your Arc-enabled cluster.
   Cluster name	Select the cluster that you want to deploy Azure IoT Operations to.
   Custom location name	Optional: Replace the default name for the custom location.

   

4. Select Next: Configuration.

5. On the Configuration tab, provide the following information:

   Parameter	Value
   Azure IoT Operations name	Optional: Replace the default name for the Azure IoT Operations instance.
   MQTT broker configuration	Optional: Edit the default settings for the MQTT broker. For more information, see Configure core MQTT broker settings.
   Dataflow profile configuration	Optional: Edit the default settings for dataflows. For more information, see Configure dataflow profile.

   

6. Select Next: Dependency management.

7. On the Dependency management tab, select an existing schema registry or use these steps to create one:

   1. Select Create new.

   2. Provide a Schema registry name and Schema registry namespace.

   3. Select Select Azure Storage container.

   4. Choose a storage account from the list of hierarchical namespace-enabled accounts, or select Create to create one.

      Schema registry requires an Azure Storage account with hierarchical namespace and public network access enabled. When creating a new storage account, choose a General purpose v2 storage account type and set Hierarchical namespace to Enabled.

   5. Select a container in your storage account or select Container to create one.

   6. Select Apply to confirm the schema registry configurations.

8. On the Dependency management tab, select the Secure settings deployment option.

   

9. In the Deployment options section, provide the following information:

   Parameter	Value
   Subscription	Select the subscription that contains your Azure key vault.
   Azure Key Vault	Select an Azure key vault select Create new. Ensure that your key vault has Vault access policy as its permission model. To check this setting, select Manage selected vault > Settings > Access configuration.
   User assigned managed identity for secrets	Select an identity or select Create new.
   User assigned managed identity for AIO components	Select an identity or select Create new. Don't use the same managed identity as the one you selected for secrets.

   

10. Select Next: Automation.

11. One at a time, run each Azure CLI command on the Automation tab in a terminal:

    1. Sign in to Azure CLI interactively with a browser even if you already signed in before. If you don't sign in interactively, you might get an error that says Your device is required to be managed to access your resource when you continue to the next step to deploy Azure IoT Operations.

       az login
       

    2. If you didn't prepare your Azure CLI environment as described in the prerequisites, do so now in a terminal of your choice:

       az upgrade
       az extension add --upgrade --name azure-iot-ops
       

    3. If you chose to create a new schema registry on the previous tab, copy and run the az iot ops schema registry create command.

    4. Prepare your cluster for Azure IoT Operations deployment by deploying dependencies and foundational services, including schema registry. Copy and run the az iot ops init command.

       Tip

       The init command only needs to be run once per cluster. If you're reusing a cluster that already had Azure IoT Operations version 0.7.0 deployed on it, you can skip this step.

       This command might take several minutes to complete. You can watch the progress in the deployment progress display in the terminal.

    5. Deploy Azure IoT Operations to your cluster. Copy and run the az iot ops create command.

       This command might take several minutes to complete. You can watch the progress in the deployment progress display in the terminal.

    6. Enable secret sync on your Azure IoT Operations instance. Copy and run the az iot ops secretsync enable command. This command:

       • Creates a federated identity credential using the user-assigned managed identity.
       • Adds a role assignment to the user-assigned managed identity for access to the Azure Key Vault.
       • Adds a minimum secret provider class associated with the Azure IoT Operations instance.

    7. Assign a user-assigned managed identity to your Azure IoT Operations instance. Copy and run the az iot ops identity assign command.

       This command also creates a federated identity credential using the OIDC issuer of the indicated connected cluster and the Azure IoT Operations service account.

12. Once all of the Azure CLI commands complete successfully, you can close the Install Azure IoT Operations wizard.

Azure CLI

1. Sign in to Azure CLI interactively with a browser even if you already signed in before.

   az login
   

   If at any point you get an error that says Your device is required to be managed to access your resource, run az login again and make sure that you sign in interactively with a browser.

Create a storage account and schema registry

Azure IoT Operations requires a schema registry on your cluster. Schema registry requires an Azure storage account so that it can synchronize schema information between cloud and edge.

1. Create a storage account with hierarchical namespace enabled.

   az storage account create --name <NEW_STORAGE_ACCOUNT_NAME> --resource-group <RESOURCE_GROUP> --enable-hierarchical-namespace
   

2. Create a schema registry that connects to your storage account.

   az iot ops schema registry create --name <NEW_SCHEMA_REGISTRY_NAME> --resource-group <RESOURCE_GROUP> --registry-namespace <NEW_SCHEMA_REGISTRY_NAMESPACE> --sa-resource-id $(az storage account show --name <STORAGE_ACCOUNT_NAME> --resource-group <RESOURCE_GROUP> -o tsv --query id)
   

   Note

   This command requires that you have role assignment write permissions because it assigns a role to give schema registry access to the storage account. By default, the role is the built-in Storage Blob Data Contributor role, or you can create a custom role with restricted permissions to assign instead.

   Use the optional parameters to customize your schema registry, including:

   Optional parameter	Value	Description
   --custom-role-id	Role definition ID	Provide a custom role ID to assign to the schema registry instead of the default Storage Blob Data Contributor role. At a minimum, the role needs blob read and write permissions. Format: /subscriptions/<SUBSCRIPTION_ID>/providers/Microsoft.Authorization/roleDefinitions/<ROLE_ID>.
   --sa-container	string	Storage account container to store schemas. If this container doesn't exist, this command creates it. The default container name is schemas.

3. Copy the resource ID from the output of the schema registry create command to use in the next section.

Deploy Azure IoT Operations

1. Prepare your cluster with the dependencies that Azure IoT Operations requires by running az iot ops init.

   Tip

   The init command only needs to be run once per cluster. If you're reusing a cluster that already had Azure IoT Operations version 0.7.0 deployed on it, you can skip this step.

   az iot ops init --cluster <CLUSTER_NAME> --resource-group <RESOURCE_GROUP> --sr-resource-id <SCHEMA_REGISTRY_RESOURCE_ID>
   

   This command might take several minutes to complete. You can watch the progress in the deployment progress display in the terminal.

   Use the optional parameters to customize your cluster, including:

   Optional parameter	Value	Description
   --no-progress		Disable the deployment progress display in the terminal.
   --enable-fault-tolerance	false, true	Enable fault tolerance for Azure Arc Container Storage. At least three cluster nodes are required.
   --ops-config	observability.metrics.openTelemetryCollectorAddress=<FULLNAMEOVERRIDE>.azure-iot-operations.svc.cluster.local:<GRPC_ENDPOINT>	If you followed the optional prerequisites to prepare your cluster for observability, provide the OpenTelemetry (OTel) collector address you configured in the otel-collector-values.yaml file. The sample values used in Configure observability are fullnameOverride=aio-otel-collector and grpc.enpoint=4317.
   --ops-config	observability.metrics.exportInternalSeconds=<CHECK_INTERVAL>	If you followed the optional prerequisites to prepare your cluster for observability, provide the check_interval value you configured in the otel-collector-values.yaml file. The sample value used in Configure observability is check_interval=60.

2. Deploy Azure IoT Operations. This command takes several minutes to complete:

   az iot ops create --name <NEW_INSTANCE_NAME> --cluster <CLUSTER_NAME> --resource-group <RESOURCE_GROUP>
   

   This command might take several minutes to complete. You can watch the progress in the deployment progress display in the terminal.

   Use the optional parameters to customize your instance, including:

   Optional parameter	Value	Description
   --no-progress		Disable the deployment progress display in the terminal.
   --enable-rsync-rules		Enable the resource sync rules on the instance to project resources from the edge to the cloud.
   --add-insecure-listener		Add an insecure 1883 port config to the default listener. Not for production use.
   --custom-location	String	Provide a name for the custom location created for your cluster. The default value is location-{hash(5)}.
   --broker-config-file	Path to JSON file	Provide a configuration file for the MQTT broker. For more information, see Advanced MQTT broker config and Configure core MQTT broker settings.

Once the create command completes successfully, you have a working Azure IoT Operations instance running on your cluster. At this point, your instance is configured for most testing and evaluation scenarios. If you want to prepare your instance for production scenarios, continue to the next section to enable secure settings.

Set up secret management and user assigned managed identity (optional)

Secret management for Azure IoT Operations uses Azure Secret Store to sync the secrets from an Azure Key Vault and store them on the edge as Kubernetes secrets.

Azure secret requires a user-assigned managed identity with access to the Azure Key Vault where secrets are stored. Dataflows also requires a user-assigned managed identity to authenticate cloud connections.

1. Create an Azure Key Vault if you don't have one available. Use the az keyvault create command.

   az keyvault create --resource-group "<RESOURCE_GROUP>" --location "<LOCATION>" --name "<KEYVAULT_NAME>" --enable-rbac-authorization 
   

2. Create a user-assigned managed identity that will be assigned access to the Azure Key Vault.

   az identity create --name "<USER_ASSIGNED_IDENTITY_NAME>" --resource-group "<RESOURCE_GROUP>" --location "<LOCATION>" --subscription "<SUBSCRIPTION>" 
   

3. Configure the Azure IoT Operations instance for secret synchronization. This command:

   • Creates a federated identity credential using the user-assigned managed identity.
   • Adds a role assignment to the user-assigned managed identity for access to the Azure Key Vault.
   • Adds a minimum secret provider class associated with the Azure IoT Operations instance.

   az iot ops secretsync enable --name <INSTANCE_NAME> --resource-group <RESOURCE_GROUP> --mi-user-assigned <USER_ASSIGNED_MI_RESOURCE_ID> --kv-resource-id <KEYVAULT_RESOURCE_ID>
   

4. Create a user-assigned managed identity which can be used for cloud connections. Don't use the same identity as the one used to set up secrets management.

   az identity create --name "<USER_ASSIGNED_IDENTITY_NAME>" --resource-group "<RESOURCE_GROUP>" --location "<LOCATION>" --subscription "<SUBSCRIPTION>" 
   
   You will need to grant the identity permission to whichever cloud resource this will be used for. 
   
   

5. Run the following command to assign the identity to the Azure IoT Operations instance. This command also creates a federated identity credential using the OIDC issuer of the indicated connected cluster and the Azure IoT Operations service account.

   az iot ops identity assign --name <INSTANCE_NAME> --resource-group <RESOURCE_GROUP> --mi-user-assigned <USER_ASSIGNED_MI_RESOURCE_ID>
   

While the deployment is in progress, you can watch the resources being applied to your cluster. If your terminal supports it, the init and create commands display the deployment progress.

Otherwise, or if you choose to disable the progress interface with --no-progress added to the commands, you can use kubectl commands to view the pods on your cluster:

kubectl get pods -n azure-iot-operations

It can take several minutes for the deployment to complete. Rerun the get pods command to refresh your view.

After the deployment is complete, use az iot ops check to evaluate IoT Operations service deployment for health, configuration, and usability. The check command can help you find problems in your deployment and configuration.

az iot ops check

You can also check the configurations of topic maps, QoS, and message routes by adding the --detail-level 2 parameter for a verbose view.

Next steps

If your components need to connect to Azure endpoints like SQL or Fabric, learn how to Manage secrets for your Azure IoT Operations Preview deployment.