Quickstart: Send asset telemetry to the cloud using the data lake connector for Azure IoT MQ Preview

Article
05/12/2024

Important

Azure IoT Operations Preview – enabled by Azure Arc is currently in PREVIEW. You shouldn't use this preview software in production environments.

See the Supplemental Terms of Use for Microsoft Azure Previews for legal terms that apply to Azure features that are in beta, preview, or otherwise not yet released into general availability.

In this quickstart, you use the data lake connector for Azure IoT MQ to forward telemetry from your OPC UA assets to a Microsoft Fabric lakehouse for storage and analysis.

Prerequisites

Before you begin this quickstart, you must complete the following quickstarts:

You also need a Microsoft Fabric subscription. You can sign up for a free Microsoft Fabric (Preview) Trial. In your Microsoft Fabric subscription, ensure that the following settings are enabled for your tenant:

To learn more, see Microsoft Fabric > About tenant settings.

What problem will we solve?

To use a tool such as Power BI to analyze your OPC UA data, you need to send the data to a cloud-based storage service. The data lake connector for Azure IoT MQ subscribes to MQTT topics and ingests the messages into Delta tables in a Microsoft Fabric lakehouse. The next quickstart shows you how to use Power BI to analyze the data in the lakehouse.

Grant access to your Microsoft Fabric workspace

You need to allow the MQ extension on your cluster to connect to your Microsoft Fabric workspace. You made a note of the MQ extension name in the deployment quickstart. The name of the extension looks like mq-z2ewy.

Tip

If you need to find the unique name assigned to your MQ extension, run the following command in your Codespaces terminal to list your cluster extensions: az k8s-extension list --resource-group <your-resource-group-name> --cluster-name $CLUSTER_NAME --cluster-type connectedClusters -o table

Navigate to the Microsoft Fabric Power BI experience. To ensure you can see the Manage access option in your Microsoft Fabric workspace, create a new workspace:

Select Workspaces in the left navigation bar, then select New workspace:
Enter a name for your workspace, such as yournameaioworkspace, and select Apply. Make a note of this name, you need it later.

Tip

Don't include any spaces in the name of your workspace.

To grant the MQ extension access to your Microsoft Fabric workspace:

In your Microsoft Fabric workspace, select Manage access:
Select Add people or groups, then paste the name of the MQ extension you made a note of previously and grant it at least Contributor access:
Select Add to grant the MQ extension contributor permissions in the workspace.

Create a lakehouse

Create a lakehouse in your Microsoft Fabric workspace:

Select New and More options, then choose Lakehouse from the list.
Enter aiomqdestination as the name for your lakehouse and select Create.

Configure a connector

Your codespace comes with the following sample connector configuration file, /workspaces/explore-iot-operations/samples/quickstarts/datalake-connector.yaml:

apiVersion: mq.iotoperations.azure.com/v1beta1
kind: DataLakeConnector
metadata:
  name: my-datalake-connector
  namespace: azure-iot-operations
spec:
  protocol: v5
  image:
    repository: mcr.microsoft.com/azureiotoperations/datalake
    tag: 0.4.0-preview
    pullPolicy: IfNotPresent
  instances: 1
  logLevel: info
  databaseFormat: delta
  target:
    fabricOneLake:
      endpoint: https://onelake.dfs.fabric.microsoft.com
      names:
        workspaceName: <your-workspace-name>
        lakehouseName: aiomqdestination
      fabricPath: tables
      authentication:
        systemAssignedManagedIdentity:
          audience: https://storage.azure.com
  localBrokerConnection:
    endpoint: aio-mq-dmqtt-frontend:8883
    tls:
      tlsEnabled: true
      trustedCaCertificateConfigMap: aio-ca-trust-bundle-test-only
    authentication:
      kubernetes: {}
---
apiVersion: mq.iotoperations.azure.com/v1beta1
kind: DataLakeConnectorTopicMap
metadata:
  name: datalake-topicmap
  namespace: azure-iot-operations
spec:
  dataLakeConnectorRef: "my-datalake-connector"
  mapping:
    allowedLatencySecs: 1
    messagePayloadType: "json"
    maxMessagesPerBatch: 10
    clientId: id
    mqttSourceTopic: "azure-iot-operations/data/thermostat"
    qos: 1
    table:
      tableName: OPCUA
      schema:
      - name: externalAssetId
        format: utf8
        optional: false
        mapping: $property.externalAssetId
      - name: assetName
        format: utf8
        optional: false
        mapping: DataSetWriterName
      - name: CurrentTemperature
        format: float32
        optional: false
        mapping: Payload.temperature.Value
      - name: Pressure
        format: float32
        optional: true
        mapping: "Payload.Tag 10.Value"
      - name: Timestamp
        format: timestamp
        optional: false
        mapping: $received_time

Open the datalake-connector.yaml file in a text editor and replace <your-workspace-name> with the name of your Microsoft Fabric workspace. You made a note of this value when you created the workspace.
Save the file.
Run the following command to create the connector:
```
kubectl apply -f datalake-connector.yaml
```

After a short time, the data from your MQ broker begins to populate the table in your lakehouse.

Screenshot that shows data from the pipeline appearing in the lakehouse table.

Tip

Make sure that no other processes write to the OPCUA table in your lakehouse. If you write to the table from multiple sources, you might see corrupted data in the table.

How did we solve the problem?

In this quickstart, you used the data lake connector for Azure IoT MQ to ingest the data into a Microsoft Fabric lakehouse in the cloud. In the next quickstart, you use Power BI to analyze the data in the lakehouse.

Clean up resources

If you're not going to continue to use this deployment, delete the Kubernetes cluster where you deployed Azure IoT Operations and remove the Azure resource group that contains the cluster.

You can also delete your Microsoft Fabric workspace.

Next step

Quickstart: Get insights from your asset telemetry

Share via