Edit

Share via

Transform data in an Azure IoT Data Processor Preview pipeline

  • Article

Important

Azure IoT Operations Preview – enabled by Azure Arc is currently in PREVIEW. You shouldn't use this preview software in production environments.

See the Supplemental Terms of Use for Microsoft Azure Previews for legal terms that apply to Azure features that are in beta, preview, or otherwise not yet released into general availability.

Use the transform stage to carry out structural transformations on messages in a pipeline, such as:

  • Rename tags and properties
  • Unbatch data
  • Add new properties
  • Add calculated values

The transform stage uses jq to support data transformation:

  • Each pipeline partition transforms messages independently of each other.
  • The stage outputs a transformed message based on the jq expression you provide.
  • Create a jq expression to transform a message based on the structure of the incoming message to the stage.

Prerequisites

To configure and use a transform pipeline stage, you need:

  • A deployed instance of Azure IoT Data Processor Preview.
  • An understanding of jq expressions.

Configure the stage

The transform stage JSON configuration defines the details of the stage. To author the stage, you can either interact with the form-based UI or provide the JSON configuration on the Advanced tab:

Name Value Required Example
Name A name to show in the Data Processor UI. Yes Transform1
Description A user-friendly description of what the transform stage does. No Rename Tags
Query The transformation jq expression. Yes .payload.values |= (map({(.tag): (.numVal // .boolVal)}) | add)

Sample configuration

The following transformation example converts the array of tags in the input message into an object that contains all the tags and their values:

{
    "displayName": "TransformInput", 
    "description": "Make array of tags into one object", 
    "query": ".payload.values |= (map({(.tag): (.numVal // .boolVal)}) | add)"
}

The output from the transform stage looks like the following example:

{
  "systemProperties": {
    "partitionKey": "foo",
    "partitionId": 5,
    "timestamp": "2023-01-11T10:02:07Z"
  },
  "qos": 1,
  "topic": "/assets/foo/tags/bar",
  "properties": {
    "responseTopic": "outputs/foo/tags/bar",
    "contentType": "application/json",
    "payloadFormat": 1,
    "correlationData": "base64::Zm9v",
    "messageExpiry": 412
  },
  "userProperties": [
    {
      "key": "prop1",
      "value": "value1"
    },
    {
      "key": "prop2",
      "value": "value2"
    }
  ],
  "payload": {
    "values": {
      "temperature": 250,
      "pressure": 30,
      "humidity": 10,
      "runningStatus": true
    }
  }
}

Related content