Filter data in an Azure IoT Data Processor Preview pipeline
Important
Azure IoT Operations Preview – enabled by Azure Arc is currently in PREVIEW. You shouldn't use this preview software in production environments.
See the Supplemental Terms of Use for Microsoft Azure Previews for legal terms that apply to Azure features that are in beta, preview, or otherwise not yet released into general availability.
Use a filter stage to filter out messages that you don't need for further processing in the pipeline. The stage emits the original message to the filter stage unchanged if the filter criteria is met, otherwise the stage drops the message from the pipeline.
- Each pipeline partition filters out messages independently of other partitions.
- The output of the filter stage is the original message if the stage doesn't filter it out.
Prerequisites
To configure and use a filter pipeline stage, you need a deployed instance of Azure IoT Data Processor Preview that includes the optional Data Processor component.
Configure the stage
The filter stage JSON configuration defines the details of the stage. To author the stage, you can either interact with the form-based UI, or provide the JSON configuration on the Advanced tab:
Name | Value | Required | Default | Example |
---|---|---|---|---|
Display name | A name to show in the Data Processor UI. | Yes | - | Filter1 |
Description | A user-friendly description of what the filter stage does. | No | - | Filter out anomalies |
Query | The jq expression | Yes | - | .payload.temperature > 0 and .payload.pressure < 50 |
jq expression
Filter queries in Data Processor use the jq language to define the filter condition:
- The jq provided in the query must be syntactically valid.
- The result of the filter query must be a boolean value.
- Messages that evaluate to
true
are emitted unchanged from the filter stage to subsequent stages for further processing. Messages that evaluate tofalse
are dropped from the pipeline. - All messages for which the filter doesn't return a boolean result are treated as an error case and dropped from the pipeline.
- The filter stage adheres to the same restriction on jq usage as defined in the jq expression guide.
When you create a filter query to use in the filter stage:
- Test your filter query with your messages to make sure a boolean result is returned.
- Configure the filter query based on how the message arrives at the filter stage.
- To learn more about building your filter expressions, see thejq expressions guide.
Sample configuration
The following JSON example shows a complete filter stage configuration:
{
"displayName": "Filter name",
"description": "Filter description",
"query": "(.properties.responseTopic | contains(\"bar\")) or (.properties.responseTopic | contains(\"baz\")) and (.payload | has(\"temperature\")) and (.payload.temperature > 0)"
}
This filter checks for messages where .properties.responseTopic
contains bar
or baz
and the message payload has a property called temperature
with a value greater than 0
.
Related content
Feedback
https://aka.ms/ContentUserFeedback.
Coming soon: Throughout 2024 we will be phasing out GitHub Issues as the feedback mechanism for content and replacing it with a new feedback system. For more information see:Submit and view feedback for