Events
Mar 31, 11 PM - Apr 2, 11 PM
The biggest Fabric, Power BI, and SQL learning event. March 31 – April 2. Use code FABINSIDER to save $400.
Register todayThis browser is no longer supported.
Upgrade to Microsoft Edge to take advantage of the latest features, security updates, and technical support.
The job diagram in the Azure portal can help you visualize your job's query steps (logical concept) or streaming node (physical concept) with its input source, output destination, and metrics. You can use the job diagram to examine the metrics for each step or streaming node and quickly identify the source of a problem when you troubleshoot issues.
There are two types of job diagrams:
Physical diagram: it visualizes the key metrics of Stream Analytics job with the physical computation concept: streaming node dimension. A streaming node represents a set of compute resources that's used to process job's input data. To learn more details about the streaming node dimension, see Azure Stream Analytics node name dimension.
Inside each streaming node, there are Stream Analytics processors available for processing the stream data. Each processor represents one or more steps in your query. You can visualize the processor topology in each streaming node by using the processor diagram in physical job diagram.
Logical diagram: it visualizes the key metrics of Stream Analytics job with the logical concept: query step based on job's queries. To learn more, see Debugging with the logical job diagram (preview) in Azure portal.
This article describes the two types of job diagrams to guide you.
Important
This feature is currently in PREVIEW. See the Supplemental Terms of Use for Microsoft Azure Previews for legal terms that apply to Azure features that are in beta, preview, or otherwise not yet released into general availability.
In the Azure portal, locate and select a Stream Analytics job. Then select Job diagram (preview) under Developer tools:
In the top left corner, you can switch the two types of job diagram by clicking Logical, Physical.
The following screenshot shows a physical job diagram with a default time period (last 30 minutes).
Command bar section: it's the command area where you can configure the time range of the job metrics, switch/configure heatmap visualization, search a streaming node, and switch view between Diagram and Table.
Heatmap settings: the heatmap setting enables you to sort the nodes in diagram based on your wanted metrics and sorting type. The metrics can be CPU/memory utilization, watermark delay, input event, and backlogged input events.
Time range: you can choose different time range and job run to view the diagram and metrics.
Job run: Job run is inside Time range. When a job is started, restarted or scaled-up/down (SU changes), a new job run will be generated. One job run maps on physical job diagram.
Diagram/Table view switcher: you can switch the view between diagram and table. The table view is shown as below:
Diagram/Table section: it's the place where you can view the metrics (aggregated within the selected time range) in streaming node level with diagram view or table view. Each box in this section represents a streaming node that is used to process the input data. The metrics on each node are:
For more information about the metrics definition, see Azure Stream Analytics node name dimension.
Chart section: it's the place where you can view the historical metrics data within the selected time range. The default metrics shown in the default chart are SU (Memory) % Utilization and CPU % Utilization". You can also add more charts by clicking Add chart.
The Diagram/Table section and Chart section can be interactive with each other. You can select multiple nodes in Diagram/Table section to get the metrics in Chart section filtered by the selected nodes and vice versa.
To learn more about how to debug with physical diagram, see Debugging with the physical job diagram (preview) in Azure portal.
The processor diagram in physical job diagram visualizes the processor topology inside the specific streaming node. To access the processor diagram, you'll need to open the physical job diagram, locate a streaming node that you want to check its processor diagram, and then open the processor diagram by clicking the streaming node name.
Processor type: it shows the type of the processor, stands for certain data processing purpose. It's available in each processor node.
Processor type | Description |
---|---|
Input or Output | This processor is used for reading input or writing output data streams. |
ReferenceData | This processor is used for fetching the reference data. |
Computing | This processor is used for processing the stream data according to the query logic, for example, aggregating, filtering, grouping with window, etc. To learn more about the stream data computation query functions, see Azure Stream Analytics Query Language Reference. |
MarshallerUpstream and MarshallerDownstream | When there's stream data interaction among streaming nodes, there will be two marshaller processors: 1). MarshallerUpstream for sending the data in the upstream streaming node and 2). MarshallerDownstream for receiving the data in the downstream streaming node. |
Merger | This processor is to receive the crossing-partition stream data, which were outputted from several upstream streaming nodes. The best practice to optimize job performance is to update query to remove the merger processor to make the job become parallel since the merger processor is the bottleneck of the job. The job diagram simulator feature within Visual Studio Code ASA extension can help you simulating your query locally when you optimizing your job query. To learn more, see Optimize query using job diagram simulator (preview). |
Adapter type: it shows the type of the input or output adapter. Stream Analytics supports various input sources and output destinations. Each input source or output destination has a dedicated adapter type. It's only available in input processor and output processor. For example, "InputBlob" represents the ADLS Gen2 input where the input processor receives the data from; "OutputDocumentDb" represents the Cosmos DB output where the output processor outputs the data to.
To learn more details of the input and output types, see Azure Stream Analytics inputs overview, and Azure Stream Analytics outputs overview.
Partition IDs: it shows which partition IDs' data are being processed by this processor. It's only available in input processor and output processor.
Serializer type: it shows the type of the serialization. Stream Analytics supports several serialization types. It's only available in input processor and output processor.
The logical job diagram has a similar layout to the physical diagram, with three sections, but it has different metrics and configuration settings.
Command bar section: in logical diagram, you can operate the cloud job (Stop, Delete), and configure the time range of the job metrics. The diagram view is only available for logical diagrams.
Diagram section: the node box in this selection represents the job's input, output, and query steps. You can view the metrics in the node directly or in the chart section interactively by clicking certain node in this section. For more information about the metrics definition, see Azure Stream Analytics node name dimension.
Chart section: the chart section in a logical diagram has two tabs: Metrics and Activity Logs.
When a logical job diagram is loaded, this job's metrics: Watermark delay, Input events, Output Events, and Backlogged Input Events are shown in the chart section for the latest 30 minutes.
The interaction between Diagram section and Chart section is also available in logical diagram as well. The metrics data will be filtered by the node's properties.
To learn more about how to debug with logical diagrams, see Debugging with the logical job diagram (preview) in Azure portal.
Events
Mar 31, 11 PM - Apr 2, 11 PM
The biggest Fabric, Power BI, and SQL learning event. March 31 – April 2. Use code FABINSIDER to save $400.
Register todayTraining
Module
Visualize real-time data with Azure Stream Analytics and Power BI - Training
By combining the stream processing capabilities of Azure Stream Analytics and the data visualization capabilities of Microsoft Power BI, you can create real-time data dashboards.
Certification
Microsoft Certified: Azure Data Engineer Associate - Certifications
Demonstrate understanding of common data engineering tasks to implement and manage data engineering workloads on Microsoft Azure, using a number of Azure services.