Monitor Apache Spark run series

The Apache Spark run series automatically classifies the following into respective run series:

• Your Apache Spark applications from your recurring pipeline activities, or manual notebook runs.
• Apache Spark job runs from the same notebook or Apache Spark job definition into respective run series.

The run series feature visually represents the duration trend for each Spark application instance along with the corresponding data input and output trendline for Spark applications. It also auto-scans the run series and detects whether there are any anomalous Spark application runs. This feature enables you to view details for a particular Spark application.

Access the monitor run series feature

You can access the monitor run series feature from the Monitoring hub's historical view:

1. Open the Microsoft Fabric portal and go to Monitoring hub menu.

2. Open your Spark job definition or notebook and expand its More options drop-down list and then select Historical runs.

3. Select the job you want to view and expand More options, then select Monitor run series.

   

You can access the monitor run series feature from the notebook or Spark job definition's Recent runs panel:

1. Open the Microsoft Fabric homepage and select a workspace where you want to view the job.

2. Selecting Spark job definition or Notebook item context menu shows the recent run option.

3. Select Recent runs.

4. select an application and expand its More options drop-down list and then select Monitor run series.

   

You can access the monitor run series feature from the Spark application Monitoring detail page:

1. Go to the Apache Spark application monitoring details page.

2. Select the Monitor run series option.

   

View Spark application performance

In the Spark runs graph, you can view the duration trend of this run series. Each vertical bar represents an instance of the notebook/Spark job definition activity run, and its height indicates the run duration. You can also click on each running instance to view more detailed information and zoom in or out on specific time windows.

• Duration
• Duration(Anomaly)
• Read bytes
• Write bytes

Select the color icon to select or unselect the corresponding content in all graph.

When you select an instance of the notebook/Spark job definition activity run in the graph, the instance's Duration time distribution, Executors execution distribution, and Spark configuration are detailed at the bottom of the graph.

Focus mode lets you expand a visual to see more details. Maybe you have a visual that is a little crowded and you want to zoom in on it. This function is a perfect use of focus mode.

If the bar is marked red, an exception has been detected for that run instance. You can view the following information: Total duration, Expected duration, and Potential causes for this instance in the Anomalies panel.

Related content

• Run Series Analysis Overview
• Apache Spark monitoring overview
• Browse item's recent runs
• Monitor Apache Spark jobs within notebooks
• Monitor Apache Spark job definition
• Monitor Apache Spark application details