Events
Mar 31, 11 PM - Apr 2, 11 PM
The biggest Fabric, Power BI, and SQL learning event. March 31 – April 2. Use code FABINSIDER to save $400.
Register todayThis browser is no longer supported.
Upgrade to Microsoft Edge to take advantage of the latest features, security updates, and technical support.
With Microsoft Fabric, you can use Apache Spark to run notebooks, jobs, and other kinds of applications in your workspace. This article explains how to monitor your Apache Spark application, allowing you to keep an eye on the recent run status, issues, and progress of your jobs.
You can view all Apache Spark applications from Spark job definition, or notebook item context menu shows the recent run option -> Recent runs.
You can select the name of the application you want to view in the application list, in the application details page you can view the application details.
Open the Recent runs page of the notebook or Spark job definition, you can view the status of the Apache application.
Open an Apache Spark application job from the Spark job definition or notebook item context menu shows the Recent run option -> Recent runs -> select a job in recent runs page.
In the Apache Spark application monitoring details page, the job runs list is displayed in the Jobs tab, you can view the details of each job here, including Job ID, Description, Status, Stages, Tasks, Duration, Processed, Data read, Data written and Code snippet.
The executor usage graph under the Resources tab visualizes the allocation and utilization of Spark executors for the current Spark application in near real-time during Spark execution. You can refer to: Monitor Apache Spark Applications Resource Utilization.
In the Apache Spark application monitoring page, click the Properties button to open/collapse the summary panel. You can view the details for this application in Details.
For the Logs tab, you can view the full log of Livy, Prelaunch, Driver log with different options selected in the left panel. And you can directly retrieve the required log information by searching keywords and view the logs by filtering the log status. Click Download Log to download the log information to the local.
Sometimes no logs are available, such as the status of the job is queued and cluster creation failed.
Live logs are only available when app submission fails, and driver logs are also provided.
For the Data tab, you can copy the data list on clipboard, download the data list and single data, and check the properties for each data.
The Item snapshots tab allows you to browse and view items associated with the Apache Spark application, including Notebooks, Spark job definition, and/or Pipelines. The item snapshots page displays the snapshot of the code and parameter values at the time of execution for Notebooks. It also shows the snapshot of all settings and parameters at the time of submission for Spark job definitions. If the Apache Spark application is associated with a pipeline, the related item page also presents the corresponding pipeline and the Spark activity.
In the Item snapshots screen, you can:
Note
The Notebook snapshots feature currently doesn't support notebooks that are in a running state or in a high-concurrency Spark session.
The diagnostic panel provides users with real-time recommendations and error analysis, which are generated by the Spark Advisor through an analysis of the user's code. With built-in patterns, the Apache Spark Advisor helps users avoid common mistakes and analyzes failures to identify their root cause.
The next step after viewing the details of an Apache Spark application is to view Spark job progress below the Notebook cell. You can refer to:
Events
Mar 31, 11 PM - Apr 2, 11 PM
The biggest Fabric, Power BI, and SQL learning event. March 31 – April 2. Use code FABINSIDER to save $400.
Register todayTraining
Module
Use Apache Spark in Microsoft Fabric - Training
Apache Spark is a core technology for large-scale data analytics. Microsoft Fabric provides support for Spark clusters, enabling you to analyze and process data at scale.
Certification
Microsoft Certified: Fabric Data Engineer Associate - Certifications
As a Fabric Data Engineer, you should have subject matter expertise with data loading patterns, data architectures, and orchestration processes.