Configure high concurrency mode for Fabric notebooks

When you run a notebook in Microsoft Fabric, an Apache Spark session is started and is used to run the queries submitted as part of the notebook cell executions. With High Concurrency Mode enabled, there's no need to start new spark sessions every time to run a notebook.

If you already have a High Concurrency session running, you could attach notebooks to the high concurrency session getting a spark session instantly to run the queries and achieve a greater session utilization rate.

Note

The high concurrency mode-based session sharing is always within a single user boundary. The notebooks need to have matching spark configurations, should be part of the same workspace, share the same default lakehouse and libraries to share a single spark session.

Configure high concurrency mode

By default, all the Fabric workspaces are enabled with high concurrency Mode. Use the following steps to configure the high concurrency feature:

1. Click on Workspace Settings Option in your Fabric Workspace

   

2. Navigate to the Synapse section > Spark Compute > High Concurrency

3. In the High Concurrency section, you could choose to enable or disable the setting.

   

4. Enabling the high concurrency option allows users to start a high concurrency session in their notebooks or attach to existing high concurrency session.

5. Disabling the high concurrency mode hides the section to configure the time period of inactivity and also hides the option to start a new high concurrency session from the notebook menu.

   

Run notebooks in high concurrency session

1. Open the Fabric workspace

2. Create a Notebook or open an existing Notebook

3. Navigate to the Run tab in the menu ribbon and select on the session type dropdown that has Standard selected as the default option.

   

4. Select New high concurrency session.

5. Once the high concurrency session has started, you could now add upto 10 notebooks in the high concurrency session.

   

6. Create a new notebook and by navigating to the Run menu as mentioned in the above steps, in the drop down menu you will now see the newly created high concurrency session listed.

7. Selecting the existing high concurrency session attaches the second notebook to the session.

   

8. Once the notebook has been attached, you can start executing the notebook steps instantly.

9. The high concurrency session status also shows the number of notebooks attached to a given session at any point in time.

10. At any point in time if you feel the notebook attached to a high concurrency session requires more dedicated compute, you can choose to switch the notebook to a standard session by selecting the option to detach the notebook from the High Concurrency in the Run menu tab.

    

11. You can view the session status, type and session ID by navigating to status bar, select the Session ID allows you to explore the jobs executed in this high concurrency session and view logs of the spark session in the monitoring detail page.

    

Monitoring and debugging notebooks running in high concurrency session

Monitoring and debugging are often a non-trivial task when you are running multiple notebooks in a shared session. For high concurrency mode in Fabric, separation of logs is offered which would allow users to trace the logs emitted by spark events from different notebooks.

1. When the session is in progress or in completed state, you can view the session status by navigating to the Run menu and selecting the All Runs option

2. This would open up the run history of the notebook showing the list of current active and historic spark sessions

   

3. Users by selecting a session, can access the monitoring detail view, which shows the list of all the spark jobs that have been run in the session.

4. In the case of high concurrency session, users could identify the jobs and its associated logs from different notebooks using the Related notebook tab, which shows the notebook from which that job has been run.

   

Related content

In this document, you get a basic understanding of a session sharing through high concurrency mode in notebooks. Advance to the next articles to learn how to create and get started with your own Data Engineering experiences using Lakehouse and Notebooks:

• To get started with Lakehouse, see Creating a Lakehouse.
• To get started with Notebooks, see How to use a Notebook