Run shell commands in Azure Databricks web terminal
The Azure Databricks web terminal provides a convenient and highly interactive way to run shell commands in a command-line interface (CLI), including Databricks CLI commands, to take actions on Databricks objects programmatically. It’s especially useful for advanced use cases, such as batch operations on multiple files, which existing user interfaces (UIs) might not fully support.
Multiple users can use the web terminal on one compute.
You can use the web terminal to do the following:
- Make quick file edits using Vim or Emacs.
- Monitor resource usage with commands like
htop
(cluster usage) ornvidia-smi
(GPU usage). - Run non-Spark Python scripts.
- Execute file operations with shell commands such as
mv
andmkdir
. - Install and manage libraries on compute.
- Use the Databricks CLI to automate various aspects of Azure Databricks.
Requirements
Warning
Azure Databricks proxies the web terminal service from port 7681 on the compute’s Spark driver. This web proxy is intended for use only with the web terminal. If the port is occupied when the compute starts or there is some sort of conflict, the web terminal might not work as expected. If other web services are launched on port 7681, compute users might be exposed to potential security exploits. Neither Databricks nor Microsoft is is responsible for any issues that result from the installation of unsupported software on a compute.
- Web terminal is disabled by default for all workspace users. To enable it, see Enable the web terminal.
- CAN ATTACH TO permission on a compute.
- The compute must use an access mode of Single user or No isolation shared.
Launch the web terminal
You can launch the web terminal from the compute details page or from a notebook.
From a notebook
To launch the web terminal from a notebook:
- Connect the notebook to compute.
- At the bottom of the notebook’s right sidebar, click the Open bottom panel icon
.
- Alternatively, click the attached compute drop-down, hover over the attached compute, then click Web Terminal.
The web terminal opens in a panel at the bottom of the screen. The buttons at the upper-right of the panel allow you to:
- Open a new terminal session in a new tab
.
- Reload a terminal session
.
- Close the bottom panel
. To reopen the panel, click
at the bottom of the right sidebar.
From the compute details page
To launch the web terminal from the compute details page:
- On the workspace’s sidebar, click Compute.
- On the All-purpose compute tab, click the name of the compute.
- Click Start to start the compute.
- On the Apps tab, click Web Terminal.
A new tab opens with the web terminal UI and the Bash prompt.
Use web terminal
In the web terminal, you can run commands from root inside the container of the compute driver node.
Each user can have up to 100 active web terminal sessions (tabs) open. Idle web terminal sessions may time out and the web terminal web application will reconnect, resulting in a new shell process. If you want to keep your Bash session, Databricks recommends using tmux.
Run Databricks CLI commands
You can also use the web terminal to run Databricks CLI commands. The available CLI is always the latest version, and authentication is based on the current user. The compute must meet the following requirements:
- The cluster must have Databricks Runtime 15.0 or above installed.
- The workspace must not be enabled for Private Link.
Launch the web terminal and run the following command to output information about the current user:
databricks current-user me
Bundle commands are also available, which allows you to create and manage your Databricks Asset Bundles directly from the web terminal within the Databricks workspace. For example, to create, deploy, and run a simple bundle using the default template:
From the web terminal root, navigate to your workspace home and run
bundle init
:cd /Workspace/Users/someone@example.com databricks bundle init
Accept the default template prompts, then change to the bundle directory and deploy it:
cd my_project databricks bundle deploy
You can view the deployed
my_project
bundle in the Databricks workspace UI.Finally, run the default job in the bundle:
databricks bundle run my_project_job
Navigate to Job Runs to see the running job.
Limitations
Azure Databricks does not support running Spark jobs from the web terminal.
Azure Databricks web terminal is not available in the following compute types:
- Job compute
- Serverless compute
- Compute launched with the
DISABLE_WEB_TERMINAL=true
environment variable set. - Compute launched with access mode set to Shared.
- Compute launched with the Spark configuration
spark.databricks.pyspark.enableProcessIsolation
set totrue
.
There is a hard limit of 12 hours since the initial page load, after which any connection, even if active, will be terminated. You can refresh the web terminal to reconnect. Databricks recommends using tmux to preserve your shell session.
ARM compute resources cannot use web terminals to access workspace files, including files in Git folders.
Enabling Docker Container Services disables web terminal.