Connect to Databricks using an SSH tunnel

Important

This feature is in Beta.

Cost and pricing for features in Beta are subject to change.

The Azure Databricks SSH (Secure Shell) tunnel connects Visual Studio Code, Cursor, or your terminal to the Databricks workspace and Azure Databricks serverless, serverless GPU, or classic compute. Use it to edit workspace files and run or debug Python scripts and notebooks, with your Python environment and code execution on the remote compute. Connect directly to serverless compute, or set up a reusable SSH connection to a dedicated classic cluster.

Requirements

Start the SSH connection from your local machine. The tunnel cannot be launched from another remote environment or a Docker container.

To connect to serverless or classic compute, you must have:

  • Databricks CLI version 1.5.0 or above installed on your local machine and authentication configured. Check your installed version with databricks --version. See Install or update the Databricks CLI.
  • Either of the following:
    • Visual Studio Code version 1.110.0 (Universal) or above and the Remote - SSH extension (1.0.46+) installed.
    • Cursor version 2.6.11 (Universal) or above.

To connect to serverless GPU compute, the AI Runtime feature must be enabled. See AI Runtime.

Additional requirements for classic compute

To connect to classic (dedicated, single-user) compute:

Choose an SSH connection

Use the connection method for your compute type:

Compute Connection method Setup requirement
Serverless compute databricks ssh connect No separate SSH setup step.
Add --ide vscode or --ide cursor to launch an IDE.
Serverless GPU compute databricks ssh connect --accelerator <accelerator> No separate SSH setup step.
Add --ide vscode or --ide cursor to launch an IDE.
For supported accelerators, see --accelerator.
Classic dedicated, single-user compute Run databricks ssh setup --name <connection-name> in your terminal.
Then start the SSH session from the IDE or run ssh <connection-name>.		 Select a dedicated, single-user cluster that meets the classic compute requirements.
Shared clusters are not supported.

The SSH tunnel and the Databricks IDE extension are not compatible and should not be used together. For a comparison with Databricks Connect and other development options, see Connect from your IDE.

Connect to serverless compute

Upon meeting the requirements above, run databricks ssh connect from your terminal or IDE.

The --ide value selects the editor to launch, regardless of which editor or terminal you run the command from.

Visual Studio Code

databricks ssh connect --ide vscode

The CLI opens a Visual Studio Code window connected to the remote compute.

Cursor

databricks ssh connect --ide cursor

The CLI opens a Cursor window connected to the remote compute.

Terminal

databricks ssh connect

The CLI starts an interactive SSH session.

For command options, including named connections and authentication profiles, see databricks ssh connect.

After connecting, open your workspace project and select the Python environment. If the connection fails, see Troubleshoot SSH tunnel connections.

Connect to serverless GPU compute with AI Runtime

With AI Runtime enabled, add --accelerator to use GPU compute:

databricks ssh connect --accelerator GPU_1xA10

databricks ssh connect gives you an interactive session on a single node. For long-running training jobs or multi-node distributed training, submit the workload with databricks air instead. See Use the Databricks CLI with AI Runtime.

Connect to classic compute

For a reusable connection to a classic dedicated cluster, set up an SSH host name, then select that host in your IDE or connect from a terminal.

Set up the SSH connection

Note

Setting up the SSH connection is only required if you are connecting to classic compute.

Run databricks ssh setup on your local machine. Replace <connection-name> with a name for this connection, such as my-connection:

databricks ssh setup --name <connection-name>

The CLI prompts you to select a cluster and adds a host entry to your SSH configuration. You can also select a cluster directly. Replace <cluster-id> with the ID of your dedicated cluster:

databricks ssh setup --name <connection-name> --cluster <cluster-id>

To configure how long the SSH server waits after the last client disconnects, add --shutdown-delay to the setup command. The default is 10 minutes. For example, to set a 30-minute delay for connections through this SSH host entry:

databricks ssh setup --name <connection-name> --cluster <cluster-id> --shutdown-delay 30m

Note

For IntelliJ users, Databricks recommends adding --auto-start-cluster=false to the setup command and starting the cluster manually before connecting. This is because JetBrains IDEs start all configured clusters on launch, which can result in unexpected compute charges.

Connect using Visual Studio Code or Cursor

  1. For Visual Studio Code, install the Remote SSH extension. Cursor includes a remote SSH extension by default.

  2. In the IDE main menu, click View > Command Palette. Select Remote-SSH: Settings. Alternatively, select Preferences: Open User Settings (JSON) to modify settings.json directly.

  3. Under Remote.SSH: Default Extensions (or remote.SSH.defaultExtensions in settings.json), add ms-Python.Python and ms-toolsai.jupyter.

    If you are modifying settings.json:

    "remote.SSH.defaultExtensions": [
    "ms-Python.Python",
    "ms-toolsai.jupyter"
]

    Note

    Optionally, increase the value of Remote.SSH: Connect Timeout (or remote.SSH.connectTimeout in settings.json) to further reduce the chance of timeout errors. The default timeout is 360.

  4. In the Command Palette, select Remote-SSH: Connect to Host.

  5. From the drop-down menu, select the connection you set up in the first step. The IDE proceeds to connect in a new window.

Connect using IntelliJ IDEs

  1. Follow the Remote server tutorial to get set up.
  2. On the new connection screen, enter:
    • Username: root
    • Host: <connection-name>

Connect to classic compute from a terminal

Use the connection name you configured with databricks ssh setup:

ssh <connection-name>

For a direct connection without a reusable SSH host entry, you can also use the CLI. Replace <cluster-id> with your dedicated cluster ID:

databricks ssh connect --cluster <cluster-id>

See databricks ssh connect.

Open workspace projects through SSH

By default, the databricks ssh connect command opens to an ephemeral directory. To access workspace files, navigate to your workspace directory from the IDE or terminal:

  • In Visual Studio Code or Cursor, from the Command Palette (Cmd/Ctrl+Shift+P) select Open Folder and navigate to /Workspace/Users/<your-username>.
  • From a terminal window, change your directory: cd /Workspace/Users/<your-username>.

Note

Files in /Workspace, /Volumes, and /dbfs persist across cluster restarts. Files in /home, /root, and other local paths are ephemeral and lost on restart.

Run Python and notebooks in the SSH session

Select the Python environment provided by your SSH session before running code. For classic compute, this environment includes the built-in Databricks Runtime libraries and compute-scoped libraries.

  1. Open the Command Palette (Cmd/Ctrl+Shift+P) and select Python: Select Interpreter.

  2. Select the pythonEnv-xxx virtual environment from the list. If you set up Python dependencies using the --base-environment flag, select the longer virtual environment name from the list of options. If the virtual environment does not appear:

    1. Run echo $DATABRICKS_VIRTUAL_ENV from a terminal within the IDE.

      Example output: /local_disk0/.ephemeral_nfs/envs/pythonEnv-xxx/bin/python

    2. Paste the full output as the interpreter path in the Python: Select Interpreter prompt.

  3. Open a new terminal and the virtual environment should automatically activate.

  4. To run a Jupyter notebook, make sure that the virtual environment is selected as the kernel. Click Select Kernel in the upper right of the notebook.

Run and debug Python files and .ipynb notebooks using the standard Python and Jupyter extensions.

To use Spark in a Python file on serverless compute, initialize a session explicitly:

from databricks.connect import DatabricksSession
spark = DatabricksSession.builder.serverless().profile("DEFAULT").getOrCreate()

Manage dependencies for SSH sessions

Choose dependency management for your compute type:

For non-Python packages, see Non-Python dependencies.

Workspace base environments (recommended for serverless and AI Runtime)

Note

This feature requires the Serverless workspace base environment support in Jobs preview to be enabled. See Manage Azure Databricks previews.

Use a workspace base environment with serverless environment version 4 or below to pre-configure Python dependencies. The SSH tunnel does not support --base-environment with serverless environment version 5 or above. Create a base environment using the workspace UI or the Databricks CLI databricks environments create-workspace-base-environment command.

Specify the environment using the --base-environment option when connecting:

databricks ssh connect --base-environment my-workspace-env

The --base-environment option accepts a display name, a workspace YAML path, or a resource ID. You cannot combine it with --cluster. See databricks ssh connect.

Cluster libraries (recommended for classic compute)

Install dependencies using the workspace UI under Compute > Libraries. These persist across cluster restarts and are available in pythonEnv-xxx. See Cluster libraries.

Non-Python dependencies

To persist non-Python dependencies, use an init script that installs the packages when the compute starts. Optionally, store the packages in a Unity Catalog volume and reference them from the init script. See What are init scripts?.

Project-specific notebook setup

For project-scoped dependencies, run a notebook containing %pip install commands at the start of each session:

# Install from pyproject.toml
%pip install .

# Install from a requirements file
%pip install -r requirements.txt

# Install a wheel from Volumes or Workspace
%pip install /Volumes/catalog/schema/volume/your_library.whl

%pip commands include Databricks-specific guardrails and propagate dependencies to Spark executor nodes. This enables user-defined functions (UDFs) with custom dependencies.

For more examples, see Manage libraries with %pip commands.

You do not need to rerun the setup notebook if you reconnect before the SSH server shuts down. By default, the server shuts down 10 minutes after the last client disconnects. To change this delay, use --shutdown-delay with the command for your connection method:

  • For a reusable classic compute connection, set the delay with databricks ssh setup. The generated SSH host entry applies the delay when you connect from your IDE or run ssh <connection-name>.
  • For a direct CLI connection, set the delay with databricks ssh connect.

Note

Multiple SSH sessions on the same cluster share one virtual environment.

Use Git in an SSH session

Note

This feature requires the Git CLI support for Git folders preview to be enabled. See Manage Azure Databricks previews.

You can use the Git CLI in the SSH tunnel with newly created Git folders and the Git credentials you've configured in the Databricks workspace. See Use Git CLI commands.

If the CLI prompts for credentials instead of picking them up automatically, you must connect your Git provider to Databricks. See Connect your Git provider to Databricks.

SSH tunnel limitations

The SSH tunnel provided by Databricks has the following limitations:

  • Shared clusters are not supported.
  • The Databricks IDE extension and the SSH tunnel are not yet compatible and should not be used together.
  • Files edited outside /Workspace, /Volumes, and /dbfs are lost on cluster restart.
  • A maximum of 10 SSH connections are allowed per cluster.
  • Inactive sessions might drop after 1 hour.
  • The SSH tunnel cannot be launched from other remote environments or Docker containers.
  • You might encounter performance or connection issues when three or more Jupyter notebooks are open simultaneously.

Notebook behavior in SSH sessions

There are some differences in notebooks when using the SSH tunnel:

  • Python files don’t define any Databricks globals (like spark or dbutils). You must import them explicitly with from databricks.sdk.runtime import spark.
  • For ipynb notebooks, these features are available:
    • Databricks globals: display, displayHTML, dbutils, table, sql, udf, getArgument, sc, sqlContext, spark
    • %sql magic command to execute SQL cells

To work with Python source “notebooks”:

  • Search for jupyter.interactiveWindow.cellMarker.codeRegex and set it to:

    ^# COMMAND ----------|^# Databricks notebook source|^(#\\s*%%|#\\s*\\<codecell\\>|#\\s*In\\[\\d*?\\]|#\\s*In\\[ \\])

  • Search for jupyter.interactiveWindow.cellMarker.default and set it to:

    # COMMAND ----------

Troubleshoot SSH tunnel connections

Start with the symptom that matches the failure:

Symptom First check
The SSH connection fails or times out. Check workspace HTTPS and WebSocket access. For classic compute, also check the cluster state and IDE timeout. See SSH connection fails or times out.
The CLI reports that it cannot find code. Install the Visual Studio Code shell command in your local PATH. See code command not found.
CLI authentication fails. Check the CLI profile and, for classic compute, cluster permissions. See CLI authentication errors.
The CLI reports invalid_grant with Refresh token is invalid. Sign in again to the workspace using the same CLI profile, then retry the connection. See Refresh token is invalid.
Python code or notebooks fail after connecting. Check the selected interpreter or notebook kernel. See Python code or notebooks fail in the SSH session.
Files disappear after a cluster restart. Check whether the files were stored on an ephemeral local path. See Files disappear or environment resets after cluster restart.

SSH connection fails or times out

  • For classic compute, verify that the cluster is running in the workspace UI.
  • Check that your network, VPN, and firewall allow HTTPS and WebSocket traffic to your workspace domain. The SSH tunnel does not use port 22. It tunnels all traffic over the same HTTPS endpoint as other Databricks CLI commands. See Configure domain name firewall rules.
  • If a proxy or firewall inspects TLS traffic, check that it allows the WebSocket upgrade request. An intermediary that blocks or strips the upgrade causes the connection to fail or time out. Ask your network administrator to allow WebSocket upgrades to your workspace domain, or to bypass TLS inspection for that domain.
  • If you connect through a corporate HTTP proxy, set the HTTPS_PROXY environment variable, or NO_PROXY to bypass the proxy for your workspace domain. The SSH tunnel uses the same proxy configuration as other Databricks CLI commands. See Proxy server configuration.
  • Check that your local machine has HTTPS access to github.com and release-assets.githubusercontent.com.
  • Increase the SSH timeout. See Connect using Visual Studio Code or Cursor.
  • For key mismatch errors, delete ~/.databricks/ssh-tunnel-keys and re-run databricks ssh setup.
  • For "remote host identification has changed" errors, check the ~/.ssh/known_hosts file and delete entries related to your cluster.
  • SSH sessions might drop after 1 hour and no more than 10 SSH connections can be made to a single cluster. See SSH tunnel limitations.

code command not found

If you see Error: exec: "code": executable file not found in $PATH, open the Command Palette (Cmd/Ctrl+Shift+P), select Shell Command: Install 'code' command in PATH, and restart your IDE or terminal session.

CLI authentication errors

  • Sign in to your workspace using databricks auth login.
  • For classic compute, confirm that you have CAN MANAGE permissions on the cluster.

Refresh token is invalid

If databricks ssh connect fails with error getting token, invalid_grant, and Refresh token is invalid, the authorization server rejected the CLI's saved OAuth refresh token. Sign in again to obtain a new session:

  1. Identify the configuration profile for the workspace you want to connect to:

    databricks auth profiles --skip-validate

  2. Sign in again using that profile. Replace <workspace-url> with the workspace URL, without the /oidc/v1/token path, and <profile-name> with the matching profile name:

    databricks auth login --host <workspace-url> --profile <profile-name>

    Complete the sign-in flow in your browser.

  3. Retry the connection with the same profile:

    databricks ssh connect --ide vscode --profile <profile-name>

    Use --ide cursor for Cursor, or omit --ide for a terminal session.

If the error persists, check that the failed command targets the same workspace and profile you just signed in to. See OAuth user-to-machine (U2M) authentication.

Python code or notebooks fail in the SSH session

Files disappear or environment resets after cluster restart

  • Files in /Workspace, /Volumes, and /dbfs mounts persist across cluster restarts. Files in /home, /root, and other local paths are ephemeral and lost on restart.
  • Use cluster library management for persistent dependencies. Automate reinstalls using init scripts if needed. See What are init scripts?.

SSH setup fails on Windows (WSL)

Run databricks ssh setup directly on Windows, not within WSL. The Windows Visual Studio Code instance cannot find SSH configurations created on the WSL side.

FAQ

Does serverless compute require SSH setup?

No. After installing and authenticating the Databricks CLI, use databricks ssh connect for serverless compute. The databricks ssh setup command configures reusable SSH host entries for classic dedicated compute.

Does the Databricks SSH tunnel require port 22?

No. The SSH tunnel carries traffic over HTTPS to your workspace domain and requires WebSocket support. Your network, VPN, firewall, and proxy must allow that traffic. See SSH connection fails or times out.

How is the SSH tunnel different from Databricks Connect?

Databricks Connect allows you to write code using Spark APIs and run them remotely on Databricks compute instead of in the local Spark session. The Databricks Visual Studio Code extension uses Databricks Connect to provide built-in debugging of user code on Databricks.

The SSH tunnel allows you to access the workspace from your IDE and moves your entire development environment onto the compute. Python, the kernel, and code execution run on Azure Databricks with access to the compute resources.

How is my code and data secured?

SSH traffic is encrypted end-to-end, and code runs on the Azure Databricks compute you connect to. Authentication uses the Databricks CLI and your configured workspace profile. See What authentication methods are supported?.

Which IDEs are supported?

Visual Studio Code and Cursor are officially supported. Any IDE with SSH capabilities is compatible, but only VS Code and Cursor are tested.

Are all Databricks notebook features available from the IDE?

Some features such as display(), dbutils, and %sql are available with limitations or manual setup. See Notebook behavior in SSH sessions.

Will my cluster start automatically when I connect using the SSH tunnel?

Yes, but if it takes longer to start the cluster than the connect timeout, the connection attempt will fail. To prevent this, increase the value of Remote.SSH: Connect Timeout from the command palette (or remote.SSH.connectTimeout in settings.json) to further reduce the chance of timeout errors.

How do I know if my cluster is running?

Navigate to Compute in the Databricks workspace UI, and check the status of the cluster. The cluster must show Running for SSH connection to work.

How do I disconnect my SSH/IDE session?

You can disconnect a session by closing your IDE window, using the Disconnect option in your IDE, closing your SSH terminal, or running the exit command in the terminal.

How do I stop the cluster and avoid charges when I'm not working?

To stop immediately, terminate the cluster from the workspace UI. Navigate to Compute in the Databricks workspace UI, find your cluster, and click Terminate or Stop.

Set a short auto-termination policy on your cluster from the workspace UI. After you disconnect, the SSH server waits for the shutdown-delay period (default: 10 minutes), then the cluster's idle timeout applies.

How should I handle persistent dependencies?

Dependencies installed during a session are lost after cluster restart. Use persistent storage (/Workspace/Users/<your-username>) for requirements and setup scripts. Use cluster libraries or init scripts for automation.

What authentication methods are supported?

Authentication uses the Databricks CLI and your ~/.databrickscfg profiles file. SSH keys are handled by the SSH tunnel.

Can I connect to external databases or services from the cluster?

Yes, as long as your cluster networking allows outbound connections and you have the necessary libraries.

Can I use additional IDE extensions?

Most extensions work when installed within your remote SSH session, depending on your IDE and cluster. Visual Studio Code by default doesn’t install local extensions on remote hosts. You can manually install them by opening the extensions panel and enabling your local extensions on the remote host. You can also configure Visual Studio Code to always install certain extensions remotely. See Connect to Databricks.

Yes, however workspace admins must allowlist URLs of Visual Studio Code and Cursor extension marketplaces. Your local machine must also have the ability to access the internet.

  • Last updated on