Use compute

  • Article

This article explains how to connect to compute in your Azure Databricks workspace. You need access to compute to run data engineering, data science, and data analytics workloads.

Who can access or create compute?

The ability to access or create compute depends on a user’s entitlements.

  • If your workspace is enabled for serverless compute for notebooks (Public Preview), all users in the workspace have access to the serverless compute resource to run interactive workloads in notebooks and workflows.

  • Workspace admins can create any type of compute. They also inherit the CAN MANAGE permission on all compute created in their workspace.

  • Non-admin users with the Unrestricted cluster creation entitlement have access to all configuration settings when creating compute. They can access compute they’ve been given permissions to and can create any type of new compute. To learn about available configuration settings, see Compute configuration reference. Workspace admins can assign this entitlement to any user, group, or service principal. See Manage entitlements.

  • Non-admin users without the Unrestricted cluster creation entitlement can only access compute they are granted permissions to or compute they create using policies they are assigned permission to.

Use serverless compute (Public Preview)

Important

This feature is in Public Preview.

If your workspace has been enabled for the serverless compute Public Preview, you will automatically have access to the serverless compute resource in any of your available notebooks. Serverless compute gives you on-demand access to scalable compute in notebooks, letting you immediately write and run your Python or SQL code.

To attach to the serverless compute, click the Connect drop-down menu in the notebook and select Serverless. In new notebooks, the attached compute automatically defaults to serverless upon code execution if no other resource has been selected.

For details on enablement, see Enable serverless compute public preview.

Use compute configured by another user

If you don’t have unrestricted cluster creation permissions, you only have access to the compute and compute policies granted to you by your workspace admins. Users can have any of these permissions on a compute:

  • CAN ATTACH TO: Allows you to attach your notebook to compute and view the compute metrics and Spark UI.
  • CAN RESTART: Allows you to start, restart, and terminate compute. Also includes CAN ATTACH TO permissions.
  • CAN MANAGE: Allows you to edit compute details, permissions, and size. Also includes CAN ATTACH TO and CAN RESTART permissions.
  • NO PERMISSIONS: No permissions on the compute.

If you have permissions to attach to a compute, you can select it from the Connect drop-down menu in an opened notebook or from the Compute drop-down menu when creating a new job. For more information on compute permissions, see Compute permissions.

Create new compute using a policy

If you have permission to a compute policy, you can create your own compute. Policies have minimal configuration options and are designed to be efficient resources using their default settings. If you do wish to edit any settings, you can learn about each setting in the configuration settings reference.

  1. Click New > Cluster in your workspace sidebar.
  2. Select a policy from the Policy drop-down menu.
  3. (Optional) Update the name of the compute.
  4. (Optional) Configure any available settings.
  5. Click Create compute.

You now have a compute resource you can use to run your workloads.

Policies

Workspace admins can create and manage the compute policies in your workspace. If you don’t have access to a policy that allows you to create the compute you need, reach out to your workspace admin. For more on policies, see Create and manage compute policies.

Your workspace might have custom policies or use the Azure Databricks default policies. The default policies include:

  • Personal Compute: Allows users to create an individually assigned single-node compute resource with minimal configuration options.
  • Shared Compute: Allows users to create larger multi-node resource intended for multiple users to share.
  • Power User Compute: Allows users to create larger multi-node resources. The policy is intended for single-user workloads that require more compute resources than Personal Compute allows.
  • Job Compute: Allows users to create a general-purpose default compute for jobs.

By default, all users have access to the Personal Compute policy. If you don’t see the Personal Compute policy, your organization has removed it from your workspace.

Unrestricted compute creation

If you are a workspace admin or a user with the Unrestricted cluster creation entitlement, you can create compute using the Unrestricted policy. This gives you access to all compute settings in the New compute UI. For a reference of all available settings, see Compute configuration reference.