Create clusters and SQL warehouses that can access Unity Catalog
This article shows how to create a Azure Databricks cluster or SQL warehouse that can access data in Unity Catalog.
SQL warehouses are required in order to run Databricks SQL workloads, such as queries, dashboards, and visualizations. SQL warehouses allow you to access Unity Catalog data and run Unity Catalog-specific commands by default, as long as your workspace is attached to a Unity Catalog metastore.
Clusters are used to run workloads in the Data Science & Engineering and Databricks Machine Learning persona-based environments, using notebooks or automated jobs. To create a cluster that can access Unity Catalog, the workspace you are creating the cluster in must be attached to a Unity Catalog metastore and must use a Unity-Catalog-capable access mode (shared or single user).
You can work with data in Unity Catalog using either of these compute resources, depending on the environment you are using: SQL warehouses for Databricks SQL, or clusters for the Data Science & Engineering and Databricks Machine Learning environments.
For information about clusters configuration UI changes that are available in preview, see Create a cluster.
When you create any cluster in Azure Databricks, you must select an access mode that is specific to the type of workload that you want to use the cluster for. Unity Catalog enforces security using specific cluster access modes. If a cluster is not configured with one of the Unity-Catalog-capable access modes (shared or single user), the cluster can’t access data in Unity Catalog.
The following table lists all available access modes:
|Access Mode||Visible to user||UC Support||Supported Languages||Notes|
|Single User||Always||Yes||Python, SQL, Scala, R||Can be assigned to and used by a single user only. Dynamic views are not supported. Credential passthrough is not supported.|
|Shared||Always (Premium plan required)||Yes||Python, SQL||Init scripts, third-party libraries, and JARS are not supported. Credential passthrough is not supported.|
|No Isolation Shared||Hidden (Enforce User Isolation required)||No||Python, SQL, Scala, R||Admin console configuration required to enforce user isolation|
|Custom||Hidden (For all new clusters)||No||Python, SQL, Scala, R||This option is shown only if you have existing clusters without a specified access mode.|
You can upgrade an existing cluster to meet the requirements of Unity Catalog by setting its cluster access mode to Single User or Shared.
- Your Azure Databricks account must be on the Premium Plan.
- You must have permission to create a cluster. See Configure cluster creation entitlement.
A cluster is designed for running workloads such as notebooks and automated jobs.
To create a cluster that can access Unity Catalog, the workspace must be attached to a Unity Catalog metastore.
To create a cluster:
On the sidebar, use the persona switcher to select either Data Science and Engineering or Machine Learning.
On the sidebar, click Create > Cluster.
Choose the access mode you want to use.
For clusters running on standard Databricks Runtime versions, select either Single user or Shared access mode to connect to Unity Catalog. If you use Databricks Runtime for Machine Learning, you must select Single user access mode to connect to Unity Catalog. See What is cluster access mode?
Complete your cluster configuration and click Create Cluster.
When the cluster is available, it will be able to run workloads that use Unity Catalog.
A SQL warehouse is required to run workloads in Databricks SQL, such as queries, dashboards, and visualizations. By default all SQL Warehouses can connect to Unity Catalog. See Create a SQL warehouse for specific configuration options.
To access the GA version of Unity Catalog, you should use the preview channel for your SQL warehouses. Under Advanced Settings, set Channel to Preview.
Submit and view feedback for