Troubleshoot UNITY_CATALOG_INITIALIZATION_FAILED

This page describes how to diagnose and resolve the UNITY_CATALOG_INITIALIZATION_FAILED error in Azure Databricks pipelines.

Overview

UNITY_CATALOG_INITIALIZATION_FAILED is a catch-all error that appears when Unity Catalog cannot initialize storage for a pipeline or workload during cluster startup. Despite its name, the failure is usually not caused by a Unity Catalog misconfiguration. Instead, it almost always indicates an underlying infrastructure problem: a networking issue preventing the cluster from reaching the Azure Databricks control plane, or a permissions issue preventing access to the cloud storage backing the Unity Catalog metastore.

This error surfaces across multiple Azure Databricks products, including Lakeflow Connect pipelines, Lakeflow pipelines, and AI Search.

General error message

Encountered an error with Unity Catalog while setting up the pipeline on cluster [CLUSTER_ID].
Ensure that your Unity Catalog configuration is correct, and that required resources
(e.g., catalog, schema) exist and are accessible. Also verify that the cluster has
appropriate permissions to access Unity Catalog.

The error message instructs you to check your Unity Catalog configuration, but the root cause is often a networking or cloud permissions issue as described in the following sections.

Root causes and resolutions

Managed identity missing RBAC role on ADLS storage

Cause: The managed identity associated with your Unity Catalog storage credential lacks the Storage Blob Data Contributor role on the Azure Data Lake Storage Gen2 (ADLS Gen2) account backing the Unity Catalog metastore. Unity Catalog initialization attempts to access the internal __unitystorage path on this storage account, and a 403 AuthorizationFailure response surfaces as UNITY_CATALOG_INITIALIZATION_FAILED.

Resolution:

1. In the Azure portal, navigate to the storage account backing your Unity Catalog metastore.
2. Go to Access control (IAM) > Role assignments and confirm that the managed identity used by your Unity Catalog storage credential has the Storage Blob Data Contributor role assigned at the storage account or container level.
3. If the role is missing, click Add > Add role assignment, select Storage Blob Data Contributor, and assign it to the managed identity.
4. Restart the pipeline after making any changes.

For more information, see Step 2: Grant the managed identity access to the storage account and Connect to an Azure Data Lake Storage Gen2 (ADLS Gen2) external location.

Unity Catalog resources not configured correctly

Cause: The catalog, schema, or connection referenced by the pipeline does not exist or is not accessible from the workspace. This is the case described by the error message itself and is less common than the infrastructure issues above.

Resolution:

1. Verify that the catalog and schema referenced in the pipeline exist and are accessible from the workspace. In Azure Databricks, go to Catalog and confirm that the catalog is visible and that you have at least USE CATALOG and USE SCHEMA privileges.
2. If you are using Lakeflow Connect, verify that the connection used by the pipeline is valid. Go to Catalog > External Data > Connections and confirm that the connection is listed and accessible.
3. Verify that the cluster or pipeline has the necessary Unity Catalog privileges. See Manage privileges in Unity Catalog.