Use Unity Catalog service credentials to connect to external cloud services
Article
This article describes how to use a service credential in Unity Catalog to connect to external cloud services. A service credential object in Unity Catalog encapsulates a long-term cloud credential that provides access to an external cloud service that users need to connect to from Azure Databricks.
Before you can use a service credential to connect to an external cloud service, you must have:
An Azure Databricks workspace that is enabled for Unity Catalog.
A compute resource that is on Databricks Runtime 16.2 or above.
SQL warehouses are not supported.
The Public Preview version of service credentials is available on Databricks Runtime 15.4 LTS and above, with Python support but no Scala support.
A service credential created in your Unity Catalog metastore that gives access to the cloud service.
The ACCESS privilege on the service credential or ownership of the service credential.
Use a service credential in your code
This section provides examples of using service credentials in a notebook. Replace placeholder values. These examples don’t necessarily show the installation of required libraries, which depend on the client service you want to access.
Python example: configure an Azure SDK client to use a specific service credential
Python
from azure.keyvault.secrets import SecretClient # example Azure SDK client
credential = dbutils.credentials.getServiceCredentialsProvider('your-service-credential')
vault_url = "https://your-keyvault-name.vault.azure.net/"
client = SecretClient(vault_url=vault_url, credential=credential)
Scala example: configure an Azure SDK client to use a specific service credential
Scala
import com.azure.security.keyvault.secrets.{SecretClient, SecretClientBuilder}
// Get credentials from dbutilsval credential = dbutils.credentials.getServiceCredentialsProvider("your-service-credential")
// URL of the Key Vaultval vaultUrl = "https://your-keyvault-name.vault.azure.net/"// Create the SecretClientval client: SecretClient = newSecretClientBuilder()
.vaultUrl(vaultUrl)
.credential(credential)
.buildClient()
Specify a default service credential for a compute resource
You can optionally specify a default service credential for an all-purpose or jobs compute cluster by setting an environment variable. By default, the SDK uses that service credential if no authentication is provided. Users still require ACCESS on that service credential to connect to the external cloud service. Databricks does not recommend this approach, because it makes your code less portable than naming the service credential in your code.
Note
Serverless compute and SQL warehouses don’t support environment variables, and therefore they don’t support default service credentials.
The following code samples do not specify a service credential. Instead, they use the service credential specified in the DATABRICKS_DEFAULT_SERVICE_CREDENTIAL_NAME environment variable:
Python
If you are using a default service credential, you don’t need to specify credentials as an argument:
Python
from azure.identity import DefaultAzureCredential
from azure.keyvault.secrets import SecretClient
vault_url = "https://your-keyvault-name.vault.azure.net/"
client = SecretClient(vault_url=vault_url, credential=credential)
Manage data ingestion and preparation, model training and deployment, and machine learning solution monitoring with Python, Azure Machine Learning and MLflow.
This article describes administrative tasks for Unity Catalog-governed service credentials, which are securable objects that let you govern access to external cloud services in Azure Databricks.