These features and Azure Databricks platform improvements were released in April 2023.
Note
Releases are staged. Your Azure Databricks account might not be updated until a week or more after the initial release date.
Databricks Marketplace (Public Preview): an open marketplace for data, analytics, and AI
April 27, 2023
You can now use Databricks Marketplace to discover, access, and deploy hundreds of data products into your AI and analytics workflows. Databricks Marketplace takes advantage of Delta Sharing to provide secure and seamless sharing of both free and commercial data products across any cloud or region. See What is Databricks Marketplace?.
Configure the Python formatter
April 25, 2023
For files and notebooks in Databricks Repos, you can now configure the Python formatter based on a Black specification. See Python black formatter library.
Workspace files are GA
April 24, 2023
You can now work with non-notebook files in Azure Databricks. Workspaces files are enabled by default in all workspaces. See What are workspace files?.
Cluster-scoped init scripts can now be stored in workspace files
April 24, 2023
You can now store cluster-scoped init scripts in workspace files, regardless of Databricks Runtime version used by your compute. Databricks recommends storing all cluster-scoped init scripts in workspace files. See Store init scripts in workspace files.
Create an Azure Key Vault-backed secret scope with a service principal
April 19, 2023
You can now create an Azure Key Vault-backed secret scope using a service principal. See Manage secret scopes
New Delta Sharing privileges enable delegation of share, recipient, and provider management tasks
April 18, 2023
The new privileges USE SHARE, USE RECIPIENT, USE PROVIDER, and SET SHARE PERMISSION enable delegation of share, recipient, and provider management tasks that would otherwise require the metastore admin role or object ownership. These privileges enable you to limit the number of users who have the powerful metastore admin role. See Privilege types that apply only to Delta Sharing or Databricks Marketplace.
Databricks Connect V2 (Public Preview)
April 17, 2023
Beginning with Databricks Runtime 13.0, Databricks Connect builds upon Spark Connect, enabling developers to use the power of Databricks and their IDE of choice to interactively develop and debug their code in any Azure Databricks workspace.
New cluster metrics UI
April 14, 2023
Clusters that use Databricks Runtime 13.0 and above have access to the new cluster metrics UI. This feature replaces Ganglia as the cluster metrics monitoring tool. The new cluster metrics UI includes a live view of cluster performance and custom minute-by-minute historical views. For more information, see View compute metrics.
Databricks Runtime 13.0 is GA
April 14, 2023
Databricks Runtime 13.0 and Databricks Runtime 13.0 ML are now generally available.
When Azure Databricks blocks a request, Azure Databricks generates a new event in the account-level audit logs with service accounts and event accountIpAclsValidationFailed. See Diagnostic log reference.
Databricks Terraform provider updated to version 1.14.3
April 14, 2023
Version 1.14.3 adds exporter support for databricks_sql_alert, adds support for notifications in databricks_pipeline, and more. For more details, see the changelog for version 1.14.3.
Diagnostic log entries for changed admin settings for workspace
April 14, 2023
There are now audit log events for changes to some admin settings in the workspace diagnostic logs. Within the accounts service, the new event action names are deleteSetting and setSetting. See Diagnostic log reference. Not all settings generate auditable events.
For initial customer-managed key for managed disks, Azure Databricks now terminates compute resources
April 10, 2023
Previously, initially adding a customer-managed key for managed disks restarts all running compute resources (clusters, pools). Compute resources are now terminated not restarted, which means you must manually start them after the update is complete. Azure Databricks prevents clusters from running or starting while adding a key. Plan downtime and terminate all compute resources before adding the key to prevent auto-termination, which can cause issues for some types of workloads.
For a workspace that already has a customer-managed key for managed disks, if you rotate the key, you don’t need to terminate compute resources.
Databricks Terraform provider updated to versions 1.14.1 and 1.14.2
April 4-6, 2023
Version 1.14.1 excludes roles in SCIM API list calls to reduce load on the Databricks SCIM service, and more. Version 1.14.2 adds a SQL file task to the databricks_job resource, fixes several bugs, and more. For more details, see the changelog for versions 1.14.1 and 1.14.2.
Combined SQL user settings and general user settings
April 3-5, 2023
The SQL user settings have been combined with the general workspace user settings to create a unified experience for workspace users. All SQL user settings are now accessed from Settings:
The Password tab is now the Account & Password tab.
The Query snippets tab is visible to users with the SQL entitlement.
Legacy notebook visualizations deprecated
April 4, 2023
Creation and editing of legacy notebook visualizations are deprecated and will be removed from the product soon. Existing legacy visualizations will continue to be rendered. Details will appear in notifications within the product.
Create or modify table from file upload page supports JSON file uploads
Manage data ingestion and preparation, model training and deployment, and machine learning solution monitoring with Python, Azure Machine Learning and MLflow.