July 2023
These features and Azure Databricks platform improvements were released in July 2023.
Note
Releases are staged. Your Azure Databricks workspace might not be updated until a week or more after the initial release date.
Workspace admins can now create account groups
July 31, 2023
Workspace admins can now create account groups from their identity federated workspaces. Previously workspace admins could create only workspace-local groups. Account groups can be granted access to data in a Unity Catalog metastore and access to identity-federated workspaces. Databricks recommends that you use account groups instead of workspace-local groups to take advantage of Unity Catalog and a central place to administer identity.
See Manage account groups using the workspace admin settings page.
Group manager role is in Public Preview
July 31, 2023
You can now grant Databricks users, service principals, and groups permissions to manage groups. Group managers can manage group membership. They can also assign other users the group manager role. Account admins have the group manager role on all groups in the account. Workspace admins have the group manager role on account groups that they create.
See Who can manage account groups?.
Databricks CLI updated to version 0.202.0 (Public Preview)
July 27, 2023
The Databricks command-line interface (Databricks CLI) has been updated to version 0.202.0. For details, see the changelog for version 0.202.0.
Databricks SDK for Python updated to version 0.3.0 (Beta)
July 27, 2023
Databricks SDK for Python version 0.3.0 introduces support for the Account Network Policy service; handles nested query parameters; adds, removes, and creates various methods, fields, dataclasses, and services; and more. For details, see the changelog for version 0.3.0.
Databricks SDK for Go updated to version 0.14.1 (Beta)
July 27, 2023
Databricks SDK for Go version 0.14.1 handles nested query parameters appropriately, fixing filtering in QueryHistory listings. For details, see the changelog for version 0.14.1.
Databricks SDK for Go updated to version 0.14.0 (Beta)
July 26, 2023
Databricks SDK for Go version 0.14.0 adds support for the Account Network Policy service, and adds, removes, and changes several methods, fields, and types. For details, see the changelog for version 0.14.0.
Run another job as a task in an Azure Databricks job
July 25, 2023
You can use the new Run Job task to add another job as a task in your Azure Databricks job, allowing you to decompose a large processing workflow into multiple component jobs, or create reusable components to use in multiple jobs. For information about using the task in the Jobs UI, see Schedule and orchestrate workflows. For information about using the task with the Jobs REST API, see Jobs in the REST API 2.1 reference or the Jobs API 2.0 reference.
Workspace network update gets UI support (GA)
July 24, 2023
Previously you could use an ARM template with command line or the Azure portal to enable secure cluster connectivity or enable Azure Private Link.
You can now also use the Azure portal UI (with no template) for network updates directly on your Azure Databricks workspace instance. Both types of network updates are now generally available.
All users can access data products in Databricks Marketplace by default
July 24, 2023
Starting on July 24 and rolling out gradually to all eligible workspaces, all users will be able to install free or previously purchased datasets using Databricks Marketplace. This is made possible by granting the USE MARKETPLACE ASSETS privilege to all users in all Unity Catalog metastores. This new privilege has no cost impact, since all Marketplace transactions take place outside of Databricks. If you’d like to disable access, see Disable Marketplace access.
Classic keyboard shortcuts mode
July 21, 2023
A new setting allows you to use features from the new editor (such as autocomplete as you type, syntax highlighting, code formatting, and more) while retaining the same familiar keyboard shortcuts from the previous editor. To access this setting, open the editor configuration and toggle Notebook editor shortcuts mode to Classic (Code Mirror).
Lakehouse Federation lets you run queries against external database providers (Public Preview)
July 21, 2023
Lakehouse Federation is the query federation platform for Azure Databricks, taking advantage of Unity Catalog to enable users and systems to run queries against multiple external data sources without needing to migrate all data to a unified system.
Lakehouse Federation is intended for use cases like:
- Ad hoc reporting.
- Proof-of-concept work.
- The exploratory phase of new ETL pipelines or reports.
- Supporting workloads during incremental migration.
In each of these scenarios, query federation gets you to insights faster, because you can query the data in place and avoid complex and time-consuming ETL processing. Plus you get the advantages of Unity Catalog interfaces and data governance, including fine-grained access control, data lineage, and search.
See What is Lakehouse Federation?.
Move to trash enabled for Repos
July 19, 2023
Deleting repos now works the same way as deleting other workspace assets. In the Repos list, right-click the repo name and select Move to Trash. After 30 days, the Trash folder is automatically deleted permanently.
See Delete an object.
Create alerts for slow-running or stuck jobs
July 18, 2023
You can now configure an expected duration for an Azure Databricks job or job task and add notifications if a job or task exceeds the expected duration. This feature lets you get alerts for slow-running jobs without the run being canceled. To configure an expected duration in the Azure Databricks Jobs UI, see Configure an expected completion time or a timeout for a job. To configure the expected duration with the Jobs 2.1 API, see Jobs in the REST API reference.
Databricks SDK for Go updated to version 0.13.0 (Beta)
July 18, 2023
Databricks SDK for Go version 0.13.0 adds support for Azure managed identities authentication for account-level operations, adds, changes, and removes several methods and fields, and more. For details, see the changelog for version 0.13.0.
Databricks SDK for Python updated to version 0.2.0 (Beta)
July 18, 2023
The Databricks SDK for Python version 0.2.0 adds a local implementation of dbutils.widgets, adds, changes, and removes several methods, fields, and dataclasses, and more. For details, see the changelog for version 0.2.0.
Databricks CLI updated to version 0.201.0 (Public Preview)
July 18, 2023
The Databricks command-line interface (Databricks CLI) version 0.201.0 improves the auth login experience, supports tab completion for referencing Azure Databricks authentication configuration profiles, makes additions, changes, and removals of several command groups and commands, and more. For details, see the changelog for version 0.201.0.
Databricks SDK for Python updated to version 0.2.1 (Beta)
July 18, 2023
Databricks SDK for Python version 0.2.1 supports older versions of urllib. For details, see the changelog for version 0.2.1.
Databricks Assistant is in Public Preview
July 18, 2023
The Databricks Assistant works as an AI-based companion pair-programmer to make you more efficient as you create notebooks, queries, and files. It can help you rapidly answer questions by generating, optimizing, completing, explaining, and fixing code and queries. See the Databricks Assistant FAQ for more information and for instructions on how to enable the Assistant.
Deactivate users and service principals from your account
July 13, 2023
You can now deactivate users and service principals from your Azure Databricks account. A deactivated user cannot log in to the Azure Databricks account or identity federated workspaces. However, all of the user’s permissions and workspace objects remain unchanged. For more information, see Deactivate a user in your Azure Databricks account and Deactivate a service principal in your Azure Databricks account.
Account-level SCIM provisioning now deactivates users when they are deactivated in the identity provider
July 13, 2023
Account-level SCIM provisioning now deactivates users when they are deactivated in your identity provider. Previously, when a user was deactivated in an identity provider, account-level SCIM provisioning deleted them from the Azure Databricks account. For more information, see Sync users and groups from Microsoft Entra ID and Deactivate a user in your Azure Databricks account.
Trash directory admin access
July 13, 2023
Workspace admins can now access other users’ Trash directories.
Prevention of MIME type sniffing and XSS attack page rendering are now always enabled
July 12, 2023
Azure Databricks reduces the risk of MIME type sniffing and XSS attack page rendering by adding appropriate HTTP headers. These features were enabled by default and previously could be disabled. For improved security, Azure Databricks now always enables both features and there are no longer workspace admin settings to disable them. For workspaces that previously disabled one or both of these features, both features are now enabled.
Unity Catalog volumes are in Public Preview
July 12, 2023
You can now use volumes in Unity Catalog to manage access to cloud object storage locations on Databricks Runtime 13.2 and above. Volumes provide capabilities for accessing, storing, governing, and organizing files. See What are Unity Catalog volumes?.
Simplified experience for submitting product feedback from the workspace
July 11, 2023
It’s now easier to send product feedback from your workspace. You can submit feedback and attach screenshots with fewer steps and without leaving the workspace UI. For more information, see Submit product feedback.
Databricks extension for Visual Studio Code updated to version 1.1.0
July 10, 2023
The Databricks extension for Visual Studio Code version 1.1.0 enables Databricks Connect integration by default, adds experimental features for Jupyter notebooks, and more. For details, see the changelog for version 1.1.0.
Functions now displayed in Catalog Explorer (Public Preview)
July 10, 2023
SQL and Python user-defined functions are now visible in Catalog Explorer. You can view function details and manage permissions using this new UI.
Databricks Terraform provider updated to version 1.21.0
July 7, 2023
Databricks Terraform provider version 1.21.0 adds support for subscriptions in dashboards and alert SQL tasks in databricks_job, defines generic Azure Databricks data utilities for defining workspace and account-level data sources, and more. For details, see the changelog for version 1.21.0.
The maximum offset for the List all jobs and List job runs API requests is now limited
July 6, 2023
The maximum offset you can specify in the List all jobs and List job runs requests in the Jobs API is now limited. To avoid this limit when you use these requests, use token-based pagination which does not have this limit. See GET /api/2.1/jobs/list and GET /api/2.1/jobs/runs/list in the REST API reference.
Databricks Runtime 13.2 is GA
July 6, 2023
Databricks Runtime 13.2 and Databricks Runtime 13.2 ML are now generally available.
See Databricks Runtime 13.2 (EoS) and Databricks Runtime 13.2 for Machine Learning (EoS).
Delta Sharing and Databricks Marketplace support view sharing (Public Preview)
July 6, 2023
You can now share views in Delta Sharing and Databricks Marketplace. See Add views to a share.
Init scripts on DBFS reach end of life on Sept 1, 2023
July 5, 2023
On Sept 1, 2023, support for init scripts on DBFS will reach end of life and the feature will no longer function. Store init scripts in workspace files to ensure that they continue to function after Sept 1, 2023.