February 2024

  • Article

These features and Azure Databricks platform improvements were released in February 2024.

Note

Releases are staged. Your Azure Databricks account might not be updated until a week or more after the initial release date.

Use Delta Live Tables in Feature Engineering (Public Preview)

February 29, 2024

You can now use any Delta Live Table in Unity Catalog with a primary key as a feature table for model training or inference. Unity Catalog provides feature discovery, governance, lineage, and cross-workspace access. For more information, see Create a feature table in Unity Catalog.

February 29, 2024

The source IPv4 address for front-end PrivateLink connections is now captured in source_ip_address in audit logs. See Diagnostic log reference and Configure private connectivity to Azure Databricks.

Restrict changing a job owner and the run as setting

February 29, 2024

Workspace admins can change a job owner to any user or service principal in their workspace by default and the job run as setting to any user in their workspace or service principals that they have the Service Principal User role on.

Account admins can now restrict workspace admins to only change a job owner to themselves and the job run as setting to service principals that they have the Service Principal User role on or to themselves. You can automate enabling this setting after workspace creation. See Restrict workspace admins.

Support for Databricks managed service principals

February 28, 2024

You can now create and manage service principals directly in Azure Databricks. Previously, all service principals had to be linked to an existing Microsoft Entra ID service principal. See Databricks and Microsoft Entra ID (formerly Azure Active Directory) service principals.

Enable automatic cluster update (Public Preview)

February 27, 2024

Automatic cluster update ensures that all clusters in a workspace are periodically updated to the latest host OS image and security updates. The maintenance window is configurable. See Automatic cluster update. Enabling this feature on a workspace automatically adds the Enhanced Security and Compliance add-on. See the pricing page.

Admins can enable enhanced security and compliance features (Public Preview)

February 27, 2024

Admins can now enable features of the Configure enhanced security and compliance settings: the compliance security profile, enhanced security monitoring, and automatic cluster update. Use the Azure portal or an ARM template. Use of these features automatically adds the Enhanced Security and Compliance add-on. See the pricing page.

The automatic cluster update feature and the ability to enable Enhanced Security and Compliance features on your own are in Public Preview. However, the compliance security profile itself and enhanced security monitoring are generally available (GA).

Support for Cloudflare R2 storage to avoid cross-region egress fees (Public Preview)

February 26, 2024

You can now use Cloudflare R2 as cloud storage for data registered in Unity Catalog. Cloudflare R2 is intended primarily for Delta Sharing use cases in which you want to avoid the data egress fees charged by cloud providers when data crosses regions. R2 storage supports all of the Databricks data and AI assets supported in AWS S3, Azure Data Lake Storage Gen2, and Google Cloud Storage. Support for R2 requires Databricks Runtime 14.3 or SQL Warehouse 2024.15. See Use Cloudflare R2 replicas or migrate storage to R2 and Create a storage credential for connecting to Cloudflare R2.

Add data UI supports XML file format

February 23, 2024

The add data UI now supports XML file upload and ingestion from cloud object storage. See Load data using the add data UI.

Use AI Functions to invoke a generative AI model from Foundation Model APIs

February 22, 2024

You can now use AI Functions to invoke a state-of-the-art generative AI model from Databricks Foundation Model APIs to perform AI tasks like sentiment analysis and classification.

See AI functions using Databricks Foundation Model APIs for available functions.

Unity Catalog volumes are GA

February 22, 2024

Unity Catalog volumes are now generally available in Databricks Runtime 14.3 LTS and above. Databricks recommends using volumes for managing read and write access to non-tabular data in cloud object storage. See Create and work with volumes.

Full-page AI-powered search

February 16-22, 2024

Azure Databricks search now leverages DatabricksIQ to provide a more intelligent AI-powered search experience. The new full-page search experience gives you more space to see results, more metadata for your objects, and more filters to narrow down your results.

See Search for workspace objects.

Run SQL notebook jobs on a SQL warehouse

February 8, 2024

You can now schedule and execute notebook jobs using a SQL warehouse as the compute resource. For details, see Use a notebook with a SQL warehouse.

File arrival triggers in Azure Databricks Workflows is GA

February 7, 2024

File arrival triggers is now generally available in all cloud providers. With this release, you can use file arrival triggers to run a Azure Databricks job when new files arrive in a Unity Catalog volume in addition to the existing support for Unity Catalog external locations. See Trigger jobs when new files arrive.

This release also removes a limitation with using file arrival triggers with an Azure firewall.

February 7, 2024

You can now search for machine learning models in Unity Catalog using the global search bar in the Databricks workspace UI. See Manage model lifecycle in Unity Catalog and Search for workspace objects.

Databricks Git server proxy is GA

February 6, 2024

You can use Databricks Git server proxy to configure secure access from Databricks Repos to your on-premises Git server. See Set up private Git connectivity for Databricks Git folders (Repos).

Databricks Git server proxy no longer requires CAN_ATTACH_TO permissions

February 6, 2024

The GA release of Databricks Git server proxy has removed the need to provide all users with CAN_ATTACH_TO permissions.

If you previously configured Databricks Git server proxy with CAN_ATTACH_TO privileges, see Remove global CAN_ATTACH_TO permissions.

Workspace file support for the dbt and SQL file tasks is GA

February 1, 2024

Support for using dbt projects and SQL files stored in Azure Databricks workspace folders is now generally available, adding to the existing support for using files stored in a remote Git repository. See Create and run Azure Databricks Jobs and Use dbt transformations in an Azure Databricks job.

Databricks Connect is GA for Scala

February 1, 2024

Beginning with Databricks Runtime 14.3 LTS, Databricks Connect is now generally available for Scala.

Create tables from files in volumes

February 1, 2024

You can now use the UI to create Unity Catalog managed tables from data files stored in Unity Catalog volumes. See Create table from volumes.

Databricks Runtime 14.3 LTS is GA

February 1, 2024

Databricks Runtime 14.3 LTS and Databricks Runtime 14.3 LTS ML are now generally available.

See Databricks Runtime 14.3 LTS and Databricks Runtime 14.3 LTS for Machine Learning.

Delta Sharing supports tables that use deletion vectors (Public Preview)

February 1, 2024

Delta Sharing now supports the sharing of tables that use deletion vectors, which are a storage optimization feature that you can enable on Delta tables. Recipients can read tables that use deletion vectors using a SQL warehouse, a cluster running Databricks Runtime 14.1 or above, or compute that is running open source delta-sharing-spark 3.1 or above. Support is rolling out gradually over the coming month.

See Add tables with deletion vectors or column mapping to a share, Read tables with deletion vectors or column mapping enabled, and Read tables with deletion vectors or column mapping enabled.