April 2024
These features and Azure Databricks platform improvements were released in April 2024.
Note
Releases are staged. Your Azure Databricks account might not be updated until a week or more after the initial release date.
Databricks Runtime 15.1 is GA
April 30, 2024
Databricks Runtime 15.1 and Databricks Runtime 15.1 ML are now generally available.
See Databricks Runtime 15.1 and Databricks Runtime 15.1 for Machine Learning.
Databricks Assistant: Threads & history
April 29, 2024
Databricks Assistant now offers an opt-in experience that tracks query threads and history throughout editor contexts in your session. You can continue in the same thread, or start a new one if you want. See For an account: Disable or enable Databricks Assistant features for how to enable this experience.
Cancel pending serving endpoint updates in Model Serving
April 29, 2024
Mosaic AI Model Serving now supports cancelling in progress endpoint configuration updates from the Serving UI. See Modify a custom model endpoint.
Meta Llama 3 is now supported in Model Serving for Azure
April 25, 2024
Mosaic AI Model Serving now supports Meta Llama 3, a dense model architecture built and trained by Meta. Llama 3 is supported as part of Foundation Model APIs. Meta-Llama-3-70-Instruct is available in pay-per-token serving endpoint regions, and the suite of Llama 3 models is available in provisioned throughput serving endpoint regions. See Use Foundation Model APIs.
Data lineage now captures reads on tables with column masks and row-level security
April 25, 2024
Data lineage now captures reads on tables that include column masks and row-level security. See Filter sensitive table data using row filters and column masks and Capture and view data lineage using Unity Catalog.
Notebooks now automatically detect SQL
April 24, 2024
Databricks notebooks automatically add the %sql magic command when you enter SQL in a notebook set to a default language other than SQL. Previously, to prevent an error, you needed to manually add the %sql magic command.
New columns added to the billable usage system table (Public Preview)
April 24, 2024
The billable usage system table (system.billing.usage) now includes new columns that help you identify the specific product and features associated with the usage. See Billable usage system table reference.
Delta Sharing supports tables that use column mapping (Public Preview)
April 23, 2024
Delta Sharing now supports the sharing of tables that use column mapping. Recipients can read tables that use column mapping using a SQL warehouse, a cluster running Databricks Runtime 14.1 or above, or compute that is running open source delta-sharing-spark 3.1 or above.
See Add tables with deletion vectors or column mapping to a share, Read tables with deletion vectors or column mapping enabled, and Read tables with deletion vectors or column mapping enabled.
Get serving endpoint schemas (Public Preview)
April 22, 2024
You can now get the serving endpoint schema for your model serving or feature serving endpoints. This capability is available in Public Preview. See Get a model serving endpoint schema.
Creation and installation of workspace libraries is no longer available
April 22, 2024
Workspace libraries, which have been deprecated since November 2023, can no longer be created or installed on compute. Note that storing libraries as workspace files is distinct from workspace libraries and is still fully supported. You can install libraries stored as workspace files directly to compute or job tasks. See Libraries.
Jobs created through the UI are now queued by default
April 18, 2024
Queueing of job runs is now automatically enabled when a job is created in the Azure Databricks Jobs UI. When queueing is enabled, and a concurrency limit is reached, job runs are placed in a queue until capacity is available. See Enable queueing of job runs.
Configuring access to resources from serving endpoints is GA
April 17, 2024
You can now configure environment variables to access resources outside of your feature serving and model serving endpoints. This capability is generally available. See Configure access to resources from model serving endpoints.
Serverless compute for workflows is in public preview
April 16, 2024
Serverless compute for workflows allows you to run your Azure Databricks job without configuring and deploying infrastructure. With serverless compute for workflows, Azure Databricks efficiently manages the compute resources that run your job, including optimizing and scaling compute for your workloads. See Run your Azure Databricks job with serverless compute for workflows.
Lakehouse Federation supports foreign tables with case-sensitive identifiers
April 15, 2024
Lakehouse Federation is now able to federate foreign tables with case-sensitive identifiers for MySQL, SQL Server, BigQuery, Snowflake, and Postgres connections. See What is Lakehouse Federation?.
Compute cloning now clones any libraries installed on the original compute
April 11, 2024
When cloning compute, any libraries installed on the original compute will also be cloned. For cases where this behavior is unwanted, there is an alternative Create without libraries button on the compute clone page. See Clone a compute.
Private connectivity from the serverless compute is GA
April 10, 2024
Private connectivity from the serverless compute is now generally available. You can enforce that your serverless SQL warehouses connect to Azure storage over Azure Private Link. Private Endpoints can be configured and managed in the account using Network Connectivity Configurations (NCC). See Configure private connectivity from serverless compute.
UI experience for OAuth app registration
April 9, 2024
You can now enable custom OAuth applications in the Azure Databricks UI. You can also disable and modify both published and custom OAuth applications in the UI. For more information, see Enable custom OAuth applications using the Azure Databricks UI.
Delta Live Tables notebook developer experience improvements (Public Preview)
April 3, 2024
Databricks has released a set of Public Preview features in Azure Databricks notebooks that assist in the development and debugging of Delta Live Tables code. See Develop and debug Delta Live Tables pipelines in notebooks.