March 2023

  • Article

These features and Azure Databricks platform improvements were released in March 2023.

Note

Releases are staged. Your Azure Databricks account might not be updated until a week or more after the initial release date.

Databricks Terraform provider updated to version 1.14.0

March 31, 2023

Version 1.14.0 adds an example for automatically rotating an access token, a run_if condition and a file arrival trigger to the databricks_job resource, and more. For more details, see the changelog for version 1.14.0.

Databricks Runtime 7.3 LTS ML support ends

March 31, 2023

Support for Databricks Runtime 7.3 LTS for Machine Learning ended on March 31. See Databricks runtime support lifecycles.

Distributed training with TorchDistributor

March 27, 2023

With Databricks Runtime 13.0 ML and above, you can perform distributed training on PyTorch ML models using TorchDistributor. See Distributed training with TorchDistributor.

TorchDistributor is an open-source module in PySpark that helps users do distributed training with PyTorch on their Spark clusters, so it lets you launch PyTorch training jobs as Spark jobs.

Databricks Runtime 13.0 (Beta)

March 24, 2023

Databricks Runtime 13.0 and Databricks Runtime 13.0 ML are now available as Beta releases.

See Databricks Runtime 13.0 (unsupported) and Databricks Runtime 13.0 for Machine Learning (unsupported).

Improved file editor

March 24, 2023

The file editor has been updated to bring many of the notebook features to your file editing experience. Additionally, you now can directly execute Python, SQL, R, and Scala files from the file editor. For details, see Use the Databricks notebook and file editor.

Change the DBFS Root Storage Redundancy for existing Azure Databricks Workspaces

March 23, 2023

Users now have the option to change the redundancy of their DBFS root storage account. See Change workspace storage redundancy options.

.ipynb (Jupyter) notebook support in Repos (preview)

March 22, 2023

Support for Jupyter notebooks (.ipynb files) is available in Repos. You can clone repositories with .ipynb notebooks, work in Databricks UI, and then commit and push as .ipynb notebooks. Metadata such as a notebook dashboard is preserved. Admins can control whether outputs can be committed or not.

You can also:

  • Create new .ipynb notebooks.
  • Convert notebooks to .ipynb file format.
  • View diffs as Code diff (code changes in cells) or Raw diff (code changes in JSON, including metadata).

See Allow committing .ipynb notebook output.

Support for reload4j

March 21, 2023

Reload4j 1.2.19 is replacing log4j in Databricks Runtime versions 10.4 and earlier.

New IP addresses for East Asia region

March 15, 2023

Azure Databricks has added a new dedicated control plane for the eastasia region. If you host a Azure Databricks workspace in your own VNet (also known as VNet injection) in the East Asia region, you likely manage network traffic for your workspace using custom routes (UDR). If so, you must update those routes using the additional IP addresses and domains listed at IP addresses and domains.

Execute SQL cells in the notebook in parallel

March 15, 2023

You can now run SQL cells in Azure Databricks notebooks in parallel while attached to an interactive cluster. See Execute SQL cells in parallel.

Create job tasks using Python code stored in a Git repo

March 14, 2023

You can now retrieve Python code from a Git provider when you add a Python task to an Azure Databricks job, simplifying integration of your jobs with your CI/CD workflows. See Use Python code from a remote Git repository.

Databricks Terraform provider updated to version 1.13.0

March 14, 2023

Version 1.13.0 adds a databricks_sql_alert resource, and more. For more details, see the changelog for version 1.13.0.

Databricks Terraform provider updated to version 1.12.0

March 9, 2023

Version 1.12.0 adds a databricks_model_serving resource, deprecates the schedule block within the databricks_sql_query resource, and more. For more details, see the changelog for version 1.12.0.

SQL admin console and workspace admin console combined

March 9, 2023

The SQL admin console has been combined with the general admin settings to create a unified experience for admin users. All SQL admin settings are now accessed from the admin console.

Model Serving is GA

March 7, 2023

Model Serving, formerly Serverless Real-Time Inference, is now generally available.

Model Serving provides a highly available and low-latency service for deploying models. The service automatically scales up or down to meet demand changes within the chosen concurrency range. See Model serving with Azure Databricks.

Automatic feature lookup is GA

March 7, 2023

Automatic feature lookup with Model Serving is now generally available. For details, see Automatic feature lookup with MLflow models on Databricks.

For new workspaces that use UDRs, Azure storage IP needs an additional UDR

March 6, 2023

Due to changes in DBFS root storage for new workspaces, new workspaces that use VNet injection and define user-defined routes (UDRs) using IP addresses instead of the recommended service tags must add an additional Azure UDR for each VNet. To configure the two UDRs, see DBFS root storage IP address. This change does not affect existing workspaces.

Azure Databricks DBFS root storage uses Azure Data Lake Storage Gen2 for new workspaces

March 6, 2023

New Azure Databricks workspaces use Azure Data Lake Storage Gen2 storage accounts for the DBFS root. Existing Azure Databricks workspaces continue to use Blob storage for the DBFS root. See Introduction to Azure Data Lake Storage Gen2.

New Catalog Explorer availability

March 6, 2023

A new version of Catalog Explorer is now available in all workspaces that use a supported version of Databricks Runtime 7.3 or later.

View frequent queries and users of a table using the Insights tab

March 6, 2023

You can now use the Insights tab in Catalog Explorer to view the most frequent queries and users of any table registered in Unity Catalog. You must have the SELECT privilege on a table to see this data, and you only see queries that you have permission to view. See View frequent queries and users of a table.

March 3, 2023

You can now search for exact phrases using quotation marks in global search. See Search for workspace objects.

View lineage information for your Azure Databricks jobs

March 3, 2023

If Unity Catalog is enabled in your workspace, you can view lineage information for your jobs in the Azure Databricks Jobs UI, including upstream tables the job reads from and downstream tables the job writes to. See View lineage information for a job.

Databricks Runtime 12.2 LTS and Databricks Runtime 12.2 LTS ML are GA

March 2, 2023

Databricks Runtime 12.2 LTS and Databricks Runtime 12.2 LTS ML are now generally available.

See Databricks Runtime 12.2 LTS and Databricks Runtime 12.2 LTS for Machine Learning.

Workspace files are now in Public Preview

March 6, 2023

You can now work with non-notebook files in Azure Databricks. Workspaces files are enabled by default in all workspaces. See What are workspace files?.