November 2020

  • Article

These features and Azure Databricks platform improvements were released in November 2020.

Note

Releases are staged. Your Azure Databricks account may not be updated until a week or more after the initial release date.

Databricks Runtime 6.6 series support ends

November 26, 2020

Support for Databricks Runtime 6.6, Databricks Runtime 6.6 for Machine Learning, and Databricks Runtime 6.6 for Genomics ended on November 26. See Databricks runtime support lifecycles.

MLflow Model Registry GA

November 18 - December 1, 2020: Version 3.33

MLflow Model Registry is now GA. Several improvements have been made since Model Registry was released for Public Preview:

  • Audit logging for actions on model registry objects. Actions in Model Registry are now captured in audit logs. See the modelRegistry entry in the audit log reference for the logged actions and parameters.
  • Comments for model versions. You can now add comments to model versions, allowing you to use Model Registry for team discussions to help manage your model productionization pipeline.
  • Tags on models and model versions. You can create tags for models and model versions, and search for them using the API.
  • Improvements to the URL of the registered models page. The URL of this page now keeps its history, so you can navigate with the browser back and forward buttons as you make queries and view models from this page. You can also share the URL with colleagues who will see the same view.

Filter experiment runs based on whether a registered model is associated

November 18 - December 1, 2020: Version 3.33

When viewing runs for an experiment, you can now filter runs based on whether they created a model version or not. For more information, see Filter runs.

November 18 - December 1, 2020: Version 3.33

The Partner Integrations gallery has moved from the Account menu to the Add Data tab. For more information, see Technology partners.

Cluster policies now use allowlist and blocklist as policy type names

November 18 - December 1, 2020: Version 3.33

Cluster policies now use “allowlist” and “blocklist” as policy types, replacing “whitelist” and “blacklist.” See Compute policy reference. Note that this was originally announced as a version 3.31 feature, which was incorrect.

Automatic retries when the creation of a job cluster fails

Important

This update was reverted following the release of version 3.33.

November 18 - December 1, 2020: Version 3.33

Azure Databricks now automatically retries the creation of job clusters when specific recoverable errors occur. Job runs remain in RunLifeCycleState: PENDING until successful cluster launch. Each attempt has a different cluster_id and name. When cluster creation succeeds, the run transitions to RunLifeCycleState: RUNNING.

November 18 - December 1, 2020: Version 3.33

You can now view a table of contents for your notebooks and use it to quickly navigate within a notebook. The notebook table of contents is automatically created based on the Markdown headings. For more information, see Notebook table of contents.

Databricks SQL (Public Preview)

November 18, 2020

Databricks is pleased to introduce Databricks SQL, an intuitive environment for running ad-hoc queries and creating dashboards on data stored in your data lake. Databricks SQL empowers your organization to operate a multi-cloud lakehouse architecture that provides data warehousing performance with data lake economics while providing a delightful SQL analytics user experience. Databricks SQL:

  • Integrates with the BI tools you use today, like Tableau and Microsoft Power BI, to query the most complete and recent data in your data lake.
  • Complements existing BI tools with a SQL-native interface that allows data analysts and data scientists to query data lake data directly within Azure Databricks.
  • Enables you to share query insights through rich visualizations and drag-and-drop dashboards with automatic alerting for important data changes.
  • Uses Create a SQL warehouse to bring reliability, quality, scale, security, and performance to your data lake, so you can run traditional analytics workloads using your most recent and complete data.

See the What is data warehousing on Azure Databricks? for details.

Single Node clusters now support Databricks Container Services

November 4-10, 2020: Version 3.32

You can now use Databricks Container Services on Single Node clusters. For more information, see Single-node or multi-node compute and Customize containers with Databricks Container Service.

Databricks Runtime 7.4 GA

November 3, 2020

Databricks Runtime 7.4, Databricks Runtime 7.4 ML, and Databricks Runtime 7.4 for Genomics are now generally available.

For information, see the full release notes at Databricks Runtime 7.4 (unsupported) and Databricks Runtime 7.4 for ML (unsupported).

Databricks JDBC driver update

November 3, 2020

A new version of the Databricks JDBC driver has been released. The new version contains a number of bug fixes, most notably, the driver now returns the correct number of modified rows from DML operations when it is provided by Databricks Runtime.

Databricks Connect 7.3 (Beta)

November 3, 2020

Databricks Connect 7.3 is now available as a Beta release.

Databricks Connect 7.3 lets you use Microsoft Entra ID tokens to authenticate to Azure Databricks and supports Microsoft Entra ID credential passthrough. This enables you to authenticate automatically to Azure Data Lake Storage Gen1 and Azure Data Lake Storage Gen2 from Databricks Connect using the same Microsoft Entra ID identity that you use to authenticate to Azure Databricks.

For more information, see Databricks Connect and Databricks Connect release notes.