January 2021

These features and Azure Databricks platform improvements were released in January 2021.

Note

Releases are staged. Your Azure Databricks account may not be updated until a week or more after the initial release date.

This month saw the release of Azure Databricks platform version 3.37. There was no release of versions 3.35 or 3.36.

Secure cluster connectivity (no public IPs) is GA

January 28, 2021

Secure cluster connectivity (no public IPs), now GA, lets you launch clusters in which all nodes have only private IP addresses, providing enhanced security. You can enable secure cluster connectivity for new workspaces. If you have workspaces with public IPs that you would like to migrate, you should create new workspaces enabled for secure cluster connectivity and migrate your resources to the new workspaces. Contact your Microsoft or Databricks account team for details.

If you use secure cluster connectivity, there are changes for egress from your workspace’s subnets:

• If you use the default (managed) VNet, Azure Databricks now creates a NAT gateway, rather than using the default SNAT. The NAT gateway is created within the managed resource group that Azure Databricks creates and manages. You cannot modify this resource group nor any resources provisioned in it.
• If you use the optional VNet injection feature to provide your own VNet, you have several options for controlling egress. The documentation has been updated to reflect the updated guidance for workspace deployments.

For details, see Secure cluster connectivity.

Databricks Runtime 7.1 series support ends

January 21, 2021

Support for Databricks Runtime 7.1, Databricks Runtime 7.1 for Machine Learning, and Databricks Runtime 7.1 for Genomics ended on January 21. See Databricks support lifecycles.

Start clusters faster with Docker images preloaded into instance pools

January 20-28, 2021: Version 3.37

When you create an instance pool using the API, you can now specify preloaded Docker images. Pool clusters that use a preloaded Docker image start faster as they do not have to wait for the image to download. See Create instance pool.

Notebook find and replace now supports changing all occurrences of a match

January 20-28, 2021: Version 3.37

You can now choose to replace all matches when using the find and replace capability in a notebook. For details, see Find and replace text.

Single Node clusters (GA)

January 20-28, 2021: Version 3.37

A Single Node cluster is a cluster consisting of a Spark driver and no Spark workers. In contrast, Standard mode clusters require at least one Spark worker to run Spark jobs. Single Node clusters are helpful in the following situations:

• Running single node machine learning workloads that need Spark to load and save data
• Lightweight exploratory data analysis (EDA)

For details, see Single-node or multi-node compute.

Free form cluster policy type renamed to Unrestricted

January 20-28, 2021: Version 3.37

The Free form cluster policy type has been renamed to Unrestricted. For details, see Create and manage compute policies and Policies.

Cluster policy field not shown if a user only has access to one policy

January 20-28, 2021: Version 3.37

When you create a standard or jobs cluster, the Cluster Policy field does not appear if you have access to only one policy or if no policies have been defined.

Databricks Runtime 7.0 series support ends

January 14, 2021

Support for Databricks Runtime 7.0, Databricks Runtime 7.0 for Machine Learning, and Databricks Runtime 7.0 for Genomics ended on January 14. See Databricks support lifecycles.