February 2021
These features and Azure Databricks platform improvements were released in February 2021.
Note
Releases are staged. Your Azure Databricks account may not be updated until a week or more after the initial release date.
New Azure Databricks Power BI connector (GA)
February 26, 2021
The new Power BI connector for Azure Databricks, released in public preview in September 2020, is now GA. It provides:
- Simple connection configuration: the new Power BI Azure Databricks connector is integrated into Power BI, and you configure it using a simple dialog with a couple of clicks.
- Authentication based on Microsoft Entra ID credentials—no more need for administrators to configure PAT tokens.
- Faster imports and optimized metadata calls, thanks to the new Azure Databricks ODBC driver, which comes with significant performance improvements.
- Access to Azure Databricks data through Power BI respects Azure Databricks table access control and Azure storage account permissions associated with your Microsoft Entra ID identity.
For more information, see Connect Power BI to Azure Databricks.
Diagnostic logging of user activities in Azure Databricks is GA
February 23 - March 2, 2021: Version 3.40
Capture a detailed audit trail of the activities that your users perform in Azure Databricks, now generally available.
Added modification_time to the DBFS REST API get-status and list responses
February 23 - March 2, 2021: Version 3.40
A file object’s last modification time is now available in the DBFS REST API. The modification_time
is now returned in the FileInfo response structure by both the get-status and list API endpoints. An example JSON response for a get-status call on a directory:
{"path": "/test-directory", "is_dir": true, "file_size": 0, "modification_time": 1610366751000}
See DBFS API.
Easily copy long experiment names in MLflow
February 16-23, 2021: Version 3.39
You can now copy the experiment name from the experiment page, making it easier to set the active MLflow experiment to which a notebook logs runs. See Copy experiment name.
Adjust memory size and number of cores for serving clusters
February 16-23, 2021: Version 3.39
You can now customize clusters used for MLflow model serving. See Feature store integration.
Web terminal is now GA
February 16-23, 2021: Version 3.39
Web terminal is now generally available. It provides a convenient and highly interactive way for you to run shell commands and use editors on the Spark driver node. See Run shell commands in Azure Databricks web terminal.
Apache Spark connector for Azure SQL Database and SQL Server supports Databricks Runtime 7.x and above
February 16, 2021
The Spark connector for Azure SQL Database and SQL Server now supports Spark 3 and therefore Databricks Runtime 7.x and above.
Databricks Runtime 7.2 series support ends
February 11, 2021
Support for Databricks Runtime 7.2, Databricks Runtime 7.2 for Machine Learning, and Databricks Runtime 7.2 for Genomics ended on February 11. See Databricks support lifecycles.
Databricks Runtime 7.6 GA
February 8, 2021
Databricks Runtime 7.6 and Databricks Runtime 7.6 ML are now generally available.
For information, see the full release notes at Databricks Runtime 7.6 (EoS) and Databricks Runtime 7.6 for Machine Learning (EoS).
Databricks Runtime 8.0 (Beta)
February 1, 2021
Databricks Runtime 8.0 and Databricks Runtime 8.0 ML are now available as Beta releases.
For information, see the full release notes at Databricks Runtime 8.0 (EoS) and Databricks Runtime 8.0 for ML (EoS).
Databricks Runtime for Genomics now deprecated
February 1, 2021
Databricks is no longer building new Databricks Runtime for Genomics releases and will remove support for Databricks Runtime for Genomics on September 24, 2022, when Databricks Runtime for Genomics 7.3 LTS support ends. At that point Databricks Runtime for Genomics will no longer be available for selection when you create a cluster.
For more information about the Databricks Runtime deprecation policy and schedule, see Supported Databricks Runtime LTS releases.
View more readable JSON in the MLflow run artifact display
February 4-11, 2021: Version 3.38
JSON output in the MLflow artifact panel is now in a more readable format.
Provide comments in the Model Registry using REST API
February 4-11, 2021: Version 3.38
You can now add comments in the Model Registry using the REST API.
Easily specify default cluster values in API calls
February 4-11, 2021: Version 3.38
Now you can use the applyDefaultPolicyValues
field when you create and edit all-purpose clusters using the Databricks Clusters API. When true
, it uses policy default values for missing cluster attributes.
Tune cluster worker configuration according to current worker allocation
February 4-11, 2021: Version 3.38
The cluster list displays the number of workers allocated to each running cluster. This number is now shown in the cluster detail page so you can easily compare current sizing of the cluster with the cluster configuration details and make configuration adjustments as needed.
Pass context specific information to a job’s task with task parameter variables
February 4-11, 2021: Version 3.38
You can now pass variables to tasks containing context specific information such as the job ID or execution start time. For more information see What is a dynamic value reference?.
Error messages from job failures no longer contain possibly sensitive information
February 4-11, 2021: Version 3.38
Job failure error messages no longer include stack traces from Java exceptions. The stack traces, included in the miscMessage
field, potentially exposed sensitive information to users.