March 2018
Releases are staged. Your Azure Databricks account may not be updated until a week after the initial release date.
Command execution details
March 27 - April 3, 2018: Version 2.68
When you run a command in a notebook, you will now see detailed progress information.
Databricks CLI supports --profile
March 27 - April 3, 2018: Version 2.68
Databricks CLI 0.6.1 supports --profile in all positions.
ACLs enabled by default for new Premium SKU customers
March 27 - April 3, 2018: Version 2.68
Access control lists (ACLs) are now enabled by default for all new customers in the Premium SKU. Existing customers must continue to enable ACLs manually.
See Access controls lists can no longer be disabled.
Azure Databricks is now generally available
March 22, 2018
We’re pleased to announce that Azure Databricks is now generally available. In the past weeks, we have added functionality to help make the Azure Databricks experience even better, including:
- Virtual network (VNet) peering, which enables you to use Apache Kafka on HDInsight with Azure Databricks
- Microsoft Power BI now includes a built-in Spark connector
- Azure Synapse Analytics connector
- Azure Event Hubs (and IoT Hub) connector
- Azure Data Factory v2 (preview) support
- Secrets management (preview)
- More Azure regions
- Improved performance and documentation
And of course Azure Databricks continues to provide easy integrations with Azure Blob storage, Azure Data Lake Store, and Azure Cosmos DB.
As a supplement to the documentation provided on this site, learn.microsoft.com provides introductory material, information about Azure account management, and end-to-end tutorials.
New doc site theme
March 21, 2018
We’ve updated the look and feel of our documentation site. We hope you like it!
Autoscaling local storage
March 13-20, 2018: Version 2.67
All clusters on Azure Databricks launch with autoscaling local storage enabled. This means that Azure Databricks automatically attaches additional managed disks to cluster worker VMs whenever they run low on disk.
See Enable autoscaling local storage for more information.
Virtual network (VNet) peering
March 13-20, 2018: Version 2.67
Adds support for virtual network (VNet) peering, which allows the virtual network in which your Azure Databricks resource is running to peer with another Azure virtual network.
See Peer virtual networks for more information.
Cluster event log
Mar 13-20, 2018: Version 2.67
The cluster details page has a new Event Log tab that displays important cluster life cycle events. Historical events can be viewed for 60 days, which is comparable with other data retention times in Azure Databricks.
See Compute event logs for more information.
Databricks CLI: 0.6.0 release
Mar 13, 2018: databricks-cli 0.6.0
Databricks CLI now supports Python 3.
See Databricks CLI (legacy) for more information.
Job run management
Mar 13-20, 2018: Version 2.67
You can now delete a job run in the job details page and the job run page.
The job run Get Output endpoint is GA, and the maximum output returned has been increased to 5 MB.
Edit cluster permissions now requires edit mode
Mar 13-20, 2018: Version 2.67
Previously it was possible to edit a cluster’s permissions without clicking Edit, which was inconsistent with other cluster attributes.
A side effect of this change is that you can no longer edit cluster permissions while a cluster is pending.
Databricks ML Model Export
March 1, 2018
The documentation now covers how to use Databricks ML Model Export, which allows you to export models and full ML pipelines from Apache Spark. These exported models and pipelines can be imported into other (Spark and non-Spark) platforms to do scoring and make predictions. Model Export is targeted at low-latency, lightweight ML-powered applications.
Note
This feature requires Databricks Runtime 4.0+.
See MLeap ML model export for more information.