December 2019

  • Article

These features and Azure Databricks platform improvements were released in December 2019.

Note

Releases are staged. Your Azure Databricks account may not be updated until a week or more after the initial release date.

Databricks Connect now supports Databricks Runtime 6.2

December 17, 2019

Databricks Connect now supports Databricks Runtime 6.2.

Configure clusters with your own container image using Databricks Container Services (GA)

December 16, 2019: Version 3.7

Generally available in Databricks Runtime 6.1 and Azure Databricks platform version 3.7, Databricks Container Services lets you configure a cluster with your own container image. You can pre-package complex environments within a container, publish it to a popular container registry such as ACR, ECR, or Docker Hub, and then have Azure Databricks pull the image to create a cluster. Some example use cases include:

  • Library customization - you have full control over the system libraries you want installed
  • Golden container environment - your Docker image is a locked down environment that will never change
  • Docker CI/CD integration - you can integrate Azure Databricks with your Docker CI/CD pipelines

There are many other use cases, ranging from specifying configuration to installing machine learning packages.

For details, see Customize containers with Databricks Container Service.

Databricks Runtime 6.2 for Genomics GA

December 3, 2019

Databricks Runtime 6.2 for Genomics is built on top of Databricks Runtime 6.2. It includes many improvements and upgrades from Databricks Runtime 6.1 for Genomics, including:

  • Firth logistic regression
  • User-defined sample quality control metrics
  • Pipe transformer performance improvement
  • More robust joint genotyping
  • Simplified integration with LOFTEE
  • Hail 0.26.0
  • Samtools 1.9

December 3, 2019

Now that the Azure Databricks SCIM Provisioning Connector is available in the Microsoft Entra ID (formerly Azure Active Directory) app gallery, it is easier to set up provisioning of users and groups from Microsoft Entra ID to Azure Databricks. For details, see Configure SCIM provisioning using Microsoft Entra ID (Azure Active Directory).

Databricks Runtime 5.3 and 5.4 support ends

December 3, 2019

Support for 5.3 and 5.4 ended on December 3. See Databricks runtime support lifecycles.

Databricks Runtime 6.2 ML GA

December 3, 2019

Databricks Runtime 6.2 ML GA brings many library upgrades, including:

  • TensorFlow and TensorBoard: 1.14.0 to 1.15.0.
  • PyTorch: 1.2.0 to 1.3.0.
  • tensorboardX: 1.8 to 1.9.
  • MLflow: 1.3.0 to 1.4.0.
  • Hyperopt: 0.2-db1 with Azure Databricks MLflow integrations.
  • mleap-databricks-runtime to 0.15.0 and includes mleap-xgboost-runtime.

For more information, see the complete Databricks Runtime 6.2 for ML (unsupported) release notes.

Databricks Runtime 6.2 GA

December 3, 2019

Databricks Runtime 6.2 GA brings new features, improvements, and many bug fixes, including:

  • Optimized Delta Lake insert-only merge

For more information, see the complete Databricks Runtime 6.2 (unsupported) release notes.

Databricks Connect now supports Databricks Runtime 6.1

December 3, 2019

Databricks Connect now supports Databricks Runtime 6.1. Databricks Connect allows you to connect your favorite IDE (IntelliJ, Eclipse, PyCharm, RStudio, Visual Studio), notebook server (Zeppelin, Jupyter), and other custom applications to Azure Databricks clusters and run Apache Spark code.