Databricks Runtime 6.0 (unsupported)

Article
03/01/2024

Databricks released this image in October 2019.

The following release notes provide information about Databricks Runtime 6.0, powered by Apache Spark.

New features

Python environment

Databricks Runtime 6.0 includes major changes to Python and the way Python environments are configured, including upgrading Python to 3.7.3, refining the list of installed Python packages, and upgrading those installed packages to newer versions. For details, see Installed Python libraries.

In addition, as was previously announced, Databricks Runtime 6.0 does not support Python 2.

Major changes include:

Upgraded Python from 3.5.2 to 3.7.3. Some old versions of Python packages may not be forward compatible with Python 3.7 because they depend on old versions of Cython that are not compatible with Python 3.7. Installing such a package can trigger errors similar to ‘PyThreadState’ {‘struct _ts’} has no member named ‘exc_type’ (see GitHub issue 1978 for details). Install Python 3.7 compatible versions of Python packages instead.
Major package upgrades:
- boto3 to 1.9.162
- ipython to 7.4.0
- matplotlib to 3.0.3
- numpy to 1.16.2
- pandas to 0.24.2
- pyarrow to 0.13.0
Compared with Databricks Runtime 5.5 LTS (unsupported), the following Python packages are newly included: asn1crypto, backcall, jedi, kiwisolver, parso, and PySocks.
Compared with Databricks Runtime 5.5 LTS (unsupported), the following Python packages are not installed: ansi2html, brewer2mpl, colorama, configobj, enum34, et-xmlfile, freetype-py, funcsigs, fusepy, ggplot, html5lib, ipaddress, jdcal, Jinja2, llvmlite, lxml, MarkupSafe, mpld3, msgpack-python, ndg-httpsclient, numba, openpyxl, pathlib2, Pillow, ply, pyasn1, pypng, python-geohash, scour, simplejson, and singledispatch.
The display function on Python ggplot objects is no longer supported because the ggplot package is not compatible with newer version of pandas.
Setting PYSPARK_PYTHON to /databricks/python2/bin/python is not supported because Databricks Runtime 6.0 does not support Python 2. A cluster with such setting can still start. However, Python notebooks and Python commands will not work, that is, Python command cells will fail with a “Canceled” error and a Python shell failed to start error will appear in the driver logs.
If PYSPARK_PYTHON points to a Python executable that is in an environment managed by Virtualenv , this environment will be activated for init scripts and notebooks. You can use python and pip commands that are defined in the activated environment directly without having to specify the absolute locations of these commands. By default, PYSPARK_PYTHON is set to /databricks/python3/bin/python. Thus, by default, python points to /databricks/python3/bin/python and pip points to /databricks/python3/bin/pip for init scripts and notebooks. If your PYSPARK_PYTHON points to a Python executable that is not in an environment managed by Virtualenv or if you are writing an init script to create the Python specified by PYSPARK_PYTHON, you will need to use absolute paths to access the correct python and pip. When Python library isolation is enabled (it is enabled by default), the activated environment is still the environment that PYSPARK_PYTHON is associated with. We recommend that you use Library utility (dbutils.library) (legacy) to modify the isolated environment associated with a Python notebook.

Scala and Java APIs for Delta Lake DML commands

You can now modify data in Delta tables using programmatic APIs for delete, update, and merge. These APIs mirror the syntax and semantics of their corresponding SQL commands and are great for many workloads, for example, slowly changing dimension (SCD) operations, merging change data for replication, and upserts from streaming queries.

For details, see What is Delta Lake?.

Scala and Java APIs for Delta Lake utility commands

Databricks Runtime now has programmatic APIs for the vacuum and history utility commands. These APIs mirror the syntax and semantics of their corresponding SQL commands available in earlier versions of Databricks Runtime.

You can clean up files that are no longer referenced by a Delta table and are older than the retention threshold by running vacuum on the table. Running the vacuum command on the table recursively vacuums the directories associated with the Delta table. The default retention threshold for the files is 7 days. The ability to time travel back to a version older than the retention period is lost after running vacuum. vacuum is not triggered automatically.

You can retrieve information on the operations, user, timestamp, and so on for each write to a Delta table by running the history command. The operations are returned in reverse chronological order. By default, table history is retained for 30 days.

For details, see What is Delta Lake?.

Disk caching available for Azure Lsv2 instances

[Disk caching]](../../optimizations/disk-cache.md) is now enabled by default for all Lsv2 instances.

Optimized storage using local file APIs

Local file APIs are useful as they allow you to access files from underlying distributed object storage as local files. In Databricks Runtime 6.0, we enhanced the FUSE mount that enables local file APIs to address key limitations. Databricks Runtime 6.0 significantly improves read and write speed and supports files that are larger than 2 GB. If you need faster and more reliable reads and writes such as for distributed model training, you would find this enhancement particularly useful. In addition, you would not need to load data to a local storage for your workloads, both saving cost and improving productivity.

For details, see What is the Databricks File System (DBFS)?.

Multiple matplotlib plots per notebook cell

You can now display multiple matplotlib plots per notebook cell:

Multiple matplotlib plots in cell

Service credentials for multiple Azure Data Lake Storage Gen1 accounts

You can now set up service credentials for multiple Accessing Azure Data Lake Storage Gen1 from Azure Databricks accounts for use within in a single Apache Spark session. To do this, add account.<account-name> to the configuration keys. For example, if you want to set up credentials for accounts to access both adl://example1.azuredatalakestore.net and adl://example2.azuredatalakestore.net, you can do this as follows:

spark.conf.set("fs.adl.oauth2.access.token.provider.type", "ClientCredential")

spark.conf.set("fs.adl.account.example1.oauth2.client.id", "<application-id-example1>")
spark.conf.set("fs.adl.account.example1.oauth2.credential", dbutils.secrets.get(scope = "<scope-name>", key = "<key-name-for-service-credential-example1>"))
spark.conf.set("fs.adl.account.example1.oauth2.refresh.url", "https://login.microsoftonline.com/<directory-id-example1>/oauth2/token")

spark.conf.set("fs.adl.account.example2.oauth2.client.id", "<application-id-example2>")
spark.conf.set("fs.adl.account.example2.oauth2.credential", dbutils.secrets.get(scope = "<scope-name>", key = "<key-name-for-service-credential-example2>"))
spark.conf.set("fs.adl.account.example2.oauth2.refresh.url", "https://login.microsoftonline.com/<directory-id-example2>/oauth2/token")

Improvements

Upgraded AWS SDK to 1.11.596.
Upgraded Azure Storage SDK in the WASB driver to 7.0.
OPTIMIZE now provides a summary of metrics such as number of files added, number of files removed, and max and min file size. See Compact data files with optimize on Delta Lake.

Removal

Databricks ML Model Export is removed. Use MLeap for importing and exporting models instead.

Apache Spark

Note

This article contains references to the term slave, a term that Azure Databricks does not use. When the term is removed from the software, we’ll remove it from this article.

Databricks Runtime 6.0 includes Apache Spark 2.4.3. This release includes all Spark fixes and improvements included in Databricks Runtime 5.5 LTS (unsupported), as well as the following additional bug fixes and improvements made to Spark:

[SPARK-27992][SPARK-28881][PYTHON] Allow Python to join with connection thread to propagate errors
[SPARK-27330][SS]support task abort in foreach writer (6.0, 5.x)
[SPARK-28642][SQL] Hide credentials in SHOW CREATE TABLE
[SPARK-28699][CORE] Fix a corner case for aborting indeterminate stage
[SPARK-28647][WEBUI] Recover additional metric feature
[SPARK-28766][R][DOC] Fix CRAN incoming feasibility warning on invalid URL
[SPARK-28486][CORE][PYTHON] Map PythonBroadcast’s data file to a BroadcastBlock to avoid delete by GC
[SPARK-25035][CORE] Avoiding memory mapping at disk-stored blocks replication
[SPARK-27234][SS][PYTHON] Use InheritableThreadLocal for current epoch in EpochTracker (to support Python UDFs)
[SPARK-28638][WEBUI] Task summary should only contain successful tasks’ metrics
[SPARK-28153][PYTHON] Use AtomicReference at InputFileBlockHolder (to support input_file_name with Python UDF)
[SPARK-28564][CORE] Access history application defaults to the last attempt id
[SPARK-28260] Cluster can auto-terminate while thriftserver query is still fetching results
[SPARK-26152][CORE] Synchronize Worker Cleanup with Worker Shutdown
[SPARK-28545][SQL] Add the hash map size to the directional log of ObjectAggregationIterator
[SPARK-28489][SS] Fix a bug that KafkaOffsetRangeCalculator.getRanges may drop offsets
[SPARK-28421][ML] SparseVector.apply performance optimization
[SPARK-28156][SQL] Self-join should not miss cached view
[SPARK-28152][SQL] Mapped ShortType to SMALLINT and FloatType to REAL for MsSqlServerDialect
[SPARK-28054][SQL] Fix error when insert Hive partitioned table dynamically where partition name is upper case
[SPARK-27159][SQL] update mssql server dialect to support binary type
[SPARK-28355][CORE][PYTHON] Use Spark conf for threshold at which com…
[SPARK-27989][CORE] Added retries on the connection to the driver for k8s
[SPARK-27416][SQL] UnsafeMapData & UnsafeArrayData Kryo serialization …
[SPARK-28430][UI] Fix stage table rendering when some tasks’ metrics are missing
[SPARK-27485] EnsureRequirements.reorder should handle duplicate expressions gracefully
[SPARK-28404][SS] Fix negative timeout value in RateStreamContinuousPartitionReader
[SPARK-28378][PYTHON] Remove usage of cgi.escape
[SPARK-28371][SQL] Make Parquet “StartsWith” filter null-safe
[SPARK-28015][SQL] Check stringToDate() consumes entire input for the yyyy and yyyy-[m]m formats
[SPARK-28302][CORE] Make sure to generate unique output file for SparkLauncher on Windows
[SPARK-28308][CORE] CalendarInterval sub-second part should be padded before parsing
[SPARK-28170][ML][PYTHON] Uniform Vectors and Matrix documentation
[SPARK-28160][CORE] Fix a bug that callback function may hang when unchecked exception missed
[SPARK-27839][SQL] Change UTF8String.replace() to operate on UTF8 bytes
[SPARK-28157][CORE] Make SHS clear KVStore LogInfo for the blacklisted entries
[SPARK-28128][PYTHON][SQL] pandas Grouped UDFs skip empty partitions
[SPARK-28012][SQL] Hive UDF supports struct type foldable expression
[SPARK-28164] Fix usage description of start-slave.sh
[SPARK-27100][SQL] Use Array instead of Seq in FilePartition to prevent StackOverflowError
[SPARK-28154][ML] GMM fix double caching

Maintenance updates

See Databricks Runtime 6.0 maintenance updates.

System environment

Operating System: Ubuntu 16.04.6 LTS
Java: 1.8.0_232
Scala: 2.11.12
Python: 3.7.3
R: R version 3.6.1 (2019-07-05)
Delta Lake: 0.3.0