Databricks Runtime 9.1 LTS

Artikkeli
09/03/2024

The following release notes provide information about Databricks Runtime 9.1 LTS and Databricks Runtime 9.1 LTS Photon, powered by Apache Spark 3.1.2. Databricks released this version in September 2021. Photon is in Public Preview.

Note

LTS means this version is under long-term support. See Databricks Runtime LTS version lifecycle.

Tip

To see release notes for Databricks Runtime versions that have reached end-of-support (EoS), see End-of-support Databricks Runtime release notes. The EoS Databricks Runtime versions have been retired and might not be updated.

New features and improvements

Auto Loader schema hints now work with array and map types
Avro file support for merge schema
Auto Loader incremental listing support (Public Preview)
Delta now supports arbitrary replaceWhere
Auto Loader for Google Cloud now supports file notifications (Public Preview)
CREATE FUNCTION now supports creating table functions
Kafka Streaming Source now reports estimatedTotalBytesBehindLatest metric
For structs inside of arrays, Delta MERGE INTO now resolves struct fields by name and evolves struct schemas

Auto Loader schema hints now work with array and map types

Array and map types are supported in Override schema inference with schema hints for Auto Loader.

Examples of schema hints for arrays include:

arr Array<TYPE> changes the array type.
arr.element TYPE changes the array type (by using the element keyword).
arr.element.x TYPE changes a nested field type in an array of structures.

The first two examples are hints to change the type of array arr to TYPE, but they use different syntax. In the third example, arr is an array of structures with a field x. This example shows how to change the type of x to TYPE.

Examples of schema hints for maps include:

m Map<KEY-TYPE, VALUE-TYPE> changes map key and value types.
m.key TYPE changes the type of map keys.
m.value TYPE changes the type of map values.
m.key.x TYPE changes the field type in a map key.
m.value.x TYPE changes the field type of a map value.

The first example changes both the key and value types of map m to KEY_TYPE and VALUE_TYPE respectively. The second and third examples can be used if only the key type or only the value type needs to be changed. In the fourth and fifth examples, m is a map with key and value of structure types with a field x. This example shows how to change the type of x to TYPE.

Avro file support for merge schema

The Avro file format now supports the mergeSchema option when reading files. Setting mergeSchema to true when reading Avro files will infer a schema from a set of Avro files rather than from a single file. This improves usability by inferring a schema that may be able to read all files even if their individual schemas differ. See Configuration.

Auto Loader incremental listing support (Public Preview)

In the case of lexicographically generated files, What is Auto Loader? now leverages lexical file ordering and existing optimized APIs to make the directory listing more efficient by listing from previously-ingested files rather than by listing the entire directory. Auto Loader automatically detects whether a given directory is suitable for incremental listing by default. To control this behavior explicitly, set the new cloudFiles.useIncrementalListing option to on (true), off (false), or automatic (auto). If you set this behavior to true, you can also set the cloudFiles.backfillInterval option to schedule regular backfills over your data, to make sure all of your data is completely ingested.

Delta now supports arbitrary replaceWhere

In combination with overwrite mode, the replaceWhere option can be used to simultaneously overwrite data that matches a predicate defined in the option. Previously, replaceWhere supported a predicate only over partition columns, but it can now be an arbitrary expression. See Write to a table.

Auto Loader for Google Cloud now supports file notifications (Public Preview)

Auto Loader now supports file notification mode on Google Cloud. Set .option("cloudFiles.useNotifications", "true") to allow Auto Loader to automatically set up Google Cloud Pub/Sub resources for you. With file notification mode, new files are detected and ingested as they arrive without listing the input directory. See What is Auto Loader file notification mode?.

CREATE FUNCTION now supports creating table functions

In addition to creating a scalar function that returns a scalar value, you can now create a table function that returns a set of rows. See CREATE FUNCTION (SQL and Python).

Kafka Streaming Source now reports `estimatedTotalBytesBehindLatest` metric

The Kafka streaming source now reports an estimate of how many bytes the consumer is behind the latest available byte after every batch. You can use this metric to track stream progress. See Retrieve Kafka metrics.

Example metric output:

StreamingQueryProgress {
  "batchId": 0,
  .....
  "sources": [ {
    "description" : "KafkaV2[Subscribe[topic-0]]",
    "metrics":{
      "avgOffsetsBehindLatest" : "1.0",
      "estimatedTotalBytesBehindLatest" : "80.0", // new
      "maxOffsetsBehindLatest" : "1",
      "minOffsetsBehindLatest" : "1"
    } ],
  ....
 }

For structs inside of arrays, Delta MERGE INTO now resolves struct fields by name and evolves struct schemas

Delta MERGE INTO now supports resolution of struct fields by name and automatic schema evolution for arrays of structs. When automatic schema evolution is enabled by setting spark.databricks.delta.schema.autoMerge.enabled to true, UPDATE and INSERT clauses will resolve struct fields inside of an array by name, casting to the corresponding data type that is defined in the target array and filling additional or missing fields in the source or target with null values. When automatic schema evolution is disabled, UPDATE and INSERT clauses will resolve struct fields inside of an array by name but will not be able to evolve the additional fields. See Update Delta Lake table schema.

Bug fixes

Fixed a memory leak in the Amazon S3 connector that could happen in long running jobs or services, which was caused by JVM DeleteOnExit functionality.

Library upgrades

Upgraded Python libraries:
- plotly from 4.14.3 to 5.1.0
Upgraded R libraries:
- base from 4.1.0 to 4.1.1
- bslib from 0.2.5.1 to 0.3.0
- cachem from 1.0.5 to 1.0.6
- compiler from 4.1.0 to 4.1.1
- datasets from 4.1.0 to 4.1.1
- future from 1.21.0 to 1.22.1
- gert from 1.3.1 to 1.3.2
- graphics from 4.1.0 to 4.1.1
- grDevices from 4.1.0 to 4.1.1
- grid from 4.1.0 to 4.1.1
Upgraded Java libraries:
- org.eclipse.jetty from 9.4.36.v20210114 to 9.4.42.v20210604

Apache Spark

Databricks Runtime 9.1 LTS includes Apache Spark 3.1.2. This release includes all Spark fixes and improvements included in Databricks Runtime 9.0 (EoS), as well as the following additional bug fixes and improvements made to Spark:

[SPARK-36674][SQL][CHERRY-PICK] Support ILIKE - case insensitive LIKE
[SPARK-36353][SQL][3.1] RemoveNoopOperators should keep output schema
[SPARK-35876][SQL][3.1] ArraysZip should retain field names to avoid being re-written by analyzer/optimizer
[SPARK-36398][SQL] Redact sensitive information in Spark Thrift Server log
[SPARK-36498][SQL] Reorder inner fields of the input query in byName V2 write
[SPARK-36614][CORE][UI] Correct executor loss reason caused by decommission in UI
[SPARK-36012][SQL] Add null flag in SHOW CREATE TABLE
[SPARK-36509][CORE] Fix the issue that executors are never re-scheduled if the worker stops with standalone cluster
[SPARK-36603][CORE] Use WeakReference not SoftReference in LevelDB
[SPARK-36564][CORE] Fix NullPointerException in LiveRDDDistribution.toApi
[SPARK-36086][SQL][3.1] CollapseProject project replace alias should use origin column name
[SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
[SPARK-36400][SPARK-36398][SQL][WEBUI] Make ThriftServer recognize spark.sql.redaction.string.regex
[SPARK-34054][CORE] BlockManagerDecommissioner code cleanup
[SPARK-36500][CORE] Fix temp_shuffle file leaking when a task is interrupted
[SPARK-36489][SQL] Aggregate functions over no grouping keys, on tables with a single bucket, return multiple rows
[SPARK-36464][CORE] Fix Underlying Size Variable Initialization in ChunkedByteBufferOutputStream for Writing Over 2GB Data
[SPARK-36339][SQL][3.0] References to grouping that not part of aggregation should be replaced
[SPARK-36354][CORE] EventLogFileReader should skip rolling event log directories with no logs
[SPARK-36242][CORE][3.1] Ensure spill file closed before set success = true in ExternalSorter.spillMemoryIteratorToDisk method
[SPARK-36211][PYTHON] Correct typing of udf return value
[SPARK-34222][SQL] Enhance boolean simplification rule
[SPARK-35027][CORE] Close the inputStream in FileAppender when writin…
[SPARK-36269][SQL] Fix only set data columns to Hive column names config
[SPARK-36213][SQL] Normalize PartitionSpec for Describe Table Command with PartitionSpec
[SPARK-36210][SQL] Preserve column insertion order in Dataset.withColumns
[SPARK-36079][SQL] Null-based filter estimate should always be in the range [0, 1]
[SPARK-28266][SQL] convertToLogicalRelation should not interpret path property when reading Hive tables

Maintenance updates

See Databricks Runtime 9.1 LTS maintenance updates.

System environment

Operating System: Ubuntu 20.04.4 LTS
Java: Zulu 8.56.0.21-CA-linux64
Scala: 2.12.10
Python: 3.8.10
R: 4.1.1
Delta Lake: 1.0.0

Installed Python libraries

Library	Version	Library	Version	Library	Version
Antergos Linux	2015.10 (ISO-Rolling)	appdirs	1.4.4	backcall	0.2.0
boto3	1.16.7	botocore	1.19.7	certifi	2020.12.5
chardet	4.0.0	cycler	0.10.0	Cython	0.29.23
dbus-python	1.2.16	decorator	5.0.6	distlib	0.3.2
distro-info	0.23ubuntu1	facets-overview	1.0.0	filelock	3.0.12
idna	2.10	ipykernel	5.3.4	ipython	7.22.0
ipython-genutils	0.2.0	jedi	0.17.2	jmespath	0.10.0
joblib	1.0.1	jupyter-client	6.1.12	jupyter-core	4.7.1
kiwisolver	1.3.1	koalas	1.8.1	matplotlib	3.4.2
numpy	1.19.2	pandas	1.2.4	parso	0.7.0
patsy	0.5.1	pexpect	4.8.0	pickleshare	0.7.5
Pillow	8.2.0	pip	21.0.1	plotly	5.1.0
prompt-toolkit	3.0.17	protobuf	3.17.2	psycopg2	2.8.5
ptyprocess	0.7.0	pyarrow	4.0.0	Pygments	2.8.1
PyGObject	3.36.0	pyparsing	2.4.7	python-apt	2.0.0+ubuntu0.20.4.5
python-dateutil	2.8.1	pytz	2020.5	pyzmq	20.0.0
requests	2.25.1	requests-unixsocket	0.2.0	s3transfer	0.3.7
scikit-learn	0.24.1	scipy	1.6.2	seaborn	0.11.1
setuptools	52.0.0	six	1.15.0	ssh-import-id	5.10
statsmodels	0.12.2	tenacity	8.0.1	threadpoolctl	2.1.0
tornado	6.1	traitlets	5.0.5	unattended-upgrades	0.1
urllib3	1.25.11	virtualenv	20.4.1	wcwidth	0.2.5
wheel	0.36.2

Installed R libraries

R libraries are installed from the Microsoft CRAN snapshot on 2021-09-08.

Library	Version	Library	Version	Library	Version
askpass	1.1	assertthat	0.2.1	backports	1.2.1
base	4.1.1	base64enc	0.1-3	bit	4.0.4
bit64	4.0.5	blob	1.2.2	boot	1.3-28
brew	1.0-6	brio	1.1.2	broom	0.7.9
bslib	0.3.0	cachem	1.0.6	callr	3.7.0
caret	6.0-88	cellranger	1.1.0	chron	2.3-56
class	7.3-19	cli	3.0.1	clipr	0.7.1
cluster	2.1.2	codetools	0.2-18	colorspace	2.0-2
commonmark	1.7	compiler	4.1.1	config	0.3.1
cpp11	0.3.1	crayon	1.4.1	credentials	1.3.1
curl	4.3.2	data.table	1.14.0	datasets	4.1.1
DBI	1.1.1	dbplyr	2.1.1	desc	1.3.0
devtools	2.4.2	diffobj	0.3.4	digest	0.6.27
dplyr	1.0.7	dtplyr	1.1.0	ellipsis	0.3.2
evaluate	0.14	fansi	0.5.0	farver	2.1.0
fastmap	1.1.0	forcats	0.5.1	foreach	1.5.1
foreign	0.8-81	forge	0.2.0	fs	1.5.0
future	1.22.1	future.apply	1.8.1	gargle	1.2.0
generics	0.1.0	gert	1.3.2	ggplot2	3.3.5
gh	1.3.0	gitcreds	0.1.1	glmnet	4.1-2
globals	0.14.0	glue	1.4.2	googledrive	2.0.0
googlesheets4	1.0.0	gower	0.2.2	graphics	4.1.1
grDevices	4.1.1	grid	4.1.1	gridExtra	2.3
gsubfn	0.7	gtable	0.3.0	haven	2.4.3
highr	0.9	hms	1.1.0	htmltools	0.5.2
htmlwidgets	1.5.3	httpuv	1.6.2	httr	1.4.2
hwriter	1.3.2	hwriterPlus	1.0-3	ids	1.0.1
ini	0.3.1	ipred	0.9-11	isoband	0.2.5
iterators	1.0.13	jquerylib	0.1.4	jsonlite	1.7.2
KernSmooth	2.23-20	knitr	1.33	labeling	0.4.2
later	1.3.0	lattice	0.20-44	lava	1.6.10
lifecycle	1.0.0	listenv	0.8.0	lubridate	1.7.10
magrittr	2.0.1	markdown	1.1	MASS	7.3-54
Matrix	1.3-4	memoise	2.0.0	methods	4.1.1
mgcv	1.8-36	mime	0.11	ModelMetrics	1.2.2.2
modelr	0.1.8	munsell	0.5.0	nlme	3.1-152
nnet	7.3-16	numDeriv	2016.8-1.1	openssl	1.4.5
parallel	4.1.1	parallelly	1.27.0	pillar	1.6.2
pkgbuild	1.2.0	pkgconfig	2.0.3	pkgload	1.2.1
plogr	0.2.0	plyr	1.8.6	praise	1.0.0
prettyunits	1.1.1	pROC	1.18.0	processx	3.5.2
prodlim	2019.11.13	progress	1.2.2	progressr	0.8.0
promises	1.2.0.1	proto	1.0.0	ps	1.6.0
purrr	0.3.4	r2d3	0.2.5	R6	2.5.1
randomForest	4.6-14	rappdirs	0.3.3	rcmdcheck	1.3.3
RColorBrewer	1.1-2	Rcpp	1.0.7	readr	2.0.1
readxl	1.3.1	recipes	0.1.16	rematch	1.0.1
rematch2	2.1.2	remotes	2.4.0	reprex	2.0.1
reshape2	1.4.4	rlang	0.4.11	rmarkdown	2.10
RODBC	1.3-18	roxygen2	7.1.1	rpart	4.1-15
rprojroot	2.0.2	Rserve	1.8-8	RSQLite	2.2.8
rstudioapi	0.13	rversions	2.1.1	rvest	1.0.1
sass	0.4.0	scales	1.1.1	selectr	0.4-2
sessioninfo	1.1.1	shape	1.4.6	shiny	1.6.0
sourcetools	0.1.7	sparklyr	1.7.1	SparkR	3.1.1
spatial	7.3-11	splines	4.1.1	sqldf	0.4-11
SQUAREM	2021.1	stats	4.1.1	stats4	4.1.1
stringi	1.7.4	stringr	1.4.0	survival	3.2-13
sys	3.4	tcltk	4.1.1	TeachingDemos	2.10
testthat	3.0.4	tibble	3.1.4	tidyr	1.1.3
tidyselect	1.1.1	tidyverse	1.3.1	timeDate	3043.102
tinytex	0.33	tools	4.1.1	tzdb	0.1.2
usethis	2.0.1	utf8	1.2.2	utils	4.1.1
uuid	0.1-4	vctrs	0.3.8	viridisLite	0.4.0
vroom	1.5.4	waldo	0.3.0	whisker	0.4
withr	2.4.2	xfun	0.25	xml2	1.3.2
xopen	1.0.0	xtable	1.8-4	yaml	2.2.1
zip	2.2.0

Installed Java and Scala libraries (Scala 2.12 cluster version)

Group ID	Artifact ID	Version
antlr	antlr	2.7.7
com.amazonaws	amazon-kinesis-client	1.12.0
com.amazonaws	aws-java-sdk-autoscaling	1.11.655
com.amazonaws	aws-java-sdk-cloudformation	1.11.655
com.amazonaws	aws-java-sdk-cloudfront	1.11.655
com.amazonaws	aws-java-sdk-cloudhsm	1.11.655
com.amazonaws	aws-java-sdk-cloudsearch	1.11.655
com.amazonaws	aws-java-sdk-cloudtrail	1.11.655
com.amazonaws	aws-java-sdk-cloudwatch	1.11.655
com.amazonaws	aws-java-sdk-cloudwatchmetrics	1.11.655
com.amazonaws	aws-java-sdk-codedeploy	1.11.655
com.amazonaws	aws-java-sdk-cognitoidentity	1.11.655
com.amazonaws	aws-java-sdk-cognitosync	1.11.655
com.amazonaws	aws-java-sdk-config	1.11.655
com.amazonaws	aws-java-sdk-core	1.11.655
com.amazonaws	aws-java-sdk-datapipeline	1.11.655
com.amazonaws	aws-java-sdk-directconnect	1.11.655
com.amazonaws	aws-java-sdk-directory	1.11.655
com.amazonaws	aws-java-sdk-dynamodb	1.11.655
com.amazonaws	aws-java-sdk-ec2	1.11.655
com.amazonaws	aws-java-sdk-ecs	1.11.655
com.amazonaws	aws-java-sdk-efs	1.11.655
com.amazonaws	aws-java-sdk-elasticache	1.11.655
com.amazonaws	aws-java-sdk-elasticbeanstalk	1.11.655
com.amazonaws	aws-java-sdk-elasticloadbalancing	1.11.655
com.amazonaws	aws-java-sdk-elastictranscoder	1.11.655
com.amazonaws	aws-java-sdk-emr	1.11.655
com.amazonaws	aws-java-sdk-glacier	1.11.655
com.amazonaws	aws-java-sdk-glue	1.11.655
com.amazonaws	aws-java-sdk-iam	1.11.655
com.amazonaws	aws-java-sdk-importexport	1.11.655
com.amazonaws	aws-java-sdk-kinesis	1.11.655
com.amazonaws	aws-java-sdk-kms	1.11.655
com.amazonaws	aws-java-sdk-lambda	1.11.655
com.amazonaws	aws-java-sdk-logs	1.11.655
com.amazonaws	aws-java-sdk-machinelearning	1.11.655
com.amazonaws	aws-java-sdk-marketplacecommerceanalytics	1.11.655
com.amazonaws	aws-java-sdk-marketplacemeteringservice	1.11.655
com.amazonaws	aws-java-sdk-opsworks	1.11.655
com.amazonaws	aws-java-sdk-rds	1.11.655
com.amazonaws	aws-java-sdk-redshift	1.11.655
com.amazonaws	aws-java-sdk-route53	1.11.655
com.amazonaws	aws-java-sdk-s3	1.11.655
com.amazonaws	aws-java-sdk-ses	1.11.655
com.amazonaws	aws-java-sdk-simpledb	1.11.655
com.amazonaws	aws-java-sdk-simpleworkflow	1.11.655
com.amazonaws	aws-java-sdk-sns	1.11.655
com.amazonaws	aws-java-sdk-sqs	1.11.655
com.amazonaws	aws-java-sdk-ssm	1.11.655
com.amazonaws	aws-java-sdk-storagegateway	1.11.655
com.amazonaws	aws-java-sdk-sts	1.11.655
com.amazonaws	aws-java-sdk-support	1.11.655
com.amazonaws	aws-java-sdk-swf-libraries	1.11.22
com.amazonaws	aws-java-sdk-workspaces	1.11.655
com.amazonaws	jmespath-java	1.11.655
com.chuusai	shapeless_2.12	2.3.3
com.clearspring.analytics	stream	2.9.6
com.databricks	Rserve	1.8-3
com.databricks	jets3t	0.7.1-0
com.databricks.scalapb	compilerplugin_2.12	0.4.15-10
com.databricks.scalapb	scalapb-runtime_2.12	0.4.15-10
com.esotericsoftware	kryo-shaded	4.0.2
com.esotericsoftware	minlog	1.3.0
com.fasterxml	classmate	1.3.4
com.fasterxml.jackson.core	jackson-annotations	2.10.0
com.fasterxml.jackson.core	jackson-core	2.10.0
com.fasterxml.jackson.core	jackson-databind	2.10.0
com.fasterxml.jackson.dataformat	jackson-dataformat-cbor	2.10.0
com.fasterxml.jackson.datatype	jackson-datatype-joda	2.10.0
com.fasterxml.jackson.module	jackson-module-paranamer	2.10.0
com.fasterxml.jackson.module	jackson-module-scala_2.12	2.10.0
com.github.ben-manes.caffeine	caffeine	2.3.4
com.github.fommil	jniloader	1.1
com.github.fommil.netlib	core	1.1.2
com.github.fommil.netlib	native_ref-java	1.1
com.github.fommil.netlib	native_ref-java-natives	1.1
com.github.fommil.netlib	native_system-java	1.1
com.github.fommil.netlib	native_system-java-natives	1.1
com.github.fommil.netlib	netlib-native_ref-linux-x86_64-natives	1.1
com.github.fommil.netlib	netlib-native_system-linux-x86_64-natives	1.1
com.github.joshelser	dropwizard-metrics-hadoop-metrics2-reporter	0.1.2
com.github.luben	zstd-jni	1.4.8-1
com.github.wendykierp	JTransforms	3.1
com.google.code.findbugs	jsr305	3.0.0
com.google.code.gson	gson	2.2.4
com.google.crypto.tink	tink	1.6.0
com.google.flatbuffers	flatbuffers-java	1.9.0
com.google.guava	guava	15.0
com.google.protobuf	protobuf-java	2.6.1
com.h2database	h2	1.4.195
com.helger	profiler	1.1.1
com.jcraft	jsch	0.1.50
com.jolbox	bonecp	0.8.0.RELEASE
com.lihaoyi	sourcecode_2.12	0.1.9
com.microsoft.azure	azure-data-lake-store-sdk	2.3.9
com.microsoft.sqlserver	mssql-jdbc	9.2.1.jre8
com.ning	compress-lzf	1.0.3
com.sun.mail	javax.mail	1.5.2
com.tdunning	json	1.8
com.thoughtworks.paranamer	paranamer	2.8
com.trueaccord.lenses	lenses_2.12	0.4.12
com.twitter	chill-java	0.9.5
com.twitter	chill_2.12	0.9.5
com.twitter	util-app_2.12	7.1.0
com.twitter	util-core_2.12	7.1.0
com.twitter	util-function_2.12	7.1.0
com.twitter	util-jvm_2.12	7.1.0
com.twitter	util-lint_2.12	7.1.0
com.twitter	util-registry_2.12	7.1.0
com.twitter	util-stats_2.12	7.1.0
com.typesafe	config	1.2.1
com.typesafe.scala-logging	scala-logging_2.12	3.7.2
com.univocity	univocity-parsers	2.9.1
com.zaxxer	HikariCP	3.1.0
commons-beanutils	commons-beanutils	1.9.4
commons-cli	commons-cli	1.2
commons-codec	commons-codec	1.10
commons-collections	commons-collections	3.2.2
commons-configuration	commons-configuration	1.6
commons-dbcp	commons-dbcp	1.4
commons-digester	commons-digester	1.8
commons-fileupload	commons-fileupload	1.3.3
commons-httpclient	commons-httpclient	3.1
commons-io	commons-io	2.4
commons-lang	commons-lang	2.6
commons-logging	commons-logging	1.1.3
commons-net	commons-net	3.1
commons-pool	commons-pool	1.5.4
hive-2.3__hadoop-2.7	jets3t-0.7	liball_deps_2.12
hive-2.3__hadoop-2.7	zookeeper-3.4	liball_deps_2.12
info.ganglia.gmetric4j	gmetric4j	1.0.10
io.airlift	aircompressor	0.10
io.delta	delta-sharing-spark_2.12	0.1.0
io.dropwizard.metrics	metrics-core	4.1.1
io.dropwizard.metrics	metrics-graphite	4.1.1
io.dropwizard.metrics	metrics-healthchecks	4.1.1
io.dropwizard.metrics	metrics-jetty9	4.1.1
io.dropwizard.metrics	metrics-jmx	4.1.1
io.dropwizard.metrics	metrics-json	4.1.1
io.dropwizard.metrics	metrics-jvm	4.1.1
io.dropwizard.metrics	metrics-servlets	4.1.1
io.netty	netty-all	4.1.51.Final
io.prometheus	simpleclient	0.7.0
io.prometheus	simpleclient_common	0.7.0
io.prometheus	simpleclient_dropwizard	0.7.0
io.prometheus	simpleclient_pushgateway	0.7.0
io.prometheus	simpleclient_servlet	0.7.0
io.prometheus.jmx	collector	0.12.0
jakarta.annotation	jakarta.annotation-api	1.3.5
jakarta.validation	jakarta.validation-api	2.0.2
jakarta.ws.rs	jakarta.ws.rs-api	2.1.6
javax.activation	activation	1.1.1
javax.el	javax.el-api	2.2.4
javax.jdo	jdo-api	3.0.1
javax.servlet	javax.servlet-api	3.1.0
javax.servlet.jsp	jsp-api	2.1
javax.transaction	jta	1.1
javax.transaction	transaction-api	1.1
javax.xml.bind	jaxb-api	2.2.2
javax.xml.stream	stax-api	1.0-2
javolution	javolution	5.5.1
jline	jline	2.14.6
joda-time	joda-time	2.10.5
log4j	apache-log4j-extras	1.2.17
log4j	log4j	1.2.17
maven-trees	hive-2.3__hadoop-2.7	liball_deps_2.12
net.java.dev.jna	jna	5.8.0
net.razorvine	pyrolite	4.30
net.sf.jpam	jpam	1.1
net.sf.opencsv	opencsv	2.3
net.sf.supercsv	super-csv	2.2.0
net.snowflake	snowflake-ingest-sdk	0.9.6
net.snowflake	snowflake-jdbc	3.13.3
net.snowflake	spark-snowflake_2.12	2.9.0-spark_3.1
net.sourceforge.f2j	arpack_combined_all	0.1
org.acplt.remotetea	remotetea-oncrpc	1.1.2
org.antlr	ST4	4.0.4
org.antlr	antlr-runtime	3.5.2
org.antlr	antlr4-runtime	4.8-1
org.antlr	stringtemplate	3.2.1
org.apache.ant	ant	1.9.2
org.apache.ant	ant-jsch	1.9.2
org.apache.ant	ant-launcher	1.9.2
org.apache.arrow	arrow-format	2.0.0
org.apache.arrow	arrow-memory-core	2.0.0
org.apache.arrow	arrow-memory-netty	2.0.0
org.apache.arrow	arrow-vector	2.0.0
org.apache.avro	avro	1.8.2
org.apache.avro	avro-ipc	1.8.2
org.apache.avro	avro-mapred-hadoop2	1.8.2
org.apache.commons	commons-compress	1.20
org.apache.commons	commons-crypto	1.1.0
org.apache.commons	commons-lang3	3.10
org.apache.commons	commons-math3	3.4.1
org.apache.commons	commons-text	1.6
org.apache.curator	curator-client	2.7.1
org.apache.curator	curator-framework	2.7.1
org.apache.curator	curator-recipes	2.7.1
org.apache.derby	derby	10.12.1.1
org.apache.directory.api	api-asn1-api	1.0.0-M20
org.apache.directory.api	api-util	1.0.0-M20
org.apache.directory.server	apacheds-i18n	2.0.0-M15
org.apache.directory.server	apacheds-kerberos-codec	2.0.0-M15
org.apache.hadoop	hadoop-annotations	2.7.4
org.apache.hadoop	hadoop-auth	2.7.4
org.apache.hadoop	hadoop-client	2.7.4
org.apache.hadoop	hadoop-common	2.7.4
org.apache.hadoop	hadoop-hdfs	2.7.4
org.apache.hadoop	hadoop-mapreduce-client-app	2.7.4
org.apache.hadoop	hadoop-mapreduce-client-common	2.7.4
org.apache.hadoop	hadoop-mapreduce-client-core	2.7.4
org.apache.hadoop	hadoop-mapreduce-client-jobclient	2.7.4
org.apache.hadoop	hadoop-mapreduce-client-shuffle	2.7.4
org.apache.hadoop	hadoop-yarn-api	2.7.4
org.apache.hadoop	hadoop-yarn-client	2.7.4
org.apache.hadoop	hadoop-yarn-common	2.7.4
org.apache.hadoop	hadoop-yarn-server-common	2.7.4
org.apache.hive	hive-beeline	2.3.7
org.apache.hive	hive-cli	2.3.7
org.apache.hive	hive-jdbc	2.3.7
org.apache.hive	hive-llap-client	2.3.7
org.apache.hive	hive-llap-common	2.3.7
org.apache.hive	hive-serde	2.3.7
org.apache.hive	hive-shims	2.3.7
org.apache.hive	hive-storage-api	2.7.2
org.apache.hive.shims	hive-shims-0.23	2.3.7
org.apache.hive.shims	hive-shims-common	2.3.7
org.apache.hive.shims	hive-shims-scheduler	2.3.7
org.apache.htrace	htrace-core	3.1.0-incubating
org.apache.httpcomponents	httpclient	4.5.6
org.apache.httpcomponents	httpcore	4.4.12
org.apache.ivy	ivy	2.4.0
org.apache.mesos	mesos-shaded-protobuf	1.4.0
org.apache.orc	orc-core	1.5.12
org.apache.orc	orc-mapreduce	1.5.12
org.apache.orc	orc-shims	1.5.12
org.apache.parquet	parquet-column	1.10.1-databricks9
org.apache.parquet	parquet-common	1.10.1-databricks9
org.apache.parquet	parquet-encoding	1.10.1-databricks9
org.apache.parquet	parquet-format	2.4.0
org.apache.parquet	parquet-hadoop	1.10.1-databricks9
org.apache.parquet	parquet-jackson	1.10.1-databricks9
org.apache.thrift	libfb303	0.9.3
org.apache.thrift	libthrift	0.12.0
org.apache.xbean	xbean-asm7-shaded	4.15
org.apache.yetus	audience-annotations	0.5.0
org.apache.zookeeper	zookeeper	3.4.14
org.codehaus.jackson	jackson-core-asl	1.9.13
org.codehaus.jackson	jackson-jaxrs	1.9.13
org.codehaus.jackson	jackson-mapper-asl	1.9.13
org.codehaus.jackson	jackson-xc	1.9.13
org.codehaus.janino	commons-compiler	3.0.16
org.codehaus.janino	janino	3.0.16
org.datanucleus	datanucleus-api-jdo	4.2.4
org.datanucleus	datanucleus-core	4.1.17
org.datanucleus	datanucleus-rdbms	4.1.19
org.datanucleus	javax.jdo	3.2.0-m3
org.eclipse.jetty	jetty-client	9.4.42.v20210604
org.eclipse.jetty	jetty-continuation	9.4.42.v20210604
org.eclipse.jetty	jetty-http	9.4.42.v20210604
org.eclipse.jetty	jetty-io	9.4.42.v20210604
org.eclipse.jetty	jetty-jndi	9.4.42.v20210604
org.eclipse.jetty	jetty-plus	9.4.42.v20210604
org.eclipse.jetty	jetty-proxy	9.4.42.v20210604
org.eclipse.jetty	jetty-security	9.4.42.v20210604
org.eclipse.jetty	jetty-server	9.4.42.v20210604
org.eclipse.jetty	jetty-servlet	9.4.42.v20210604
org.eclipse.jetty	jetty-servlets	9.4.42.v20210604
org.eclipse.jetty	jetty-util	9.4.42.v20210604
org.eclipse.jetty	jetty-util-ajax	9.4.42.v20210604
org.eclipse.jetty	jetty-webapp	9.4.42.v20210604
org.eclipse.jetty	jetty-xml	9.4.42.v20210604
org.eclipse.jetty.websocket	websocket-api	9.4.42.v20210604
org.eclipse.jetty.websocket	websocket-client	9.4.42.v20210604
org.eclipse.jetty.websocket	websocket-common	9.4.42.v20210604
org.eclipse.jetty.websocket	websocket-server	9.4.42.v20210604
org.eclipse.jetty.websocket	websocket-servlet	9.4.42.v20210604
org.fusesource.leveldbjni	leveldbjni-all	1.8
org.glassfish.hk2	hk2-api	2.6.1
org.glassfish.hk2	hk2-locator	2.6.1
org.glassfish.hk2	hk2-utils	2.6.1
org.glassfish.hk2	osgi-resource-locator	1.0.3
org.glassfish.hk2.external	aopalliance-repackaged	2.6.1
org.glassfish.hk2.external	jakarta.inject	2.6.1
org.glassfish.jersey.containers	jersey-container-servlet	2.30
org.glassfish.jersey.containers	jersey-container-servlet-core	2.30
org.glassfish.jersey.core	jersey-client	2.30
org.glassfish.jersey.core	jersey-common	2.30
org.glassfish.jersey.core	jersey-server	2.30
org.glassfish.jersey.inject	jersey-hk2	2.30
org.glassfish.jersey.media	jersey-media-jaxb	2.30
org.hibernate.validator	hibernate-validator	6.1.0.Final
org.javassist	javassist	3.25.0-GA
org.jboss.logging	jboss-logging	3.3.2.Final
org.jdbi	jdbi	2.63.1
org.joda	joda-convert	1.7
org.jodd	jodd-core	3.5.2
org.json4s	json4s-ast_2.12	3.7.0-M5
org.json4s	json4s-core_2.12	3.7.0-M5
org.json4s	json4s-jackson_2.12	3.7.0-M5
org.json4s	json4s-scalap_2.12	3.7.0-M5
org.lz4	lz4-java	1.7.1
org.mariadb.jdbc	mariadb-java-client	2.2.5
org.objenesis	objenesis	2.5.1
org.postgresql	postgresql	42.1.4
org.roaringbitmap	RoaringBitmap	0.9.14
org.roaringbitmap	shims	0.9.14
org.rocksdb	rocksdbjni	6.20.3
org.rosuda.REngine	REngine	2.1.0
org.scala-lang	scala-compiler_2.12	2.12.10
org.scala-lang	scala-library_2.12	2.12.10
org.scala-lang	scala-reflect_2.12	2.12.10
org.scala-lang.modules	scala-collection-compat_2.12	2.1.1
org.scala-lang.modules	scala-parser-combinators_2.12	1.1.2
org.scala-lang.modules	scala-xml_2.12	1.2.0
org.scala-sbt	test-interface	1.0
org.scalacheck	scalacheck_2.12	1.14.2
org.scalactic	scalactic_2.12	3.0.8
org.scalanlp	breeze-macros_2.12	1.0
org.scalanlp	breeze_2.12	1.0
org.scalatest	scalatest_2.12	3.0.8
org.slf4j	jcl-over-slf4j	1.7.30
org.slf4j	jul-to-slf4j	1.7.30
org.slf4j	slf4j-api	1.7.30
org.slf4j	slf4j-log4j12	1.7.30
org.spark-project.spark	unused	1.0.0
org.springframework	spring-core	4.1.4.RELEASE
org.springframework	spring-test	4.1.4.RELEASE
org.threeten	threeten-extra	1.5.0
org.tukaani	xz	1.5
org.typelevel	algebra_2.12	2.0.0-M2
org.typelevel	cats-kernel_2.12	2.0.0-M4
org.typelevel	machinist_2.12	0.6.8
org.typelevel	macro-compat_2.12	1.1.1
org.typelevel	spire-macros_2.12	0.17.0-M1
org.typelevel	spire-platform_2.12	0.17.0-M1
org.typelevel	spire-util_2.12	0.17.0-M1
org.typelevel	spire_2.12	0.17.0-M1
org.wildfly.openssl	wildfly-openssl	1.0.7.Final
org.xerial	sqlite-jdbc	3.8.11.2
org.xerial.snappy	snappy-java	1.1.8.2
org.yaml	snakeyaml	1.24
oro	oro	2.0.8
pl.edu.icm	JLargeArrays	1.5
software.amazon.ion	ion-java	1.0.2
stax	stax-api	1.0.1
xmlenc	xmlenc	0.52

Jaa

Databricks Runtime 9.1 LTS

New features and improvements

Auto Loader schema hints now work with array and map types

Avro file support for merge schema

Auto Loader incremental listing support (Public Preview)

Delta now supports arbitrary replaceWhere

Auto Loader for Google Cloud now supports file notifications (Public Preview)

CREATE FUNCTION now supports creating table functions

Kafka Streaming Source now reports `estimatedTotalBytesBehindLatest` metric

For structs inside of arrays, Delta MERGE INTO now resolves struct fields by name and evolves struct schemas

Bug fixes

Library upgrades

Apache Spark

Maintenance updates

System environment

Installed Python libraries

Installed R libraries

Installed Java and Scala libraries (Scala 2.12 cluster version)

Palaute

Lisäresursseja

Jaa

Databricks Runtime 9.1 LTS

New features and improvements

Auto Loader schema hints now work with array and map types

Avro file support for merge schema

Auto Loader incremental listing support (Public Preview)

Delta now supports arbitrary replaceWhere

Auto Loader for Google Cloud now supports file notifications (Public Preview)

CREATE FUNCTION now supports creating table functions

Kafka Streaming Source now reports estimatedTotalBytesBehindLatest metric

For structs inside of arrays, Delta MERGE INTO now resolves struct fields by name and evolves struct schemas

Bug fixes

Library upgrades

Apache Spark

Maintenance updates

System environment

Installed Python libraries

Installed R libraries

Installed Java and Scala libraries (Scala 2.12 cluster version)

Palaute

Lisäresursseja

Kafka Streaming Source now reports `estimatedTotalBytesBehindLatest` metric