Databricks Runtime 8.1 (EoS)

Άρθρο
09/03/2024

Note

Support for this Databricks Runtime version has ended. For the end-of-support date, see End-of-support history. For all supported Databricks Runtime versions, see Databricks Runtime release notes versions and compatibility.

Databricks released this version in March 2021.

The following release notes provide information about Databricks Runtime 8.1, powered by Apache Spark 3.1.1.

New features and improvements

New SQL operators reduce complex ETL on JSON strings
Control visibility of table access control lists
New Auto Loader options enable better performance
Authenticate to Azure Synapse Analytics with OAuth 2.0 and a service principal (Public Preview)
Manually insert IDENTITY columns in Azure Synapse Analytics tables
Azure Auto Loader now supports authentication with a service principal for queue operations
Track stream progress with Kafka metrics
Track stream progress with Delta Lake streaming source metrics

New SQL operators reduce complex ETL on JSON strings

Two new SQL operators—:: and :—allow you to easily extract data from JSON strings.

To cast your expression to a data type, use a double colon ::. For example, instead of specifying cast(col as int), you can now say col::int.
To extract fields from semi-structured data, use a single colon :. If you have a STRING column containing JSON data, you can extract a field from that JSON string using the syntax <json-column>:<field> which extracts the key field from your JSON <json-column> as a STRING. You can combine this with the :: syntax to cast the data to a specific type.

For example, if you have the JSON data:

{
  "store":{
    "bicycle":{
      "price":19.95,
      "color":"red"
    }
  }
}

and you create a table store_data from the data:

CREATE TABLE store_data AS SELECT
'{
   "store":{
      "bicycle":{
        "price":19.95,
        "color":"red"
      }
    }
 }' as raw

the statement SELECT raw:store.bicycle.price::double FROM store_data returns:

+--------+
| price  |
+--------+
| 19.95  |
+--------+

For details, see Query JSON strings.

Control visibility of table access control lists

To avoid leaking information by listing all defined permissions on a table, you can now run SHOW GRANTS [<user>] <object> statement only if you are:

A Databricks SQL administrator or the owner of <object>.
The user specified in [<user>].

New Auto Loader options enable better performance

Auto Loader now supports a new option cloudFiles.fetchParallelism to specify the parallelism to fetch messages from the queue service.

See What is Auto Loader?.

Authenticate to Azure Synapse Analytics with OAuth 2.0 and a service principal (Public Preview)

The Azure Synapse Analytics connector now supports authentication with OAuth 2.0 and a Microsoft Entra ID service principal, enabling applications that require authentication with a service principal instead of a username and password. To learn more, see How does authentication between Azure Databricks and Synapse work? for the Azure Synapse Analytics connector.

Manually insert `IDENTITY` columns in Azure Synapse Analytics tables

The Azure Synapse Analytics connector now supports explicitly inserting values into IDENTITY columns in Azure Synapse Analytics tables based on values set in DataFrames. This capability is supported for batch and streaming inserts and enables use cases requiring custom IDENTITY column values. For details, see the identityInsert parameter in the Azure Synapse Analytics connector Azure Databricks Synapse connector options reference.

Azure Auto Loader now supports authentication with a service principal for queue operations

You can now authenticate with a Microsoft Entra ID service principal granted queue-related permissions for queue operations with Auto Loader. This change removes the previous requirement of using a connection string for queue operations and a service principal for event operations and allows you to use a service principal for both operation types. For more details, see What is Auto Loader?.

Track stream progress with Kafka metrics

The Kafka streaming source now reports how many offsets a consumer is falling behind the latest after every batch. You can use these metrics to track stream progress. For details on how to view the metrics, see Retrieve Kafka metrics.

Track stream progress with Delta Lake streaming source metrics

A Delta Lake streaming source now reports how large the backlog is and how many files exist in the backlog after every batch. You can use these metrics to track stream progress. For details on how to view the metrics, see Metrics.

Library upgrades

Upgraded Python library:
- koalas upgraded from 1.5.0 to 1.6.0.
Newly installed Python libraries:
- plotly 4.14.3
- retrying 1.3.3
Upgraded R to 4.0.4 from 4.0.3.
Upgraded several installed R libraries. See Installed R Libraries.
Upgraded Java library:
- jetty from 9.4.34.v20201102 to 9.4.36.v20210114
OpenJDK 8 build upgraded to Zulu 8.52.0.23-CA-linux64 (build 1.8.0_282-b08)

Apache Spark

Databricks Runtime 8.1 includes Apache Spark 3.1.1. This release includes all Spark fixes and improvements included in Databricks Runtime 8.0 (EoS), as well as the following additional bug fixes and improvements made to Spark:

[SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks
[SPARK-34613] [SQL] Fix view does not capture disable hint config
[SPARK-34392] [SQL] Support ZoneOffset +h:mm in DateTimeUtils. getZoneId
[SPARK-34436] [SQL] DPP support LIKE ANY/ALL expression
[SPARK-34497] [SQL] Fix built-in JDBC connection providers to restore JVM security context changes
[SPARK-34543] [SQL] Respect the spark.sql.caseSensitive config while resolving partition spec in v1 SET LOCATION
[SPARK-34550] [SQL] Skip InSet null value during push filter to Hive metastore
[SPARK-34531] [CORE] Remove Experimental API tag in PrometheusServlet

Maintenance updates

See Databricks Runtime 8.1 maintenance updates.

System environment

Operating System: Ubuntu 18.04.5 LTS
Java: Zulu 8.52.0.23-CA-linux64 (build 1.8.0_282-b08)
Scala: 2.12.10
Python: 3.8.8 (updated from 3.8.6 in May 26, 2021 maintenance update)
R: R version 4.0.4 (2021-02-15)
Delta Lake 0.8.0

Installed Python libraries

Library	Version	Library	Version	Library	Version
appdirs	1.4.4	asn1crypto	1.4.0	backcall	0.2.0
boto3	1.16.7	botocore	1.19.7	brotlipy	0.7.0
certifi	2020.12.5	cffi	1.14.3	chardet	3.0.4
cryptography	3.1.1	cycler	0.10.0	Cython	0.29.21
decorator	4.4.2	distlib	0.3.1	docutils	0.15.2
entrypoints	0.3	filelock	3.0.12	idna	2.10
ipykernel	5.3.4	ipython	7.19.0	ipython-genutils	0.2.0
jedi	0.17.2	jmespath	0.10.0	joblib	0.17.0
jupyter-client	6.1.7	jupyter-core	4.6.3	kiwisolver	1.3.0
koalas	1.6.0	matplotlib	3.2.2	numpy	1.19.2
pandas	1.1.3	parso	0.7.0	patsy	0.5.1
pexpect	4.8.0	pickleshare	0.7.5	pip	20.2.4
plotly	4.14.3	prompt-toolkit	3.0.8	psycopg2	2.8.5
ptyprocess	0.6.0	pyarrow	1.0.1	pycparser	2.20
Pygments	2.7.2	pyOpenSSL	19.1.0	pyparsing	2.4.7
PySocks	1.7.1	python-dateutil	2.8.1	pytz	2020.5
pyzmq	19.0.2	requests	2.24.0	retrying	1.3.3
s3transfer	0.3.4	scikit-learn	0.23.2	scipy	1.5.2
seaborn	0.10.0	setuptools	50.3.1	six	1.15.0
statsmodels	0.12.0	threadpoolctl	2.1.0	tornado	6.0.4
traitlets	5.0.5	urllib3	1.25.11	virtualenv	20.2.1
wcwidth	0.2.5	wheel	0.35.1

Installed R libraries

R libraries are installed from the Microsoft CRAN snapshot on 2020-11-02.

Library	Version	Library	Version	Library	Version
askpass	1.1	assertthat	0.2.1	backports	1.2.1
base	4.0.4	base64enc	0.1-3	BH	1.72.0-3
bit	4.0.4	bit64	4.0.5	blob	1.2.1
boot	1.3-27	brew	1.0-6	brio	1.1.0
broom	0.7.2	callr	3.5.1	caret	6.0-86
cellranger	1.1.0	chron	2.3-56	class	7.3-18
cli	2.2.0	clipr	0.7.1	cluster	2.1.1
codetools	0.2-18	colorspace	2.0-0	commonmark	1.7
compiler	4.0.4	config	0.3	covr	3.5.1
cpp11	0.2.4	crayon	1.3.4	credentials	1.3.0
crosstalk	1.1.0.1	curl	4.3	data.table	1.13.4
datasets	4.0.4	DBI	1.1.0	dbplyr	2.0.0
desc	1.2.0	devtools	2.3.2	diffobj	0.3.2
digest	0.6.27	dplyr	1.0.2	DT	0.16
ellipsis	0.3.1	evaluate	0.14	fansi	0.4.1
farver	2.0.3	fastmap	1.0.1	forcats	0.5.0
foreach	1.5.1	foreign	0.8-81	forge	0.2.0
fs	1.5.0	future	1.21.0	generics	0.1.0
gert	1.0.2	ggplot2	3.3.2	gh	1.2.0
gitcreds	0.1.1	glmnet	4.0-2	globals	0.14.0
glue	1.4.2	gower	0.2.2	graphics	4.0.4
grDevices	4.0.4	grid	4.0.4	gridExtra	2.3
gsubfn	0.7	gtable	0.3.0	haven	2.3.1
highr	0.8	hms	0.5.3	htmltools	0.5.0
htmlwidgets	1.5.3	httpuv	1.5.4	httr	1.4.2
hwriter	1.3.2	hwriterPlus	1.0-3	ini	0.3.1
ipred	0.9-9	isoband	0.2.3	iterators	1.0.13
jsonlite	1.7.2	KernSmooth	2.23-18	knitr	1.30
labeling	0.4.2	later	1.1.0.1	lattice	0.20-41
lava	1.6.8.1	lazyeval	0.2.2	lifecycle	0.2.0
listenv	0.8.0	lubridate	1.7.9.2	magrittr	2.0.1
markdown	1.1	MASS	7.3-53.1	Matrix	1.3-2
memoise	1.1.0	methods	4.0.4	mgcv	1.8-33
mime	0.9	ModelMetrics	1.2.2.2	modelr	0.1.8
munsell	0.5.0	nlme	3.1-152	nnet	7.3-15
numDeriv	2016.8-1.1	openssl	1.4.3	parallel	4.0.4
parallelly	1.22.0	pillar	1.4.7	pkgbuild	1.1.0
pkgconfig	2.0.3	pkgload	1.1.0	plogr	0.2.0
plyr	1.8.6	praise	1.0.0	prettyunits	1.1.1
pROC	1.16.2	processx	3.4.5	prodlim	2019.11.13
progress	1.2.2	promises	1.1.1	proto	1.0.0
ps	1.5.0	purrr	0.3.4	r2d3	0.2.3
R6	2.5.0	randomForest	4.6-14	rappdirs	0.3.1
rcmdcheck	1.3.3	RColorBrewer	1.1-2	Rcpp	1.0.5
readr	1.4.0	readxl	1.3.1	recipes	0.1.15
rematch	1.0.1	rematch2	2.1.2	remotes	2.2.0
reprex	0.3.0	reshape2	1.4.4	rex	1.2.0
rlang	0.4.9	rmarkdown	2.6	RODBC	1.3-17
roxygen2	7.1.1	rpart	4.1-15	rprojroot	2.0.2
Rserve	1.8-7	RSQLite	2.2.1	rstudioapi	0.13
rversions	2.0.2	rvest	0.3.6	scales	1.1.1
selectr	0.4-2	sessioninfo	1.1.1	shape	1.4.5
shiny	1.5.0	sourcetools	0.1.7	sparklyr	1.5.2
SparkR	3.1.0	spatial	7.3-11	splines	4.0.4
sqldf	0.4-11	SQUAREM	2020.5	stats	4.0.4
stats4	4.0.4	stringi	1.5.3	stringr	1.4.0
survival	3.2-7	sys	3.4	tcltk	4.0.4
TeachingDemos	2.10	testthat	3.0.0	tibble	3.0.4
tidyr	1.1.2	tidyselect	1.1.0	tidyverse	1.3.0
timeDate	3043.102	tinytex	0.28	tools	4.0.4
usethis	2.0.0	utf8	1.1.4	utils	4.0.4
uuid	0.1-4	vctrs	0.3.5	viridisLite	0.3.0
waldo	0.2.3	whisker	0.4	withr	2.3.0
xfun	0.19	xml2	1.3.2	xopen	1.0.0
xtable	1.8-4	yaml	2.2.1	zip	2.1.1

Installed Java and Scala libraries (Scala 2.12 cluster version)

Group ID	Artifact ID	Version
antlr	antlr	2.7.7
com.amazonaws	amazon-kinesis-client	1.12.0
com.amazonaws	aws-java-sdk-autoscaling	1.11.655
com.amazonaws	aws-java-sdk-cloudformation	1.11.655
com.amazonaws	aws-java-sdk-cloudfront	1.11.655
com.amazonaws	aws-java-sdk-cloudhsm	1.11.655
com.amazonaws	aws-java-sdk-cloudsearch	1.11.655
com.amazonaws	aws-java-sdk-cloudtrail	1.11.655
com.amazonaws	aws-java-sdk-cloudwatch	1.11.655
com.amazonaws	aws-java-sdk-cloudwatchmetrics	1.11.655
com.amazonaws	aws-java-sdk-codedeploy	1.11.655
com.amazonaws	aws-java-sdk-cognitoidentity	1.11.655
com.amazonaws	aws-java-sdk-cognitosync	1.11.655
com.amazonaws	aws-java-sdk-config	1.11.655
com.amazonaws	aws-java-sdk-core	1.11.655
com.amazonaws	aws-java-sdk-datapipeline	1.11.655
com.amazonaws	aws-java-sdk-directconnect	1.11.655
com.amazonaws	aws-java-sdk-directory	1.11.655
com.amazonaws	aws-java-sdk-dynamodb	1.11.655
com.amazonaws	aws-java-sdk-ec2	1.11.655
com.amazonaws	aws-java-sdk-ecs	1.11.655
com.amazonaws	aws-java-sdk-efs	1.11.655
com.amazonaws	aws-java-sdk-elasticache	1.11.655
com.amazonaws	aws-java-sdk-elasticbeanstalk	1.11.655
com.amazonaws	aws-java-sdk-elasticloadbalancing	1.11.655
com.amazonaws	aws-java-sdk-elastictranscoder	1.11.655
com.amazonaws	aws-java-sdk-emr	1.11.655
com.amazonaws	aws-java-sdk-glacier	1.11.655
com.amazonaws	aws-java-sdk-iam	1.11.655
com.amazonaws	aws-java-sdk-importexport	1.11.655
com.amazonaws	aws-java-sdk-kinesis	1.11.655
com.amazonaws	aws-java-sdk-kms	1.11.655
com.amazonaws	aws-java-sdk-lambda	1.11.655
com.amazonaws	aws-java-sdk-logs	1.11.655
com.amazonaws	aws-java-sdk-machinelearning	1.11.655
com.amazonaws	aws-java-sdk-opsworks	1.11.655
com.amazonaws	aws-java-sdk-rds	1.11.655
com.amazonaws	aws-java-sdk-redshift	1.11.655
com.amazonaws	aws-java-sdk-route53	1.11.655
com.amazonaws	aws-java-sdk-s3	1.11.655
com.amazonaws	aws-java-sdk-ses	1.11.655
com.amazonaws	aws-java-sdk-simpledb	1.11.655
com.amazonaws	aws-java-sdk-simpleworkflow	1.11.655
com.amazonaws	aws-java-sdk-sns	1.11.655
com.amazonaws	aws-java-sdk-sqs	1.11.655
com.amazonaws	aws-java-sdk-ssm	1.11.655
com.amazonaws	aws-java-sdk-storagegateway	1.11.655
com.amazonaws	aws-java-sdk-sts	1.11.655
com.amazonaws	aws-java-sdk-support	1.11.655
com.amazonaws	aws-java-sdk-swf-libraries	1.11.22
com.amazonaws	aws-java-sdk-workspaces	1.11.655
com.amazonaws	jmespath-java	1.11.655
com.chuusai	shapeless_2.12	2.3.3
com.clearspring.analytics	stream	2.9.6
com.databricks	Rserve	1.8-3
com.databricks	jets3t	0.7.1-0
com.databricks.scalapb	compilerplugin_2.12	0.4.15-10
com.databricks.scalapb	scalapb-runtime_2.12	0.4.15-10
com.esotericsoftware	kryo-shaded	4.0.2
com.esotericsoftware	minlog	1.3.0
com.fasterxml	classmate	1.3.4
com.fasterxml.jackson.core	jackson-annotations	2.10.0
com.fasterxml.jackson.core	jackson-core	2.10.0
com.fasterxml.jackson.core	jackson-databind	2.10.0
com.fasterxml.jackson.dataformat	jackson-dataformat-cbor	2.10.0
com.fasterxml.jackson.datatype	jackson-datatype-joda	2.10.0
com.fasterxml.jackson.module	jackson-module-paranamer	2.10.0
com.fasterxml.jackson.module	jackson-module-scala_2.12	2.10.0
com.github.ben-manes.caffeine	caffeine	2.3.4
com.github.fommil	jniloader	1.1
com.github.fommil.netlib	core	1.1.2
com.github.fommil.netlib	native_ref-java	1.1
com.github.fommil.netlib	native_ref-java-natives	1.1
com.github.fommil.netlib	native_system-java	1.1
com.github.fommil.netlib	native_system-java-natives	1.1
com.github.fommil.netlib	netlib-native_ref-linux-x86_64-natives	1.1
com.github.fommil.netlib	netlib-native_system-linux-x86_64-natives	1.1
com.github.joshelser	dropwizard-metrics-hadoop-metrics2-reporter	0.1.2
com.github.luben	zstd-jni	1.4.8-1
com.github.wendykierp	JTransforms	3.1
com.google.code.findbugs	jsr305	3.0.0
com.google.code.gson	gson	2.2.4
com.google.flatbuffers	flatbuffers-java	1.9.0
com.google.guava	guava	15.0
com.google.protobuf	protobuf-java	2.6.1
com.h2database	h2	1.4.195
com.helger	profiler	1.1.1
com.jcraft	jsch	0.1.50
com.jolbox	bonecp	0.8.0.RELEASE
com.lihaoyi	sourcecode_2.12	0.1.9
com.microsoft.sqlserver	mssql-jdbc	9.2.1.jre8
com.microsoft.azure	azure-data-lake-store-sdk	2.3.9
com.ning	compress-lzf	1.0.3
com.sun.mail	javax.mail	1.5.2
com.tdunning	json	1.8
com.thoughtworks.paranamer	paranamer	2.8
com.trueaccord.lenses	lenses_2.12	0.4.12
com.twitter	chill-java	0.9.5
com.twitter	chill_2.12	0.9.5
com.twitter	util-app_2.12	7.1.0
com.twitter	util-core_2.12	7.1.0
com.twitter	util-function_2.12	7.1.0
com.twitter	util-jvm_2.12	7.1.0
com.twitter	util-lint_2.12	7.1.0
com.twitter	util-registry_2.12	7.1.0
com.twitter	util-stats_2.12	7.1.0
com.typesafe	config	1.2.1
com.typesafe.scala-logging	scala-logging_2.12	3.7.2
com.univocity	univocity-parsers	2.9.0
com.zaxxer	HikariCP	3.1.0
commons-beanutils	commons-beanutils	1.9.4
commons-cli	commons-cli	1.2
commons-codec	commons-codec	1.10
commons-collections	commons-collections	3.2.2
commons-configuration	commons-configuration	1.6
commons-dbcp	commons-dbcp	1.4
commons-digester	commons-digester	1.8
commons-fileupload	commons-fileupload	1.3.3
commons-httpclient	commons-httpclient	3.1
commons-io	commons-io	2.4
commons-lang	commons-lang	2.6
commons-logging	commons-logging	1.1.3
commons-net	commons-net	3.1
commons-pool	commons-pool	1.5.4
info.ganglia.gmetric4j	gmetric4j	1.0.10
io.airlift	aircompressor	0.10
io.dropwizard.metrics	metrics-core	4.1.1
io.dropwizard.metrics	metrics-graphite	4.1.1
io.dropwizard.metrics	metrics-healthchecks	4.1.1
io.dropwizard.metrics	metrics-jetty9	4.1.1
io.dropwizard.metrics	metrics-jmx	4.1.1
io.dropwizard.metrics	metrics-json	4.1.1
io.dropwizard.metrics	metrics-jvm	4.1.1
io.dropwizard.metrics	metrics-servlets	4.1.1
io.netty	netty-all	4.1.51.Final
io.prometheus	simpleclient	0.7.0
io.prometheus	simpleclient_common	0.7.0
io.prometheus	simpleclient_dropwizard	0.7.0
io.prometheus	simpleclient_pushgateway	0.7.0
io.prometheus	simpleclient_servlet	0.7.0
io.prometheus.jmx	collector	0.12.0
jakarta.annotation	jakarta.annotation-api	1.3.5
jakarta.validation	jakarta.validation-api	2.0.2
jakarta.ws.rs	jakarta.ws.rs-api	2.1.6
javax.activation	activation	1.1.1
javax.el	javax.el-api	2.2.4
javax.jdo	jdo-api	3.0.1
javax.servlet	javax.servlet-api	3.1.0
javax.servlet.jsp	jsp-api	2.1
javax.transaction	jta	1.1
javax.transaction	transaction-api	1.1
javax.xml.bind	jaxb-api	2.2.2
javax.xml.stream	stax-api	1.0-2
javolution	javolution	5.5.1
jline	jline	2.14.6
joda-time	joda-time	2.10.5
log4j	apache-log4j-extras	1.2.17
log4j	log4j	1.2.17
net.razorvine	pyrolite	4.30
net.sf.jpam	jpam	1.1
net.sf.opencsv	opencsv	2.3
net.sf.supercsv	super-csv	2.2.0
net.snowflake	snowflake-ingest-sdk	0.9.6
net.snowflake	snowflake-jdbc	3.12.8
net.snowflake	spark-snowflake_2.12	2.8.1-spark_3.0
net.sourceforge.f2j	arpack_combined_all	0.1
org.acplt.remotetea	remotetea-oncrpc	1.1.2
org.antlr	ST4	4.0.4
org.antlr	antlr-runtime	3.5.2
org.antlr	antlr4-runtime	4.8-1
org.antlr	stringtemplate	3.2.1
org.apache.ant	ant	1.9.2
org.apache.ant	ant-jsch	1.9.2
org.apache.ant	ant-launcher	1.9.2
org.apache.arrow	arrow-format	2.0.0
org.apache.arrow	arrow-memory-core	2.0.0
org.apache.arrow	arrow-memory-netty	2.0.0
org.apache.arrow	arrow-vector	2.0.0
org.apache.avro	avro	1.8.2
org.apache.avro	avro-ipc	1.8.2
org.apache.avro	avro-mapred-hadoop2	1.8.2
org.apache.commons	commons-compress	1.20
org.apache.commons	commons-crypto	1.1.0
org.apache.commons	commons-lang3	3.10
org.apache.commons	commons-math3	3.4.1
org.apache.commons	commons-text	1.6
org.apache.curator	curator-client	2.7.1
org.apache.curator	curator-framework	2.7.1
org.apache.curator	curator-recipes	2.7.1
org.apache.derby	derby	10.12.1.1
org.apache.directory.api	api-asn1-api	1.0.0-M20
org.apache.directory.api	api-util	1.0.0-M20
org.apache.directory.server	apacheds-i18n	2.0.0-M15
org.apache.directory.server	apacheds-kerberos-codec	2.0.0-M15
org.apache.hadoop	hadoop-annotations	2.7.4
org.apache.hadoop	hadoop-auth	2.7.4
org.apache.hadoop	hadoop-client	2.7.4
org.apache.hadoop	hadoop-common	2.7.4
org.apache.hadoop	hadoop-hdfs	2.7.4
org.apache.hadoop	hadoop-mapreduce-client-app	2.7.4
org.apache.hadoop	hadoop-mapreduce-client-common	2.7.4
org.apache.hadoop	hadoop-mapreduce-client-core	2.7.4
org.apache.hadoop	hadoop-mapreduce-client-jobclient	2.7.4
org.apache.hadoop	hadoop-mapreduce-client-shuffle	2.7.4
org.apache.hadoop	hadoop-yarn-api	2.7.4
org.apache.hadoop	hadoop-yarn-client	2.7.4
org.apache.hadoop	hadoop-yarn-common	2.7.4
org.apache.hadoop	hadoop-yarn-server-common	2.7.4
org.apache.hive	hive-beeline	2.3.7
org.apache.hive	hive-cli	2.3.7
org.apache.hive	hive-common	2.3.7
org.apache.hive	hive-exec-core	2.3.7
org.apache.hive	hive-jdbc	2.3.7
org.apache.hive	hive-llap-client	2.3.7
org.apache.hive	hive-llap-common	2.3.7
org.apache.hive	hive-metastore	2.3.7
org.apache.hive	hive-serde	2.3.7
org.apache.hive	hive-shims	2.3.7
org.apache.hive	hive-storage-api	2.7.2
org.apache.hive	hive-vector-code-gen	2.3.7
org.apache.hive.shims	hive-shims-0.23	2.3.7
org.apache.hive.shims	hive-shims-common	2.3.7
org.apache.hive.shims	hive-shims-scheduler	2.3.7
org.apache.htrace	htrace-core	3.1.0-incubating
org.apache.httpcomponents	httpclient	4.5.6
org.apache.httpcomponents	httpcore	4.4.12
org.apache.ivy	ivy	2.4.0
org.apache.mesos	mesos-shaded-protobuf	1.4.0
org.apache.orc	orc-core	1.5.12
org.apache.orc	orc-mapreduce	1.5.12
org.apache.orc	orc-shims	1.5.12
org.apache.parquet	parquet-column	1.10.1-databricks6
org.apache.parquet	parquet-common	1.10.1-databricks6
org.apache.parquet	parquet-encoding	1.10.1-databricks6
org.apache.parquet	parquet-format	2.4.0
org.apache.parquet	parquet-hadoop	1.10.1-databricks6
org.apache.parquet	parquet-jackson	1.10.1-databricks6
org.apache.thrift	libfb303	0.9.3
org.apache.thrift	libthrift	0.12.0
org.apache.velocity	velocity	1.5
org.apache.xbean	xbean-asm7-shaded	4.15
org.apache.yetus	audience-annotations	0.5.0
org.apache.zookeeper	zookeeper	3.4.14
org.codehaus.jackson	jackson-core-asl	1.9.13
org.codehaus.jackson	jackson-jaxrs	1.9.13
org.codehaus.jackson	jackson-mapper-asl	1.9.13
org.codehaus.jackson	jackson-xc	1.9.13
org.codehaus.janino	commons-compiler	3.0.16
org.codehaus.janino	janino	3.0.16
org.datanucleus	datanucleus-api-jdo	4.2.4
org.datanucleus	datanucleus-core	4.1.17
org.datanucleus	datanucleus-rdbms	4.1.19
org.datanucleus	javax.jdo	3.2.0-m3
org.eclipse.jetty	jetty-client	9.4.36.v20210114
org.eclipse.jetty	jetty-continuation	9.4.36.v20210114
org.eclipse.jetty	jetty-http	9.4.36.v20210114
org.eclipse.jetty	jetty-io	9.4.36.v20210114
org.eclipse.jetty	jetty-jndi	9.4.36.v20210114
org.eclipse.jetty	jetty-plus	9.4.36.v20210114
org.eclipse.jetty	jetty-proxy	9.4.36.v20210114
org.eclipse.jetty	jetty-security	9.4.36.v20210114
org.eclipse.jetty	jetty-server	9.4.36.v20210114
org.eclipse.jetty	jetty-servlet	9.4.36.v20210114
org.eclipse.jetty	jetty-servlets	9.4.36.v20210114
org.eclipse.jetty	jetty-util	9.4.36.v20210114
org.eclipse.jetty	jetty-util-ajax	9.4.36.v20210114
org.eclipse.jetty	jetty-webapp	9.4.36.v20210114
org.eclipse.jetty	jetty-xml	9.4.36.v20210114
org.fusesource.leveldbjni	leveldbjni-all	1.8
org.glassfish.hk2	hk2-api	2.6.1
org.glassfish.hk2	hk2-locator	2.6.1
org.glassfish.hk2	hk2-utils	2.6.1
org.glassfish.hk2	osgi-resource-locator	1.0.3
org.glassfish.hk2.external	aopalliance-repackaged	2.6.1
org.glassfish.hk2.external	jakarta.inject	2.6.1
org.glassfish.jersey.containers	jersey-container-servlet	2.30
org.glassfish.jersey.containers	jersey-container-servlet-core	2.30
org.glassfish.jersey.core	jersey-client	2.30
org.glassfish.jersey.core	jersey-common	2.30
org.glassfish.jersey.core	jersey-server	2.30
org.glassfish.jersey.inject	jersey-hk2	2.30
org.glassfish.jersey.media	jersey-media-jaxb	2.30
org.hibernate.validator	hibernate-validator	6.1.0.Final
org.javassist	javassist	3.25.0-GA
org.jboss.logging	jboss-logging	3.3.2.Final
org.jdbi	jdbi	2.63.1
org.joda	joda-convert	1.7
org.jodd	jodd-core	3.5.2
org.json4s	json4s-ast_2.12	3.7.0-M5
org.json4s	json4s-core_2.12	3.7.0-M5
org.json4s	json4s-jackson_2.12	3.7.0-M5
org.json4s	json4s-scalap_2.12	3.7.0-M5
org.lz4	lz4-java	1.7.1
org.mariadb.jdbc	mariadb-java-client	2.1.2
org.objenesis	objenesis	2.5.1
org.postgresql	postgresql	42.1.4
org.roaringbitmap	RoaringBitmap	0.9.0
org.roaringbitmap	shims	0.9.0
org.rocksdb	rocksdbjni	6.2.2
org.rosuda.REngine	REngine	2.1.0
org.scala-lang	scala-compiler_2.12	2.12.10
org.scala-lang	scala-library_2.12	2.12.10
org.scala-lang	scala-reflect_2.12	2.12.10
org.scala-lang.modules	scala-collection-compat_2.12	2.1.1
org.scala-lang.modules	scala-parser-combinators_2.12	1.1.2
org.scala-lang.modules	scala-xml_2.12	1.2.0
org.scala-sbt	test-interface	1.0
org.scalacheck	scalacheck_2.12	1.14.2
org.scalactic	scalactic_2.12	3.0.8
org.scalanlp	breeze-macros_2.12	1.0
org.scalanlp	breeze_2.12	1.0
org.scalatest	scalatest_2.12	3.0.8
org.slf4j	jcl-over-slf4j	1.7.30
org.slf4j	jul-to-slf4j	1.7.30
org.slf4j	slf4j-api	1.7.30
org.slf4j	slf4j-log4j12	1.7.30
org.spark-project.spark	unused	1.0.0
org.springframework	spring-core	4.1.4.RELEASE
org.springframework	spring-test	4.1.4.RELEASE
org.threeten	threeten-extra	1.5.0
org.tukaani	xz	1.5
org.typelevel	algebra_2.12	2.0.0-M2
org.typelevel	cats-kernel_2.12	2.0.0-M4
org.typelevel	machinist_2.12	0.6.8
org.typelevel	macro-compat_2.12	1.1.1
org.typelevel	spire-macros_2.12	0.17.0-M1
org.typelevel	spire-platform_2.12	0.17.0-M1
org.typelevel	spire-util_2.12	0.17.0-M1
org.typelevel	spire_2.12	0.17.0-M1
org.wildfly.openssl	wildfly-openssl	1.0.7.Final
org.xerial	sqlite-jdbc	3.8.11.2
org.xerial.snappy	snappy-java	1.1.8.2
org.yaml	snakeyaml	1.24
oro	oro	2.0.8
pl.edu.icm	JLargeArrays	1.5
software.amazon.ion	ion-java	1.0.2
stax	stax-api	1.0.1
xmlenc	xmlenc	0.52

Κοινή χρήση μέσω

Databricks Runtime 8.1 (EoS)

New features and improvements

New SQL operators reduce complex ETL on JSON strings

Control visibility of table access control lists

New Auto Loader options enable better performance

Authenticate to Azure Synapse Analytics with OAuth 2.0 and a service principal (Public Preview)

Manually insert `IDENTITY` columns in Azure Synapse Analytics tables

Azure Auto Loader now supports authentication with a service principal for queue operations

Track stream progress with Kafka metrics

Track stream progress with Delta Lake streaming source metrics

Library upgrades

Apache Spark

Maintenance updates

System environment

Installed Python libraries

Installed R libraries

Installed Java and Scala libraries (Scala 2.12 cluster version)

Σχόλια

Πρόσθετοι πόροι

Κοινή χρήση μέσω

Databricks Runtime 8.1 (EoS)

New features and improvements

New SQL operators reduce complex ETL on JSON strings

Control visibility of table access control lists

New Auto Loader options enable better performance

Authenticate to Azure Synapse Analytics with OAuth 2.0 and a service principal (Public Preview)

Manually insert IDENTITY columns in Azure Synapse Analytics tables

Azure Auto Loader now supports authentication with a service principal for queue operations

Track stream progress with Kafka metrics

Track stream progress with Delta Lake streaming source metrics

Library upgrades

Apache Spark

Maintenance updates

System environment

Installed Python libraries

Installed R libraries

Installed Java and Scala libraries (Scala 2.12 cluster version)

Σχόλια

Πρόσθετοι πόροι

Manually insert `IDENTITY` columns in Azure Synapse Analytics tables