Databricks Runtime 10.0 (unsupported)

  • Article

The following release notes provide information about Databricks Runtime 10.0 and Databricks Runtime 10.0 Photon, powered by Apache Spark 3.2.0. Databricks released these images in October 2021. Photon is in Public Preview.

New features and improvements

New version of Apache Spark

Databricks Runtime 10.0 and Databricks Runtime 10.0 Photon include Apache Spark 3.2.0. For details, see Apache Spark.

SELECT statement now supports QUALIFY clause to filter window function results

The SELECT statement now supports the QUALIFY clause. QUALIFY can be used to filter the results of window functions. One or more window functions must be present in either the SELECT list or the QUALIFY condition. For example:

SELECT * FROM t QUALIFY SUM(c2) OVER (PARTITION BY c1) > 0;

Cluster support for JDK 11 (Public Preview)

Databricks now provides cluster support for Java Development Kit (JDK) 11.

When you create a cluster, you can specify that the cluster uses JDK 11 (for both the driver and executor). To do this, add the following environment variable to Advanced Options > Spark > Environment Variables:

JNAME=zulu11-ca-amd64

Auto Loader now treats schemas as nullable

Auto Loader now treats all inferred and user-provided schemas as nullable by default. This is to avoid potential data corruption in cases where the data contains null fields for non-nullable columns. A new configuration is introduced to track this behavior, spark.databricks.cloudFiles.schema.forceNullable. By default, this configuration contains the setting of spark.sql.streaming.fileSource.schema.forceNullable, which the FileStreamSource in Apache Spark uses and is set to true by default.

Breaking changes

Breaking changes for all Spark SQL users

  • New spark.databricks.behaviorChange.SC78546CorrelatedPredicate.enabled configuration: when set to true, allows a subset of correlated equality predicates when a subquery is aggregated. The default setting is true.
  • New spark.databricks.behaviorChange.SC82201BlockAutoAlias.enabled configuration: when set to true, blocks autogenerating aliases when views are created. The default setting is true.
  • New spark.databricks.behaviorChange.SC81078CTASWithLocation.enabled configuration: when set to true, disallows CREATE TABLE AS SELECT with a non-empty location. The default setting is true. Note that when spark.sql.legacy.allowNonEmptyLocationInCTAS is also set to true, this configuration has no effect, and CREATE TABLE AS SELECT with a non-empty location is always allowed.

Breaking changes for Spark SQL users who enable ANSI mode

For information about ANSI mode, see ANSI compliance in Databricks Runtime.

  • New spark.databricks.behaviorChange.SC83587NextDayDOW.enabled configuration: when set to true, an invalid dayOfWeek argument to the next_day function throws an IllegalArgumentException in ANSI mode; otherwise it returns null. The default setting is true.
  • New spark.databricks.behaviorChange.SC83796CanCast.enabled configuration: when set to true, enables new explicit cast syntax rules in ANSI mode. The default setting is true.
  • New spark.databricks.behaviorChange.SC79099CastStringToBoolean.enabled configuration: when set to true, a parse error exception is thrown when casting a string to a Boolean; otherwise, returns null. The default setting is true.
  • New spark.databricks.behaviorChange.SC79064AbsOutsideRange.enabled configuration: when set to true, the abs function throws an exception if the input is out of range. The default setting is true.

Breaking changes for all Python users

  • The Python API delta.tables.DeltaTable.convertToDelta now returns the correct DeltaTable Python object which can be used to run Delta Lake commands. Previously it returned an internal object which could not be called directly.

Library upgrades

Apache Hadoop 3 upgrade

  • Databricks Runtime 10.0 upgrades the Hadoop dependency from Hadoop 2.7.4 to Hadoop 3.3.1.

Behavior changes

  • Hadoop 3 uses the hadoop-client-api and hadoop-client-runtime libraries instead of the hadoop-common library, which shades certain third-party dependencies that could potentially be used in Hadoop public APIs or extensions.
  • Hadoop 3 configuration options have changed since Hadoop 2. For Hadoop 3.3.1 options, see core-default.xml.
  • Databricks has updated some of the default configurations for Hadoop 3 to be consistent with Hadoop 2, to make sure storage connectors have the same default authentication settings and levels of performance:
    • fs.azure.authorization.caching.enable = false
    • fs.s3a.attempts.maximum = 10
    • fs.s3a.block.size = 67108864
    • fs.s3a.connection.timeout = 50000
    • fs.s3a.max.total.tasks = 1000
    • fs.s3a.retry.limit = 20
    • fs.s3a.retry.throttle.interval = 500ms
    • fs.s3a.assumed.role.credentials.provider = com.amazonaws.auth.InstanceProfileCredentialsProvider
    • fs.s3a.aws.credentials.provider = BasicAWSCredentialsProvider, DatabricksInstanceProfileCredentialsProvider, EnvironmentVariableCredentialsProvider, AnonymousAWSCredentialsProvider
  • The Google Cloud Storage (GCS) connector has been upgraded from 2.1.6 to 2.2.2.
  • The Amazon Redshift connector now uses the s3a:// scheme. The s3n:// scheme is deprecated.
  • Handles the OSS class names for the Amazon S3 MetadataStore API in addition to shaded classes. This enables processing OSS configurations without requiring the shaded class names.
    • For example, you can specify the org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore class in the Hadoop configuration.
  • Makes the new Configuration() and sparkContext.hadoopConfiguration consistent.
    • Now every time a new Hadoop configuration is created, it will be consistent with the predefined Hadoop configuration in sparkContext.hadoopConfiguration in the Databricks Runtime, including file system schemes and their default configuration.
  • Although Hadoop storage connectors included in the Databricks Runtime are fully compatible with Hadoop 3.3.1, they are not guaranteed to be in sync with OSS Hadoop 3.3.1 connectors and might have different behavior.
    • The Amazon S3 connector still allows (although with a warning) user:secret authentication in S3 URLs, compared to HADOOP-14833 which removes it.
  • globStatus now always returns sorted results(HADOOP-10798)
  • Added fs.s3a.endpoint if unset and fs.s3a.endpoint region is null(SPARK-35878)
    • Databricks provided Amazon S3 region auto resolution might not be triggered in some cases, due to the global endpoint being set. This is not an issue, as the AWS SDK will resolve the region correctly.
  • Add fs.s3a.downgrade.syncable.exceptions if not set(SPARK-35868)
  • LZ4 and Snappy codecs do not rely on the native Hadoop library(HADOOP-17125)

Known issues

  • SPARK-36681 Using SnappyCodec to write sequence file will fail with UnsatisfiedLinkError due to known issue in Hadoop 3.3.1(HADOOP-17891)

Apache Spark

Databricks Runtime 10.0 includes Apache Spark 3.2.0.

In this section:

Highlights

Core and Spark SQL

ANSI SQL compatibility enhancements

Performance enhancements

  • Query optimization
    • Remove redundant aggregates in the Optimizer(SPARK-33122)
    • Push down limit through Project with Join(SPARK-34622)
    • Cardinality estimation of union, sort and range operator (SPARK-33411)
    • UnwrapCastInBinaryComparison support In/InSet predicate(SPARK-35316)
    • Keep necessary stats after partition pruning(SPARK-34119)
  • Query execution
    • Enable Zstandard buffer pool by default(SPARK-34340, SPARK-34390)
    • Add code-gen for all join types of sort merge join(SPARK-34705)
    • Broadcast nested loop join improvement(SPARK-34706)
    • Support two levels of hash maps for final hash aggregation (SPARK-35141)
    • Allow concurrent writers for writing dynamic partitions and bucket table(SPARK-26164)
    • Improve performance of processing FETCH_PRIOR in Thriftserver(SPARK-33655)

Connector enhancements

  • Parquet
    • Upgrade Parquet to 1.12.1(SPARK-36726)
    • Read parquet unsigned types that are stored as int32 physical type in parquet(SPARK-34817)
    • Read Parquet unsigned int64 logical type that stored as signed int64 physical type to decimal(20, 0)(SPARK-34786)
    • Improve Parquet In filter pushdown(SPARK-32792)
  • ORC
  • Avro
    • Upgrade Avro to version 1.10.2(SPARK-34778)
    • Supporting Avro schema evolution for partitioned Hive tables with “avro.schema.literal”(SPARK-26836)
    • Add new Avro data source options to control datetime rebasing in read(SPARK-34404)
    • Adding support for user provided schema url in Avro(SPARK-34416)
    • Add support for positional Catalyst-to-Avro schema matching(SPARK-34365)
  • JSON
    • Upgrade Jackson to version 2.12.3(SPARK-35550)
    • Allow Json data sources to write non-ascii characters as codepoints(SPARK-35047)
  • JDBC
    • Calculate more precise partition stride in JDBCRelation(SPARK-34843)
  • Hive Metastore support filter by not-in(SPARK-34538)

Feature enhancements

Other notable changes

  • Monitor
    • New metrics to ExternalShuffleService(SPARK-35258)
    • Add new stage-level REST APIs and parameters (SPARK-26399)
    • Support task and executor Metrics Distributions Metrics Distributions in the REST API(SPARK-34488)
    • Add fallback metrics for hash aggregate(SPARK-35529)
  • Add count_distinct as an option to Dataset#summary(SPARK-34165)
  • Implement ScriptTransform in sql/core(SPARK-31936)
  • Make BlockManagerMaster driver heartbeat timeout configurable(SPARK-34278)
  • Make shuffle service name configurable on client side and allow for classpath-based config override on server side(SPARK-34828)
  • ExecutorMetricsPoller should keep stage entry in stageTCMP until a heartbeat occurs(SPARK-34779)
  • Replace if with filter clause in RewriteDistinctAggregates(SPARK-34882)
  • Fix failure when apply CostBasedJoinReorder on self-join(SPARK-34354)
  • CREATE TABLE LIKE should respect the reserved table properties(SPARK-34935)
  • Ship ivySettings file to driver in YARN cluster mode(SPARK-34472)
  • Resolve duplicated common columns from USING/NATURAL JOIN(SPARK-34527)
  • Hide internal view properties for describe table cmd(SPARK-35318)
  • Support resolving missing attrs for distribute/cluster by/repartition hint(SPARK-35331)
  • Ignore the error when checking the path in FileStreamSink.hasMetadata(SPARK-34526)
  • Improve s3a magic committer support by inferring missing configs(SPARK-35383)
  • Allow to omit : in the STRUCT type string (SPARK-35706)
  • Add a new operator to distinguish if AQE can optimize safely (SPARK-35786)
  • Append new nested struct fields rather than sort for unionByName with null filling(SPARK-35290)
  • ArraysZip should retain field names to avoid being re-written by analyzer/optimizer(SPARK-35876)
  • Use Void as the type name of NullType(SPARK-36224)
  • Introduce new API to FileCommitProtocol allow flexible file naming(SPARK-33298)

Changes of behavior

See the migration guides for each component: Spark Core.

Structured Streaming

Major features

  • EventTime based sessionization (session window)(SPARK-10816)
  • Upgrade Kafka client to 2.8.0(SPARK-33913)
  • Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches in Scala(SPARK-36533)

Other notable changes

  • Introduce a new Option in Kafka source to specify a minimum number of records to read per trigger(SPARK-35312)
  • Add latest offsets to source progress(SPARK-33955)

PySpark

Project Zen

Other notable changes

  • Enable pinned thread mode by default(SPARK-35303)
  • Add NullType support for Arrow executions(SPARK-33489)
  • Add Arrow self_destruct support to toPandas (SPARK-32953)
  • Add thread target wrapper API for pyspark pin thread mode(SPARK-35498)

Changes of behavior

See the migration guides.

MLlib

Performance improvements

  • BucketedRandomProjectionLSH transform optimization(SPARK-34220)
  • w2v findSynonyms optimization(SPARK-34189)
  • optimize sparse GEMM by skipping bound checking(SPARK-35707)
  • Improve performance of ML ALS recommendForAll by GEMV(SPARK-33518)

Model training improvements

  • Refactor Logistic Aggregator - support virtual centering(SPARK-34797)
  • Binary Logistic Regression with intercept support centering(SPARK-34858, SPARK-34448)
  • Multinomial Logistic Regression with intercept support centering(SPARK-34860)

BLAS improvements

  • Replace fully com.github.fommil.netlib by dev.ludovic.netlib:2.0(SPARK-35295)
  • Add a vectorized BLAS implementation(SPARK-33882)
  • Accelerate fallback BLAS with dev.ludovic.netlib(SPARK-35150)

Other notable changes

  • OVR transform fix potential column conflict(SPARK-34356)

Deprecations and removals

  • Deprecate spark.launcher.childConnectionTimeout(SPARK-33717)
  • deprecate GROUP BY … GROUPING SETS (…) and promote GROUP BY GROUPING SETS (…)(SPARK-34932)
  • Deprecate ps.broadcast API(SPARK-35810)
  • Deprecate the num_files argument(SPARK-35807)
  • Deprecate DataFrame.to_spark_io(SPARK-35811)

Maintenance updates

See Databricks Runtime 10.0 maintenance updates.

System environment

  • Operating System: Ubuntu 20.04.3 LTS
  • Java: Zulu 8.56.0.21-CA-linux64
  • Scala: 2.12.14
  • Python: 3.8.10
  • R: 4.1.1
  • Delta Lake: 1.0.0

Installed Python libraries

Library Version Library Version Library Version
Antergos Linux 2015.10 (ISO-Rolling) appdirs 1.4.4 backcall 0.2.0
boto3 1.16.7 botocore 1.19.7 certifi 2020.12.5
chardet 4.0.0 cycler 0.10.0 Cython 0.29.23
dbus-python 1.2.16 decorator 5.0.6 distlib 0.3.3
distro-info 0.23ubuntu1 facets-overview 1.0.0 filelock 3.0.12
idna 2.10 ipykernel 5.3.4 ipython 7.22.0
ipython-genutils 0.2.0 jedi 0.17.2 jmespath 0.10.0
joblib 1.0.1 jupyter-client 6.1.12 jupyter-core 4.7.1
kiwisolver 1.3.1 koalas 1.8.1 matplotlib 3.4.2
numpy 1.19.2 pandas 1.2.4 parso 0.7.0
patsy 0.5.1 pexpect 4.8.0 pickleshare 0.7.5
Pillow 8.2.0 pip 21.0.1 plotly 5.1.0
prompt-toolkit 3.0.17 protobuf 3.17.2 psycopg2 2.8.5
ptyprocess 0.7.0 pyarrow 4.0.0 Pygments 2.8.1
PyGObject 3.36.0 pyparsing 2.4.7 python-apt 2.0.0+ubuntu0.20.4.6
python-dateutil 2.8.1 pytz 2020.5 pyzmq 20.0.0
requests 2.25.1 requests-unixsocket 0.2.0 s3transfer 0.3.7
scikit-learn 0.24.1 scipy 1.6.2 seaborn 0.11.1
setuptools 52.0.0 six 1.15.0 ssh-import-id 5.10
statsmodels 0.12.2 tenacity 8.0.1 threadpoolctl 2.1.0
tornado 6.1 traitlets 5.0.5 unattended-upgrades 0.1
urllib3 1.25.11 virtualenv 20.4.1 wcwidth 0.2.5
wheel 0.36.2

Installed R libraries

R libraries are installed from the Microsoft CRAN snapshot on 2021-09-21.

Library Version Library Version Library Version
askpass 1.1 assertthat 0.2.1 backports 1.2.1
base 4.1.1 base64enc 0.1-3 bit 4.0.4
bit64 4.0.5 blob 1.2.2 boot 1.3-28
brew 1.0-6 brio 1.1.2 broom 0.7.9
bslib 0.3.0 cachem 1.0.6 callr 3.7.0
caret 6.0-88 cellranger 1.1.0 chron 2.3-56
class 7.3-19 cli 3.0.1 clipr 0.7.1
cluster 2.1.2 codetools 0.2-18 colorspace 2.0-2
commonmark 1.7 compiler 4.1.1 config 0.3.1
cpp11 0.3.1 crayon 1.4.1 credentials 1.3.1
curl 4.3.2 data.table 1.14.0 datasets 4.1.1
DBI 1.1.1 dbplyr 2.1.1 desc 1.3.0
devtools 2.4.2 diffobj 0.3.4 digest 0.6.27
dplyr 1.0.7 dtplyr 1.1.0 ellipsis 0.3.2
evaluate 0.14 fansi 0.5.0 farver 2.1.0
fastmap 1.1.0 forcats 0.5.1 foreach 1.5.1
foreign 0.8-81 forge 0.2.0 fs 1.5.0
future 1.22.1 future.apply 1.8.1 gargle 1.2.0
generics 0.1.0 gert 1.4.1 ggplot2 3.3.5
gh 1.3.0 gitcreds 0.1.1 glmnet 4.1-2
globals 0.14.0 glue 1.4.2 googledrive 2.0.0
googlesheets4 1.0.0 gower 0.2.2 graphics 4.1.1
grDevices 4.1.1 grid 4.1.1 gridExtra 2.3
gsubfn 0.7 gtable 0.3.0 haven 2.4.3
highr 0.9 hms 1.1.0 htmltools 0.5.2
htmlwidgets 1.5.4 httpuv 1.6.3 httr 1.4.2
hwriter 1.3.2 hwriterPlus 1.0-3 ids 1.0.1
ini 0.3.1 ipred 0.9-12 isoband 0.2.5
iterators 1.0.13 jquerylib 0.1.4 jsonlite 1.7.2
KernSmooth 2.23-20 knitr 1.34 labeling 0.4.2
later 1.3.0 lattice 0.20-44 lava 1.6.10
lifecycle 1.0.0 listenv 0.8.0 lubridate 1.7.10
magrittr 2.0.1 markdown 1.1 MASS 7.3-54
Matrix 1.3-4 memoise 2.0.0 methods 4.1.1
mgcv 1.8-37 mime 0.11 ModelMetrics 1.2.2.2
modelr 0.1.8 munsell 0.5.0 nlme 3.1-152
nnet 7.3-16 numDeriv 2016.8-1.1 openssl 1.4.5
parallel 4.1.1 parallelly 1.28.1 pillar 1.6.2
pkgbuild 1.2.0 pkgconfig 2.0.3 pkgload 1.2.2
plogr 0.2.0 plyr 1.8.6 praise 1.0.0
prettyunits 1.1.1 pROC 1.18.0 processx 3.5.2
prodlim 2019.11.13 progress 1.2.2 progressr 0.8.0
promises 1.2.0.1 proto 1.0.0 ps 1.6.0
purrr 0.3.4 r2d3 0.2.5 R6 2.5.1
randomForest 4.6-14 rappdirs 0.3.3 rcmdcheck 1.3.3
RColorBrewer 1.1-2 Rcpp 1.0.7 readr 2.0.1
readxl 1.3.1 recipes 0.1.16 rematch 1.0.1
rematch2 2.1.2 remotes 2.4.0 reprex 2.0.1
reshape2 1.4.4 rlang 0.4.11 rmarkdown 2.11
RODBC 1.3-19 roxygen2 7.1.2 rpart 4.1-15
rprojroot 2.0.2 Rserve 1.8-8 RSQLite 2.2.8
rstudioapi 0.13 rversions 2.1.1 rvest 1.0.1
sass 0.4.0 scales 1.1.1 selectr 0.4-2
sessioninfo 1.1.1 shape 1.4.6 shiny 1.6.0
sourcetools 0.1.7 sparklyr 1.7.2 SparkR 3.2.0
spatial 7.3-11 splines 4.1.1 sqldf 0.4-11
SQUAREM 2021.1 stats 4.1.1 stats4 4.1.1
stringi 1.7.4 stringr 1.4.0 survival 3.2-13
sys 3.4 tcltk 4.1.1 TeachingDemos 2.10
testthat 3.0.4 tibble 3.1.4 tidyr 1.1.3
tidyselect 1.1.1 tidyverse 1.3.1 timeDate 3043.102
tinytex 0.33 tools 4.1.1 tzdb 0.1.2
usethis 2.0.1 utf8 1.2.2 utils 4.1.1
uuid 0.1-4 vctrs 0.3.8 viridisLite 0.4.0
vroom 1.5.5 waldo 0.3.1 whisker 0.4
withr 2.4.2 xfun 0.26 xml2 1.3.2
xopen 1.0.0 xtable 1.8-4 yaml 2.2.1
zip 2.2.0

Installed Java and Scala libraries (Scala 2.12 cluster version)

Group ID Artifact ID Version
antlr antlr 2.7.7
com.amazonaws amazon-kinesis-client 1.12.0
com.amazonaws aws-java-sdk-autoscaling 1.11.655
com.amazonaws aws-java-sdk-cloudformation 1.11.655
com.amazonaws aws-java-sdk-cloudfront 1.11.655
com.amazonaws aws-java-sdk-cloudhsm 1.11.655
com.amazonaws aws-java-sdk-cloudsearch 1.11.655
com.amazonaws aws-java-sdk-cloudtrail 1.11.655
com.amazonaws aws-java-sdk-cloudwatch 1.11.655
com.amazonaws aws-java-sdk-cloudwatchmetrics 1.11.655
com.amazonaws aws-java-sdk-codedeploy 1.11.655
com.amazonaws aws-java-sdk-cognitoidentity 1.11.655
com.amazonaws aws-java-sdk-cognitosync 1.11.655
com.amazonaws aws-java-sdk-config 1.11.655
com.amazonaws aws-java-sdk-core 1.11.655
com.amazonaws aws-java-sdk-datapipeline 1.11.655
com.amazonaws aws-java-sdk-directconnect 1.11.655
com.amazonaws aws-java-sdk-directory 1.11.655
com.amazonaws aws-java-sdk-dynamodb 1.11.655
com.amazonaws aws-java-sdk-ec2 1.11.655
com.amazonaws aws-java-sdk-ecs 1.11.655
com.amazonaws aws-java-sdk-efs 1.11.655
com.amazonaws aws-java-sdk-elasticache 1.11.655
com.amazonaws aws-java-sdk-elasticbeanstalk 1.11.655
com.amazonaws aws-java-sdk-elasticloadbalancing 1.11.655
com.amazonaws aws-java-sdk-elastictranscoder 1.11.655
com.amazonaws aws-java-sdk-emr 1.11.655
com.amazonaws aws-java-sdk-glacier 1.11.655
com.amazonaws aws-java-sdk-glue 1.11.655
com.amazonaws aws-java-sdk-iam 1.11.655
com.amazonaws aws-java-sdk-importexport 1.11.655
com.amazonaws aws-java-sdk-kinesis 1.11.655
com.amazonaws aws-java-sdk-kms 1.11.655
com.amazonaws aws-java-sdk-lambda 1.11.655
com.amazonaws aws-java-sdk-logs 1.11.655
com.amazonaws aws-java-sdk-machinelearning 1.11.655
com.amazonaws aws-java-sdk-opsworks 1.11.655
com.amazonaws aws-java-sdk-rds 1.11.655
com.amazonaws aws-java-sdk-redshift 1.11.655
com.amazonaws aws-java-sdk-route53 1.11.655
com.amazonaws aws-java-sdk-s3 1.11.655
com.amazonaws aws-java-sdk-ses 1.11.655
com.amazonaws aws-java-sdk-simpledb 1.11.655
com.amazonaws aws-java-sdk-simpleworkflow 1.11.655
com.amazonaws aws-java-sdk-sns 1.11.655
com.amazonaws aws-java-sdk-sqs 1.11.655
com.amazonaws aws-java-sdk-ssm 1.11.655
com.amazonaws aws-java-sdk-storagegateway 1.11.655
com.amazonaws aws-java-sdk-sts 1.11.655
com.amazonaws aws-java-sdk-support 1.11.655
com.amazonaws aws-java-sdk-swf-libraries 1.11.22
com.amazonaws aws-java-sdk-workspaces 1.11.655
com.amazonaws jmespath-java 1.11.655
com.chuusai shapeless_2.12 2.3.3
com.clearspring.analytics stream 2.9.6
com.databricks Rserve 1.8-3
com.databricks jets3t 0.7.1-0
com.databricks.scalapb compilerplugin_2.12 0.4.15-10
com.databricks.scalapb scalapb-runtime_2.12 0.4.15-10
com.esotericsoftware kryo-shaded 4.0.2
com.esotericsoftware minlog 1.3.0
com.fasterxml classmate 1.3.4
com.fasterxml.jackson.core jackson-annotations 2.12.2
com.fasterxml.jackson.core jackson-core 2.12.2
com.fasterxml.jackson.core jackson-databind 2.12.2
com.fasterxml.jackson.dataformat jackson-dataformat-cbor 2.12.2
com.fasterxml.jackson.datatype jackson-datatype-joda 2.12.2
com.fasterxml.jackson.module jackson-module-paranamer 2.12.2
com.fasterxml.jackson.module jackson-module-scala_2.12 2.12.2
com.github.ben-manes.caffeine caffeine 2.3.4
com.github.fommil jniloader 1.1
com.github.fommil.netlib core 1.1.2
com.github.fommil.netlib native_ref-java 1.1
com.github.fommil.netlib native_ref-java-natives 1.1
com.github.fommil.netlib native_system-java 1.1
com.github.fommil.netlib native_system-java-natives 1.1
com.github.fommil.netlib netlib-native_ref-linux-x86_64-natives 1.1
com.github.fommil.netlib netlib-native_system-linux-x86_64-natives 1.1
com.github.luben zstd-jni 1.5.0-4
com.github.wendykierp JTransforms 3.1
com.google.code.findbugs jsr305 3.0.0
com.google.code.gson gson 2.8.6
com.google.crypto.tink tink 1.6.0
com.google.flatbuffers flatbuffers-java 1.9.0
com.google.guava guava 15.0
com.google.protobuf protobuf-java 2.6.1
com.h2database h2 1.4.195
com.helger profiler 1.1.1
com.jcraft jsch 0.1.50
com.jolbox bonecp 0.8.0.RELEASE
com.lihaoyi sourcecode_2.12 0.1.9
com.microsoft.sqlserver mssql-jdbc 9.2.1.jre8
com.microsoft.azure azure-data-lake-store-sdk 2.3.9
com.ning compress-lzf 1.0.3
com.sun.istack istack-commons-runtime 3.0.8
com.sun.mail javax.mail 1.5.2
com.tdunning json 1.8
com.thoughtworks.paranamer paranamer 2.8
com.trueaccord.lenses lenses_2.12 0.4.12
com.twitter chill-java 0.10.0
com.twitter chill_2.12 0.10.0
com.twitter util-app_2.12 7.1.0
com.twitter util-core_2.12 7.1.0
com.twitter util-function_2.12 7.1.0
com.twitter util-jvm_2.12 7.1.0
com.twitter util-lint_2.12 7.1.0
com.twitter util-registry_2.12 7.1.0
com.twitter util-stats_2.12 7.1.0
com.typesafe config 1.2.1
com.typesafe.scala-logging scala-logging_2.12 3.7.2
com.univocity univocity-parsers 2.9.1
com.zaxxer HikariCP 3.1.0
commons-cli commons-cli 1.2
commons-codec commons-codec 1.15
commons-collections commons-collections 3.2.2
commons-dbcp commons-dbcp 1.4
commons-fileupload commons-fileupload 1.3.3
commons-httpclient commons-httpclient 3.1
commons-io commons-io 2.8.0
commons-lang commons-lang 2.6
commons-logging commons-logging 1.1.3
commons-net commons-net 3.1
commons-pool commons-pool 1.5.4
dev.ludovic.netlib arpack 1.3.2
dev.ludovic.netlib blas 1.3.2
dev.ludovic.netlib lapack 1.3.2
hive-2.3__hadoop-3.2 jets3t-0.7 liball_deps_2.12
info.ganglia.gmetric4j gmetric4j 1.0.10
io.airlift aircompressor 0.19
io.delta delta-sharing-spark_2.12 0.1.0
io.dropwizard.metrics metrics-core 4.1.1
io.dropwizard.metrics metrics-graphite 4.1.1
io.dropwizard.metrics metrics-healthchecks 4.1.1
io.dropwizard.metrics metrics-jetty9 4.1.1
io.dropwizard.metrics metrics-jmx 4.1.1
io.dropwizard.metrics metrics-json 4.1.1
io.dropwizard.metrics metrics-jvm 4.1.1
io.dropwizard.metrics metrics-servlets 4.1.1
io.netty netty-all 4.1.63.Final
io.prometheus simpleclient 0.7.0
io.prometheus simpleclient_common 0.7.0
io.prometheus simpleclient_dropwizard 0.7.0
io.prometheus simpleclient_pushgateway 0.7.0
io.prometheus simpleclient_servlet 0.7.0
io.prometheus.jmx collector 0.12.0
jakarta.annotation jakarta.annotation-api 1.3.5
jakarta.servlet jakarta.servlet-api 4.0.3
jakarta.validation jakarta.validation-api 2.0.2
jakarta.ws.rs jakarta.ws.rs-api 2.1.6
javax.activation activation 1.1.1
javax.annotation javax.annotation-api 1.3.2
javax.el javax.el-api 2.2.4
javax.jdo jdo-api 3.0.1
javax.transaction jta 1.1
javax.transaction transaction-api 1.1
javax.xml.bind jaxb-api 2.2.2
javax.xml.stream stax-api 1.0-2
javolution javolution 5.5.1
jline jline 2.14.6
joda-time joda-time 2.10.5
log4j apache-log4j-extras 1.2.17
log4j log4j 1.2.17
maven-trees hive-2.3__hadoop-3.2 liball_deps_2.12
net.java.dev.jna jna 5.8.0
net.razorvine pyrolite 4.30
net.sf.jpam jpam 1.1
net.sf.opencsv opencsv 2.3
net.sf.supercsv super-csv 2.2.0
net.snowflake snowflake-ingest-sdk 0.9.6
net.snowflake snowflake-jdbc 3.13.3
net.snowflake spark-snowflake_2.12 2.9.0-spark_3.1
net.sourceforge.f2j arpack_combined_all 0.1
org.acplt.remotetea remotetea-oncrpc 1.1.2
org.antlr ST4 4.0.4
org.antlr antlr-runtime 3.5.2
org.antlr antlr4-runtime 4.8
org.antlr stringtemplate 3.2.1
org.apache.ant ant 1.9.2
org.apache.ant ant-jsch 1.9.2
org.apache.ant ant-launcher 1.9.2
org.apache.arrow arrow-format 2.0.0
org.apache.arrow arrow-memory-core 2.0.0
org.apache.arrow arrow-memory-netty 2.0.0
org.apache.arrow arrow-vector 2.0.0
org.apache.avro avro 1.10.2
org.apache.avro avro-ipc 1.10.2
org.apache.avro avro-mapred 1.10.2
org.apache.commons commons-compress 1.20
org.apache.commons commons-crypto 1.1.0
org.apache.commons commons-lang3 3.12.0
org.apache.commons commons-math3 3.4.1
org.apache.commons commons-text 1.6
org.apache.curator curator-client 2.13.0
org.apache.curator curator-framework 2.13.0
org.apache.curator curator-recipes 2.13.0
org.apache.derby derby 10.14.2.0
org.apache.hadoop hadoop-client-runtime 3.3.1
org.apache.hive hive-beeline 2.3.9
org.apache.hive hive-cli 2.3.9
org.apache.hive hive-jdbc 2.3.9
org.apache.hive hive-llap-client 2.3.9
org.apache.hive hive-llap-common 2.3.9
org.apache.hive hive-serde 2.3.9
org.apache.hive hive-shims 2.3.9
org.apache.hive hive-storage-api 2.7.2
org.apache.hive.shims hive-shims-0.23 2.3.9
org.apache.hive.shims hive-shims-common 2.3.9
org.apache.hive.shims hive-shims-scheduler 2.3.9
org.apache.htrace htrace-core4 4.1.0-incubating
org.apache.httpcomponents httpclient 4.5.13
org.apache.httpcomponents httpcore 4.4.12
org.apache.ivy ivy 2.5.0
org.apache.mesos mesos-shaded-protobuf 1.4.0
org.apache.orc orc-core 1.6.10
org.apache.orc orc-mapreduce 1.6.10
org.apache.orc orc-shims 1.6.10
org.apache.parquet parquet-column 1.12.0-databricks-0003
org.apache.parquet parquet-common 1.12.0-databricks-0003
org.apache.parquet parquet-encoding 1.12.0-databricks-0003
org.apache.parquet parquet-format-structures 1.12.0-databricks-0003
org.apache.parquet parquet-hadoop 1.12.0-databricks-0003
org.apache.parquet parquet-jackson 1.12.0-databricks-0003
org.apache.thrift libfb303 0.9.3
org.apache.thrift libthrift 0.12.0
org.apache.xbean xbean-asm9-shaded 4.20
org.apache.yetus audience-annotations 0.5.0
org.apache.zookeeper zookeeper 3.6.2
org.apache.zookeeper zookeeper-jute 3.6.2
org.checkerframework checker-qual 3.5.0
org.codehaus.jackson jackson-core-asl 1.9.13
org.codehaus.jackson jackson-mapper-asl 1.9.13
org.codehaus.janino commons-compiler 3.0.16
org.codehaus.janino janino 3.0.16
org.datanucleus datanucleus-api-jdo 4.2.4
org.datanucleus datanucleus-core 4.1.17
org.datanucleus datanucleus-rdbms 4.1.19
org.datanucleus javax.jdo 3.2.0-m3
org.eclipse.jetty jetty-client 9.4.40.v20210413
org.eclipse.jetty jetty-continuation 9.4.40.v20210413
org.eclipse.jetty jetty-http 9.4.40.v20210413
org.eclipse.jetty jetty-io 9.4.40.v20210413
org.eclipse.jetty jetty-jndi 9.4.40.v20210413
org.eclipse.jetty jetty-plus 9.4.40.v20210413
org.eclipse.jetty jetty-proxy 9.4.40.v20210413
org.eclipse.jetty jetty-security 9.4.40.v20210413
org.eclipse.jetty jetty-server 9.4.40.v20210413
org.eclipse.jetty jetty-servlet 9.4.40.v20210413
org.eclipse.jetty jetty-servlets 9.4.40.v20210413
org.eclipse.jetty jetty-util 9.4.40.v20210413
org.eclipse.jetty jetty-util-ajax 9.4.40.v20210413
org.eclipse.jetty jetty-webapp 9.4.40.v20210413
org.eclipse.jetty jetty-xml 9.4.40.v20210413
org.eclipse.jetty.websocket websocket-api 9.4.40.v20210413
org.eclipse.jetty.websocket websocket-client 9.4.40.v20210413
org.eclipse.jetty.websocket websocket-common 9.4.40.v20210413
org.eclipse.jetty.websocket websocket-server 9.4.40.v20210413
org.eclipse.jetty.websocket websocket-servlet 9.4.40.v20210413
org.fusesource.leveldbjni leveldbjni-all 1.8
org.glassfish.hk2 hk2-api 2.6.1
org.glassfish.hk2 hk2-locator 2.6.1
org.glassfish.hk2 hk2-utils 2.6.1
org.glassfish.hk2 osgi-resource-locator 1.0.3
org.glassfish.hk2.external aopalliance-repackaged 2.6.1
org.glassfish.hk2.external jakarta.inject 2.6.1
org.glassfish.jaxb jaxb-runtime 2.3.2
org.glassfish.jersey.containers jersey-container-servlet 2.34
org.glassfish.jersey.containers jersey-container-servlet-core 2.34
org.glassfish.jersey.core jersey-client 2.34
org.glassfish.jersey.core jersey-common 2.34
org.glassfish.jersey.core jersey-server 2.34
org.glassfish.jersey.inject jersey-hk2 2.34
org.hibernate.validator hibernate-validator 6.1.0.Final
org.javassist javassist 3.25.0-GA
org.jboss.logging jboss-logging 3.3.2.Final
org.jdbi jdbi 2.63.1
org.jetbrains annotations 17.0.0
org.joda joda-convert 1.7
org.jodd jodd-core 3.5.2
org.json4s json4s-ast_2.12 3.7.0-M11
org.json4s json4s-core_2.12 3.7.0-M11
org.json4s json4s-jackson_2.12 3.7.0-M11
org.json4s json4s-scalap_2.12 3.7.0-M11
org.lz4 lz4-java 1.7.1
org.mariadb.jdbc mariadb-java-client 2.2.5
org.objenesis objenesis 2.5.1
org.postgresql postgresql 42.2.19
org.roaringbitmap RoaringBitmap 0.9.14
org.roaringbitmap shims 0.9.14
org.rocksdb rocksdbjni 6.20.3
org.rosuda.REngine REngine 2.1.0
org.scala-lang scala-compiler_2.12 2.12.14
org.scala-lang scala-library_2.12 2.12.14
org.scala-lang scala-reflect_2.12 2.12.14
org.scala-lang.modules scala-collection-compat_2.12 2.4.3
org.scala-lang.modules scala-parser-combinators_2.12 1.1.2
org.scala-lang.modules scala-xml_2.12 1.2.0
org.scala-sbt test-interface 1.0
org.scalacheck scalacheck_2.12 1.14.2
org.scalactic scalactic_2.12 3.0.8
org.scalanlp breeze-macros_2.12 1.0
org.scalanlp breeze_2.12 1.0
org.scalatest scalatest_2.12 3.0.8
org.slf4j jcl-over-slf4j 1.7.30
org.slf4j jul-to-slf4j 1.7.30
org.slf4j slf4j-api 1.7.30
org.slf4j slf4j-log4j12 1.7.30
org.spark-project.spark unused 1.0.0
org.springframework spring-core 4.1.4.RELEASE
org.springframework spring-test 4.1.4.RELEASE
org.threeten threeten-extra 1.5.0
org.tukaani xz 1.8
org.typelevel algebra_2.12 2.0.0-M2
org.typelevel cats-kernel_2.12 2.0.0-M4
org.typelevel machinist_2.12 0.6.8
org.typelevel macro-compat_2.12 1.1.1
org.typelevel spire-macros_2.12 0.17.0-M1
org.typelevel spire-platform_2.12 0.17.0-M1
org.typelevel spire-util_2.12 0.17.0-M1
org.typelevel spire_2.12 0.17.0-M1
org.wildfly.openssl wildfly-openssl 1.0.7.Final
org.xerial sqlite-jdbc 3.8.11.2
org.xerial.snappy snappy-java 1.1.8.4
org.yaml snakeyaml 1.24
oro oro 2.0.8
pl.edu.icm JLargeArrays 1.5
software.amazon.ion ion-java 1.0.2
stax stax-api 1.0.1