Del via

Databricks Runtime 15.1

  • Artikkel

The following release notes provide information about Databricks Runtime 15.1, powered by Apache Spark 3.5.0.

Databricks released this version in April 2024.

Tip

To see release notes for Databricks Runtime versions that have reached end-of-support (EoS), see End-of-support Databricks Runtime release notes. The EoS Databricks Runtime versions have been retired and might not be updated.

Breaking changes

Support for Databricks Runtime 15.0 and Databricks Runtime 15.0 for Machine Learning ends on May 31, 2024 because an upgrade of the pandas library from 1.5.3 to 2.0.3 in 15.0 caused a breaking change in multiple Apache Spark APIs. For details of the breaking changes, see The pandas Python library is downgraded to 1.5.3. To resolve this incompatibility, Databricks has released Databricks Runtime 15.1 and Databricks Runtime 15.1 for Machine Learning. The functionality supported by these releases is the same as the 15.0 releases, but with the pandas version downgraded to 1.5.3.

Any existing clusters or jobs that use Databricks Runtime 15.0 releases will continue to work but will not receive product or security updates. You can no longer configure new compute that uses Databricks Runtime 15.0 or Databricks Runtime 15.0 ML in the Databricks UI. If you are not using features or APIs that require pandas 2.0+, Databricks recommends switching those workloads to Databricks Runtime 15.1.

If you are using features or APIs available only in pandas 2.0+ and are not using any functions affected by this version, Databricks recommends switching to Databricks Runtime 15.1 and upgrading the default pandas package using cluster or notebook-scoped libraries. For example, to upgrade the pandas library in a Databricks notebook, use the following command:

%pip install pandas==2.0.3

The pandas Python library is downgraded to 1.5.3

In this release, the pandas library is downgraded from version 2.0.3 to 1.5.3. This downgrade was required because incompatibilities in the 2.0.3 version of pandas caused errors in some pandas API on Spark functions. The following lists the functions affected by the 2.0.3 version of pandas:

  • pyspark.date_range: The closed parameter is missing, causing errors when this function is called by existing code.
  • The functionality of GroupBy with MultiIndex columns is changed, causing unexpected behavior.
  • Some datetime attributes of DatetimeIndex (for example day, month, year, and so forth) return int32 data types instead of int64 data types.

The PySpark spark.sql() function now fails for invalid argument types

In Databricks Runtime 15.1 and later, the args parameter for named or positional parameters passed to the spark.sql() function must be a dictionary or list. If any other object type is passed, the following error is raised: PySparkTypeError: [INVALID_TYPE] Argument `args` should not be a <arg_type>.

Storing libraries in DBFS root is deprecated and disabled by default

To improve the security of libraries in a Azure Databricks workspace, storing library files in the DBFS root is deprecated and disabled by default starting with Databricks Runtime 15.1. Databricks recommends uploading all libraries, which includes Python libraries, JAR files, and Spark connectors, to workspace files, Unity Catalog volumes, or using a package repository. See Recommendations for uploading libraries.

To re-enable storing libraries in the DBFS root, set the following Spark configuration parameter: spark.databricks.driver.dbfsLibraryInstallationAllowed true.

The default Python version is upgraded from 3.10 to 3.11

With Databricks Runtime 15.1, the default Python version is 3.11.0. For the list of upgraded Python libraries, see Library upgrades.

JDK 11 is removed

As previously announced, JDK 11 is removed from Databricks Runtime 15.1 and later. Databricks recommends upgrading to JDK 17 when upgrading to Databricks Runtime 15.1 and later. See Databricks SDK for Java.

The Python REPL now terminates gracefully

With Databricks Runtime 15.1, the notebook’s Python REPL process now terminates gracefully at exit. This causes any atexit hooks to be honored. For example, in a workflow with multiple Python notebook tasks, atexit hooks registered in the first task execute before the second Python notebook task runs.

New features and improvements

The * (star) clause is now supported in the WHERE clause

You can now use the star (*) clause in the WHERE clause to reference all columns from the SELECT list. For example, SELECT * FROM VALUES(1, 2) AS T(a1, a2) WHERE 1 IN(T.*).

The spark.sql.json.enablePartialResults configuration is now enabled by default

The Spark configuration spark.sql.json.enablePartialResults is now enabled by default, allowing the JSON parser to correctly handle parsing errors for complex types such as struct, map, and array without dropping the remaining fields. This change fixes a consistency issue for JSON data sources with column pruning.

Cluster libraries now support requirements.txt files

You can now install cluster libraries using a requirements.txt file stored in a workspace file or Unity Catalog volume. In Single user and Shared access mode clusters, the requirements.txt file can reference other files. In No isolation shared access mode clusters, only PyPI packages are supported. See Cluster libraries.

Databricks CLI now available in the web terminal

You can now use the Databricks CLI from the web terminal in the notebook. The notebook must be attached to a cluster in Single user or No isolation shared access mode. For details, see Use web terminal and Databricks CLI.

Add default Python repositories to cluster configurations

In Databricks Runtime 15.1 and later, you can configure global pip index-url and extra-index-url parameters for cluster and notebook-scoped library installation when configuring a cluster or defining a cluster policy. To do so, set the environment variables DATABRICKS_PIP_INDEX_URL and DATABRICKS_PIP_EXTRA_INDEX_URL.

Credential passthrough and Hive metastore table access controls are deprecated

Credential passthrough and Hive metastore table access controls are legacy data governance models. Upgrade to Unity Catalog to simplify the security and governance of your data by providing a central place to administer and audit data access across multiple workspaces in your account. See What is Unity Catalog?.

Support for credential passthrough and Hive metastore table access controls will be removed in an upcoming DBR version.

Library upgrades

  • Upgraded Python libraries:
    • black from 22.6.0 to 23.3.0
    • boto3 from 1.24.28 to 1.34.39
    • botocore from 1.27.96 to 1.34.39
    • certifi from 2022.12.7 to 2023.7.22
    • cryptography from 39.0.1 to 41.0.3
    • databricks-sdk from 0.1.6 to 0.20.0
    • distlib from 0.3.7 to 0.3.8
    • googleapis-common-protos from 1.62.0 to 1.63.0
    • grpcio from 1.48.2 to 1.60.0
    • grpcio-status from 1.48.1 to 1.60.0
    • importlib-metadata from 4.6.4 to 6.0.0
    • ipykernel from 6.25.0 to 6.25.1
    • ipython from 8.14.0 to 8.15.0
    • ipywidgets from 7.7.2 to 8.0.4
    • jupyter_core from 5.2.0 to 5.3.0
    • jupyterlab-widgets from 1.0.0 to 3.0.5
    • matplotlib from 3.7.0 to 3.7.2
    • pip from 22.3.1 to 23.2.1
    • platformdirs from 2.5.2 to 3.10.0
    • protobuf from 4.24.0 to 4.24.1
    • pyarrow from 8.0.0 to 14.0.1
    • Pygments from 2.11.2 to 2.15.1
    • pyodbc from 4.0.32 to 4.0.38
    • requests from 2.28.1 to 2.31.0
    • s3transfer from 0.6.2 to 0.10.0
    • scikit-learn from 1.1.1 to 1.3.0
    • scipy from 1.10.0 to 1.11.1
    • setuptools from 65.6.3 to 68.0.0
    • statsmodels from 0.13.5 to 0.14.0
    • tenacity from 8.1.0 to 8.2.2
    • tornado from 6.1 to 6.3.2
    • typing_extensions from 4.4.0 to 4.7.1
    • urllib3 from 1.26.14 to 1.26.16
    • virtualenv from 20.16.7 to 20.24.2
    • widgetsnbextension from 3.6.1 to 4.0.5
    • zipp from 1.0.0 to 3.11.0
  • Upgraded R libraries:
    • arrow from 12.0.1 to 14.0.0.2
    • askpass from 1.1 to 1.2.0
    • base from 4.3.1 to 4.3.2
    • brew from 1.0-8 to 1.0-10
    • brio from 1.1.3 to 1.1.4
    • bslib from 0.5.0 to 0.6.1
    • cli from 3.6.1 to 3.6.2
    • commonmark from 1.9.0 to 1.9.1
    • compiler from 4.3.1 to 4.3.2
    • config from 0.3.1 to 0.3.2
    • cpp11 from 0.4.4 to 0.4.7
    • credentials from 1.3.2 to 2.0.1
    • curl from 5.0.1 to 5.2.0
    • data.table from 1.14.8 to 1.15.0
    • datasets from 4.3.1 to 4.3.2
    • DBI from 1.1.3 to 1.2.1
    • dbplyr from 2.3.3 to 2.4.0
    • desc from 1.4.2 to 1.4.3
    • digest from 0.6.33 to 0.6.34
    • dplyr from 1.1.2 to 1.1.4
    • e1071 from 1.7-13 to 1.7-14
    • evaluate from 0.21 to 0.23
    • fansi from 1.0.4 to 1.0.6
    • fontawesome from 0.5.1 to 0.5.2
    • fs from 1.6.2 to 1.6.3
    • future from 1.33.0 to 1.33.1
    • future.apply from 1.11.0 to 1.11.1
    • gargle from 1.5.1 to 1.5.2
    • gert from 1.9.2 to 2.0.1
    • ggplot2 from 3.4.2 to 3.4.4
    • glmnet from 4.1-7 to 4.1-8
    • glue from 1.6.2 to 1.7.0
    • graphics from 4.3.1 to 4.3.2
    • grDevices from 4.3.1 to 4.3.2
    • grid from 4.3.1 to 4.3.2
    • gtable from 0.3.3 to 0.3.4
    • hardhat from 1.3.0 to 1.3.1
    • haven from 2.5.3 to 2.5.4
    • htmltools from 0.5.5 to 0.5.7
    • htmlwidgets from 1.6.2 to 1.6.4
    • httpuv from 1.6.11 to 1.6.14
    • httr from 1.4.6 to 1.4.7
    • httr2 from 0.2.3 to 1.0.0
    • jsonlite from 1.8.7 to 1.8.8
    • knitr from 1.43 to 1.45
    • labeling from 0.4.2 to 0.4.3
    • later from 1.3.1 to 1.3.2
    • lava from 1.7.2.1 to 1.7.3
    • lifecycle from 1.0.3 to 1.0.4
    • listenv from 0.9.0 to 0.9.1
    • lubridate from 1.9.2 to 1.9.3
    • markdown from 1.7 to 1.12
    • methods from 4.3.1 to 4.3.2
    • openssl from 2.0.6 to 2.1.1
    • parallel from 4.3.1 to 4.3.2
    • pkgbuild from 1.4.2 to 1.4.3
    • pkgload from 1.3.2.1 to 1.3.4
    • plyr from 1.8.8 to 1.8.9
    • prettyunits from 1.1.1 to 1.2.0
    • pROC from 1.18.4 to 1.18.5
    • processx from 3.8.2 to 3.8.3
    • prodlim from 2023.03.31 to 2023.08.28
    • progress from 1.2.2 to 1.2.3
    • progressr from 0.13.0 to 0.14.0
    • promises from 1.2.0.1 to 1.2.1
    • ps from 1.7.5 to 1.7.6
    • purrr from 1.0.1 to 1.0.2
    • ragg from 1.2.5 to 1.2.7
    • Rcpp from 1.0.11 to 1.0.12
    • RcppEigen from 0.3.3.9.3 to 0.3.3.9.4
    • readr from 2.1.4 to 2.1.5
    • recipes from 1.0.6 to 1.0.9
    • rematch from 1.0.1 to 2.0.0
    • remotes from 2.4.2 to 2.4.2.1
    • reprex from 2.0.2 to 2.1.0
    • rlang from 1.1.1 to 1.1.3
    • rmarkdown from 2.23 to 2.25
    • RODBC from 1.3-20 to 1.3-23
    • roxygen2 from 7.2.3 to 7.3.1
    • rprojroot from 2.0.3 to 2.0.4
    • Rserve from 1.8-11 to 1.8-13
    • RSQLite from 2.3.1 to 2.3.5
    • sass from 0.4.6 to 0.4.8
    • scales from 1.2.1 to 1.3.0
    • shiny from 1.7.4.1 to 1.8.0
    • sparklyr from 1.8.1 to 1.8.4
    • splines from 4.3.1 to 4.3.2
    • stats from 4.3.1 to 4.3.2
    • stats4 from 4.3.1 to 4.3.2
    • stringi from 1.7.12 to 1.8.3
    • stringr from 1.5.0 to 1.5.1
    • systemfonts from 1.0.4 to 1.0.5
    • tcltk from 4.3.1 to 4.3.2
    • testthat from 3.1.10 to 3.2.1
    • textshaping from 0.3.6 to 0.3.7
    • tidyr from 1.3.0 to 1.3.1
    • timechange from 0.2.0 to 0.3.0
    • timeDate from 4022.108 to 4032.109
    • tinytex from 0.45 to 0.49
    • tools from 4.3.1 to 4.3.2
    • utf8 from 1.2.3 to 1.2.4
    • utils from 4.3.1 to 4.3.2
    • uuid from 1.1-0 to 1.2-0
    • vctrs from 0.6.3 to 0.6.5
    • vroom from 1.6.3 to 1.6.5
    • waldo from 0.5.1 to 0.5.2
    • withr from 2.5.0 to 3.0.0
    • xfun from 0.39 to 0.41
    • xml2 from 1.3.5 to 1.3.6
    • yaml from 2.3.7 to 2.3.8
    • zip from 2.3.0 to 2.3.1
  • Upgraded Java libraries:
    • com.fasterxml.jackson.datatype.jackson-datatype-jsr310 from 2.15.1 to 2.16.0
    • com.google.flatbuffers.flatbuffers-java from 1.12.0 to 23.5.26
    • com.typesafe.config from 1.2.1 to 1.4.3
    • org.apache.ant.ant from 1.9.16 to 1.10.11
    • org.apache.ant.ant-jsch from 1.9.16 to 1.10.11
    • org.apache.ant.ant-launcher from 1.9.16 to 1.10.11
    • org.apache.arrow.arrow-format from 12.0.1 to 15.0.0
    • org.apache.arrow.arrow-memory-core from 12.0.1 to 15.0.0
    • org.apache.arrow.arrow-memory-netty from 12.0.1 to 15.0.0
    • org.apache.arrow.arrow-vector from 12.0.1 to 15.0.0
    • org.apache.avro.avro from 1.11.2 to 1.11.3
    • org.apache.avro.avro-ipc from 1.11.2 to 1.11.3
    • org.apache.avro.avro-mapred from 1.11.2 to 1.11.3
    • org.apache.logging.log4j.log4j-1.2-api from 2.20.0 to 2.22.1
    • org.apache.logging.log4j.log4j-api from 2.20.0 to 2.22.1
    • org.apache.logging.log4j.log4j-core from 2.20.0 to 2.22.1
    • org.apache.logging.log4j.log4j-slf4j2-impl from 2.20.0 to 2.22.1
    • org.postgresql.postgresql from 42.6.0 to 42.6.1

Apache Spark

Databricks Runtime 15.1 includes Apache Spark 3.5.0. This release includes all Spark fixes and improvements included in Databricks Runtime 14.3 LTS, as well as the following additional bug fixes and improvements made to Spark:

  • [SPARK-45527] [DBRRM-805][ES-1073714] Revert “[SC-151626][CORE] Use fraction to do the resource calculation”
  • [SPARK-47102] [DBRRM-803][SC-158253][SQL] Add the COLLATION_ENABLED config flag
  • [SPARK-46973] [SC-158140][DBRRM-777][SQL] Skip V2 table lookup when a table is in V1 table cache
  • [SPARK-46808] [SC-154113][PYTHON] Refine error classes in Python with automatic sorting function
  • [SPARK-47251] [SC-158291][PYTHON] Block invalid types from the args argument for sql command
  • [SPARK-47251] [SC-158121][PYTHON] Block invalid types from the args argument for sql command
  • [SPARK-47108] [SC-157204][CORE] Set derby.connection.requireAuthentication to false explicitly in CLIs
  • [SPARK-45599] [SC-157537][CORE] Use object equality in OpenHashSet
  • [SPARK-47099] [SC-157352][SQL] Use ordinalNumber to uniformly set the value of paramIndex for the error class UNEXPECTED_INPUT_TYPE
  • [SPARK-46558] [SC-151363][CONNECT] Extract a helper function to eliminate the duplicate code that retrieves MessageParameters from ErrorParams in GrpcExceptionConverter
  • [SPARK-43117] [SC-156177][CONNECT] Make ProtoUtils.abbreviate support repeated fields
  • [SPARK-46342] [SC-150283][SQL] Replace IllegalStateException by SparkException.internalError in sql
  • [SPARK-47123] [SC-157412][CORE] JDBCRDD does not correctly handle errors in getQueryOutputSchema
  • [SPARK-47189] [SC-157667][SQL] Tweak column error names and text
  • [SPARK-45789] [SC-157101][SQL] Support DESCRIBE TABLE for clustering columns
  • [SPARK-43256] [SC-157699][SQL] Remove error class _LEGACY_ERROR_TEMP_2021
  • [SPARK-47201] [SC-157782][PYTHON][CONNECT] sameSemantics checks input types
  • [SPARK-47183] [SC-157660][PYTHON] Fix the error class for sameSemantics
  • [SPARK-47179] [SC-157663][SQL] Improve error message from SparkThrowableSuite for better debuggability
  • [SPARK-46965] [SC-155791][CORE] Check logType in Utils.getLog
  • [SPARK-46832] [SC-156774][SQL] Introducing Collate and Collation expressions
  • [SPARK-46946] [SC-155721][SQL] Supporting broadcast of multiple filtering keys in DynamicPruning
  • [SPARK-47079] [VARIANT-22][SC-157598][PYTHON][SQL][CONNECT] Add Variant type info to PySpark
  • [SPARK-47101] [SC-157360][SQL] Allow comma to be used in top-level column names and remove check nested type definition in HiveExternalCatalog.verifyDataSchema
  • [SPARK-47173] [SC-157571][SS][UI] Fix a typo in streaming UI explanation
  • [SPARK-47113] [SC-157348][CORE] Revert S3A endpoint fixup logic of SPARK-35878
  • [SPARK-47130] [SC-157462][CORE] Use listStatus to bypass block location info when cleaning driver logs
  • [SPARK-43259] [SC-157368][SQL] Assign a name to the error class _LEGACY_ERROR_TEMP_2024
  • [SPARK-47104] [SC-157355][SQL] TakeOrderedAndProjectExec should initialize the unsafe projection
  • [SPARK-47137] [SC-157463][PYTHON][CONNECT] Add getAll to spark.conf for feature parity with Scala
  • [SPARK-46924] [SC-154905][CORE] Fix Load New button in Master/HistoryServer Log UI
  • [SPARK-47069] [SC-157370][PYTHON][CONNECT] Introduce spark.profile.show/dump for SparkSession-based profiling
  • [SPARK-46812] [SC-157075][SQL][PYTHON] Make mapInPandas / mapInArrow support ResourceProfile
  • [SPARK-46833] [SC-155866][SQL] Collations - Introducing CollationFactory which provides comparison and hashing rules for supported collations
  • [SPARK-47057] [SC-157036][PYTHON] Reeanble MyPy data test
  • [SPARK-46934] [SC-157084][SQL] Read/write roundtrip for struct type with special characters with HMS
  • [SPARK-46727] [SC-153472][SQL] Port classifyException() in JDBC dialects on error classes
  • [SPARK-46964] [SC-155740][SQL] Change the signature of the hllInvalidLgK query execution error to take an integer as 4th argument
  • [SPARK-46949] [SC-155719][SQL] Support CHAR/VARCHAR through ResolveDefaultColumns
  • [SPARK-46972] [SC-155888][SQL] Fix asymmetrical replacement for char/varchar in V2SessionCatalog.createTable
  • [SPARK-47014] [SC-156778][PYTHON][CONNECT] Implement methods dumpPerfProfiles and dumpMemoryProfiles of SparkSession
  • [SPARK-46689] [SC-156221][SPARK-46690][PYTHON][CONNECT] Support v2 profiling in group/cogroup applyInPandas/applyInArrow
  • [SPARK-47040] [SC-156773][CONNECT] Allow Spark Connect Server Script to wait
  • [SPARK-46688] [SC-156113][SPARK-46691][PYTHON][CONNECT] Support v2 profiling in aggregate Pandas UDFs
  • [SPARK-47004] [SC-156499][CONNECT] Added more tests to ClientStreamingQuerySuite to increase Scala client test coverage
  • [SPARK-46994] [SC-156169][PYTHON] Refactor PythonWrite to prepare for supporting python data source streaming write
  • [SPARK-46979] [SC-156587][SS] Add support for specifying key and value encoder separately and also for each col family in RocksDB state store provider
  • [SPARK-46960] [SC-155940][SS] Testing Multiple Input Streams with TransformWithState operator
  • [SPARK-46966] [SC-156112][Python] Add UDTF API for ‘analyze’ method to indicate subset of input table columns to select
  • [SPARK-46895] [SC-155950][CORE] Replace Timer with single thread scheduled executor
  • [SPARK-46967] [SC-155815][CORE][UI] Hide Thread Dump and Heap Histogram of Dead executors in Executors UI
  • [SPARK-46512] [SC-155826][CORE] Optimize shuffle reading when both sort and combine are used.
  • [SPARK-46958] [SC-155942][SQL] Add missing timezone to coerce default values
  • [SPARK-46754] [SC-153830][SQL][AVRO] Fix compression code resolution in avro table definition and write options
  • [SPARK-46728] [SC-154207][PYTHON] Check Pandas installation properly
  • [SPARK-43403] [SC-154216][UI] Ensure old SparkUI in HistoryServer has been detached before loading new one
  • [SPARK-39910] [SC-156203][SQL] Delegate path qualification to filesystem during DataSource file path globbing
  • [SPARK-47000] [SC-156123] Revert “[SC-156123][CORE] Use getTotalMemorySize…
  • [SPARK-46819] [SC-154560][CORE] Move error categories and states into JSON
  • [SPARK-46997] [SC-156175][CORE] Enable spark.worker.cleanup.enabled by default
  • [SPARK-46865] [SC-156168][SS] Add Batch Support for TransformWithState Operator
  • [SPARK-46987] [SC-156040][CONNECT] ProtoUtils.abbreviate avoid unnecessary setField operation
  • [SPARK-46970] [SC-155816][CORE] Rewrite OpenHashSet#hasher with pattern matching
  • [SPARK-46984] [SC-156024][PYTHON] Remove pyspark.copy_func
  • [SPARK-47000] [SC-156123][CORE] Use getTotalMemorySize in WorkerArguments
  • [SPARK-46980] [SC-155914][SQL][MINOR] Avoid using internal APIs in dataframe end-to-end tests
  • [SPARK-46931] Revert “[SC-155661][PS] Implement {Frame, Series}.to_hdf”
  • [SPARK-46618] [SC-153828][SQL] Improve error messages for DATA_SOURCE_NOT_FOUND error
  • [SPARK-46849] [SC-154586][SQL] Run optimizer on CREATE TABLE column defaults
  • [SPARK-46911] [SC-155724][SS] Adding deleteIfExists operator to StatefulProcessorHandleImpl
  • [SPARK-43273] [SQL] Support lz4raw compression codec for Parquet
  • [SPARK-47002] [SC-156223][Python] Return better error message if UDTF ‘analyze’ method ‘orderBy’ field accidentally returns a list of strings
  • [SPARK-46890] [SC-155802][SQL] Fix CSV parsing bug with existence default values and column pruning
  • [SPARK-46977] [SC-155892][CORE] A failed request to obtain a token from one NameNode should not skip subsequent token requests
  • [SPARK-46608] [SC-151806][SQL] Restore backward compatibility of JdbcDialect.classifyException
  • [SPARK-46908] [SC-155702][SQL] Support star clause in WHERE clause
  • [SPARK-46908] [SC-155702][SQL] Support star clause in WHERE clause
  • [SPARK-46852] [SC-155665][SS] Remove use of explicit key encoder and pass it implicitly to the operator for transformWithState operator
  • [SPARK-46864] [SC-155669][SS] Onboard Arbitrary StateV2 onto New Error Class Framework
  • [SPARK-45807] [SC-155706][SQL] Return View after calling replaceView(..)
  • [SPARK-46899] [SC-154651][CORE] Remove POST APIs from MasterWebUI when spark.ui.killEnabled is false
  • [SPARK-46868] [SC-154539][CORE] Support Spark Worker Log UI
  • [SPARK-46931] [SC-155661][PS] Implement {Frame, Series}.to_hdf
  • [SPARK-46940] [SC-155667][CORE] Remove unused updateSparkConfigFromProperties and isAbsoluteURI in o.a.s.u.Utils
  • [SPARK-46929] [SC-155659][CORE][CONNECT][SS] Use ThreadUtils.shutdown to close thread pools
  • [SPARK-46400] [SC-155658][CORE][SQL] When there are corrupted files in the local maven repo, skip this cache and try again
  • [SPARK-46932] [SC-155655] Clean up the imports in pyspark.pandas.test_*
  • [SPARK-46683] [SC-154120] Write a subquery generator that generates subqueries permutations to increase testing coverage
  • [SPARK-46914] [SC-154776][UI] Shorten app name in the summary table on the History Page
  • [SPARK-46831] [SC-154699][SQL] Collations - Extending StringType and PhysicalStringType with collationId field
  • [SPARK-46816] [SC-154415][SS] Add base support for new arbitrary state management operator, single valueState type, multiple state variables and underlying support for column families for RocksDBStateStoreProvider with/without changelog checkpointing
  • [SPARK-46925] [SC-154890][PYTHON][CONNECT] Add a warning that instructs to install memory_profiler for memory profiling
  • [SPARK-46927] [SC-154904][PYTHON] Make assertDataFrameEqual work properly without PyArrow
  • [SPARK-46021] [SC-148987][CORE] Support cancel future jobs belonging to a job group
  • [SPARK-46747] [SC-154789][SQL] Avoid scan in getTableExistsQuery for JDBC Dialects
  • [SPARK-46904] [SC-154704][UI] Fix display issue of History UI summary
  • [SPARK-46870] [SC-154541][CORE] Support Spark Master Log UI
  • [SPARK-46893] [SC-154780][UI] Remove inline scripts from UI descriptions
  • [SPARK-46910] [SC-154760][PYTHON] Eliminate JDK Requirement in PySpark Installation
  • [SPARK-46907] [SC-154736][CORE] Show driver log location in Spark History Server
  • [SPARK-46902] [SC-154661][UI] Fix Spark History Server UI for using un-exported setAppLimit
  • [SPARK-46687] [SC-154725][PYTHON][CONNECT] Basic support of SparkSession-based memory profiler
  • [SPARK-46623] [SC-153831][CORE][MLLIB][SQL] Replace SimpleDateFormat with DateTimeFormatter
  • [SPARK-46875] [SC-154627][SQL] When the mode is null, a NullPointException should not be thrown
  • [SPARK-46872] [SC-154543][CORE] Recover log-view.js to be non-module
  • [SPARK-46883] [SC-154629][CORE] Support /json/clusterutilization API
  • [SPARK-46886] [SC-154630][CORE] Enable spark.ui.prometheus.enabled by default
  • [SPARK-46873] [SC-154638][SS] Do not recreate new StreamingQueryManager for the same Spark Session
  • [SPARK-46829] [SC-154419][CORE] Remove createExecutorEnv from SparkEnv
  • [SPARK-46827] [SC-154280][CORE] Make RocksDBPersistenceEngine to support a symbolic link
  • [SPARK-46903] [SC-154662][CORE] Support Spark History Server Log UI
  • [SPARK-46874] [SC-154646][PYTHON] Remove pyspark.pandas dependency from assertDataFrameEqual
  • [SPARK-46889] [SC-154639][CORE] Validate spark.master.ui.decommission.allow.mode setting
  • [SPARK-46850] [SC-154443][SQL] Convert _LEGACY_ERROR_TEMP_2102 to UNSUPPORTED_DATATYPE
  • [SPARK-46704] [SC-153323][CORE][UI] Fix MasterPage to sort Running Drivers table by Duration column correctly
  • [SPARK-46796] [SC-154279][SS] Ensure the correct remote files (mentioned in metadata.zip) are used on RocksDB version load
  • [SPARK-46888] [SC-154636][CORE] Fix Master to reject /workers/kill/ requests if decommission is disabled
  • [SPARK-46818] [SC-154422][SQL] Improve error messages for Range with non-foldable inputs
  • [SPARK-46898] [SC-154649][CONNECT] Simplify the protobuf function transformation in Planner
  • [SPARK-46828] [SC-154413][SQL] Remove the invalid assertion of remote mode for spark sql shell
  • [SPARK-46733] [SC-154274][CORE] Simplify the BlockManager by the exit operation only depend on interrupt thread.
  • [SPARK-46777] [SC-154016][SS] Refactor StreamingDataSourceV2Relation catalyst structure to be more on-par with the batch version
  • [SPARK-46515] [SC-151716] Add MONTHNAME function
  • [SPARK-46823] [SC-154276][CONNECT][PYTHON] LocalDataToArrowConversion should check the nullability
  • [SPARK-46787] [SC-154404][CONNECT] bloomFilter function should throw AnalysisException for invalid input
  • [SPARK-46779] [SC-154045][SQL] InMemoryRelation instances of the same cached plan should be semantically equivalent
  • [SPARK-45827] [SC-153973] Disallow partitioning on Variant column
  • [SPARK-46797] [SC-154085][CORE] Rename spark.deploy.spreadOut to spark.deploy.spreadOutApps
  • [SPARK-46094] [SC-153421] Support Executor JVM Profiling
  • [SPARK-46746] [SC-153557][SQL][AVRO] Attach codec extension to avro datasource files
  • [SPARK-46698] [SC-153288][CORE] Replace Timer with single thread scheduled executor for ConsoleProgressBar.
  • [SPARK-46629] [SC-153835] Fix for STRUCT type DDL not picking up nullability and comment
  • [SPARK-46750] [SC-153772][CONNECT][PYTHON] DataFrame APIs code clean up
  • [SPARK-46769] [SC-153979][SQL] Refine timestamp related schema inference
  • [SPARK-46765] [SC-153904][PYTHON][CONNECT] Make shuffle specify the datatype of seed
  • [SPARK-46579] [SC-151481][SQL] Redact JDBC url in errors and logs
  • [SPARK-46686] [SC-153875][PYTHON][CONNECT] Basic support of SparkSession based Python UDF profiler
  • [SPARK-46748] Revert “[SC-153800][CORE] Remove *slav**.sh scri…
  • [SPARK-46707] [SC-153833][SQL] Added throwable field to expressions to improve predicate pushdown
  • [SPARK-46519] [SC-151278][SQL] Clear unused error classes from error-classes.json file
  • [SPARK-46677] [SC-153426][SQL][CONNECT] Fix dataframe["*"] resolution
  • [SPARK-46317] [SC-150184][PYTHON][CONNECT] Match minor behaviour matching in SparkSession with full test coverage
  • [SPARK-46748] [SC-153800][CORE] Remove *slav**.sh scripts
  • [SPARK-46663] [SC-153490][PYTHON] Disable memory profiler for pandas UDFs with iterators
  • [SPARK-46410] [SC-150776][SQL] Assign error classes/subclasses to JdbcUtils.classifyException
  • [SPARK-46277] [SC-150126][PYTHON] Validate startup urls with the config being set
  • [SPARK-46612] [SC-153536][SQL] Do not convert array type string retrieved from jdbc driver
  • [SPARK-46254] [SC-149788][PYTHON] Remove stale Python 3.8/3.7 version checking
  • [SPARK-46490] [SC-151335][SQL] Require error classes in SparkThrowable sub-classes
  • [SPARK-46383] [SC-153274][SC-147443][WARMFIX] Reduce Driver Heap Usage by Reducing the Lifespan of TaskInfo.accumulables()
  • [SPARK-46541] [SC-153109][SQL][CONNECT] Fix the ambiguous column reference in self join
  • [SPARK-46381] [SC-150495][SQL] Migrate sub-classes of AnalysisException to error classes
  • [SPARK-46351] [SC-150335][SQL] Require an error class in AnalysisException
  • [SPARK-46220] [SC-149689][SQL] Restrict charsets in decode()
  • [SPARK-46369] [SC-150340][CORE] Remove kill link from RELAUNCHING drivers in MasterPage
  • [SPARK-46052] [SC-153284][CORE] Remove function TaskScheduler.killAllTaskAttempts
  • [SPARK-46536] [SC-153164][SQL] Support GROUP BY calendar_interval_type
  • [SPARK-46675] [SC-153209][SQL] Remove unused inferTimestampNTZ in ParquetReadSupport
  • [SPARK-46717] [SC-153423][CORE] Simplify ReloadingX509TrustManager by the exit operation only depend on interrupt thread.
  • [SPARK-46722] [SC-153438][CONNECT] Add a test regarding to backward compatibility check for StreamingQueryListener in Spark Connect (Scala/PySpark)
  • [SPARK-46187] [SC-149580][SQL] Align codegen and non-codegen implementation of StringDecode
  • [SPARK-46258] [SC-149799][CORE] Add RocksDBPersistenceEngine
  • [SPARK-46216] [SC-149676][CORE] Improve FileSystemPersistenceEngine to support compressions
  • [SPARK-46189] [SC-149567][PS][SQL] Perform comparisons and arithmetic between same types in various Pandas aggregate functions to avoid interpreted mode errors
  • [SPARK-46184] [SC-149563][CORE][SQL][CONNECT][MLLIB] Reduce stack depth by replace Option.isDefined with Option.isEmpty
  • [SPARK-46700] [SC-153329][CORE] Count the last spilling for the shuffle disk spilling bytes metric
  • [SPARK-45642] [SC-150286][CORE][SQL] Fix FileSystem.isFile & FileSystem.isDirectory is deprecated
  • [SPARK-46640] [SC-153272][SQL] Fix RemoveRedundantAlias by excluding subquery attributes
  • [SPARK-46681] [SC-153287][CORE] Refactor ExecutorFailureTracker#maxNumExecutorFailures to avoid calculating defaultMaxNumExecutorFailures when MAX_EXECUTOR_FAILURES is configured
  • [SPARK-46695] [SC-153289][SQL][HIVE] Always setting hive.execution.engine to mr
  • [SPARK-46325] [SC-150228][CONNECT] Remove unnecessary override functions when constructing WrappedCloseableIterator in ResponseValidator#wrapIterator
  • [SPARK-46232] [SC-149699][PYTHON] Migrate all remaining ValueError into PySpark error framework.
  • [SPARK-46547] [SC-153174][SS] Swallow non-fatal exception in maintenance task to avoid deadlock between maintenance thread and streaming aggregation operator
  • [SPARK-46169] [SC-149373][PS] Assign appropriate JIRA numbers for missing parameters from DataFrame API.
  • [SPARK-45857] [SC-148096][SQL] Enforce the error classes in sub-classes of AnalysisException

Databricks ODBC/JDBC driver support

Databricks supports ODBC/JDBC drivers released in the past 2 years. Please download the recently released drivers and upgrade (download ODBC, download JDBC).

System environment

  • Operating System: Ubuntu 22.04.4 LTS
  • Java: Zulu 8.74.0.17-CA-linux64
  • Scala: 2.12.15
  • Python: 3.11.0
  • R: 4.3.2
  • Delta Lake: 3.1.0

Installed Python libraries

Library Version Library Version Library Version
asttokens 2.0.5 astunparse 1.6.3 azure-core 1.30.1
azure-storage-blob 12.19.1 azure-storage-file-datalake 12.14.0 backcall 0.2.0
black 23.3.0 blinker 1.4 boto3 1.34.39
botocore 1.34.39 cachetools 5.3.3 certifi 2023.7.22
cffi 1.15.1 chardet 4.0.0 charset-normalizer 2.0.4
click 8.0.4 cloudpickle 2.2.1 comm 0.1.2
contourpy 1.0.5 cryptography 41.0.3 cycler 0.11.0
Cython 0.29.32 databricks-sdk 0.20.0 dbus-python 1.2.18
debugpy 1.6.7 decorator 5.1.1 distlib 0.3.8
entrypoints 0.4 executing 0.8.3 facets-overview 1.1.1
filelock 3.13.1 fonttools 4.25.0 gitdb 4.0.11
GitPython 3.1.42 google-api-core 2.17.1 google-auth 2.28.2
google-cloud-core 2.4.1 google-cloud-storage 2.15.0 google-crc32c 1.5.0
google-resumable-media 2.7.0 googleapis-common-protos 1.63.0 grpcio 1.60.0
grpcio-status 1.60.0 httplib2 0.20.2 idna 3.4
importlib-metadata 6.0.0 ipyflow-core 0.0.198 ipykernel 6.25.1
ipython 8.15.0 ipywidgets 8.0.4 isodate 0.6.1
jedi 0.18.1 jeepney 0.7.1 jmespath 0.10.0
joblib 1.2.0 jupyter_client 7.4.9 jupyter_core 5.3.0
jupyterlab-widgets 3.0.5 keyring 23.5.0 kiwisolver 1.4.4
launchpadlib 1.10.16 lazr.restfulclient 0.14.4 lazr.uri 1.0.6
matplotlib 3.7.2 matplotlib-inline 0.1.6 mlflow-skinny 2.11.1
more-itertools 8.10.0 mypy-extensions 0.4.3 nest-asyncio 1.5.6
numpy 1.23.5 oauthlib 3.2.0 packaging 23.2
pandas 1.5.3 parso 0.8.3 pathspec 0.10.3
patsy 0.5.3 pexpect 4.8.0 pickleshare 0.7.5
Pillow 9.4.0 pip 23.2.1 platformdirs 3.10.0
plotly 5.9.0 prompt-toolkit 3.0.36 protobuf 4.24.1
psutil 5.9.0 psycopg2 2.9.3 ptyprocess 0.7.0
pure-eval 0.2.2 pyarrow 14.0.1 pyasn1 0.4.8
pyasn1-modules 0.2.8 pyccolo 0.0.52 pycparser 2.21
pydantic 1.10.6 Pygments 2.15.1 PyGObject 3.42.1
PyJWT 2.3.0 pyodbc 4.0.38 pyparsing 3.0.9
python-dateutil 2.8.2 python-lsp-jsonrpc 1.1.1 pytz 2022.7
PyYAML 6.0 pyzmq 23.2.0 requests 2.31.0
rsa 4.9 s3transfer 0.10.0 scikit-learn 1.3.0
scipy 1.11.1 seaborn 0.12.2 SecretStorage 3.3.1
setuptools 68.0.0 six 1.16.0 smmap 5.0.1
sqlparse 0.4.4 ssh-import-id 5.11 stack-data 0.2.0
statsmodels 0.14.0 tenacity 8.2.2 threadpoolctl 2.2.0
tokenize-rt 4.2.1 tornado 6.3.2 traitlets 5.7.1
typing_extensions 4.7.1 tzdata 2022.1 ujson 5.4.0
unattended-upgrades 0.1 urllib3 1.26.16 virtualenv 20.24.2
wadllib 1.3.6 wcwidth 0.2.5 wheel 0.38.4
widgetsnbextension 4.0.5 zipp 3.11.0

Installed R libraries

R libraries are installed from the Posit Package Manager CRAN snapshot on 2023-02-10.

Library Version Library Version Library Version
arrow 14.0.0.2 askpass 1.2.0 assertthat 0.2.1
backports 1.4.1 base 4.3.2 base64enc 0.1-3
bigD 0.2.0 bit 4.0.5 bit64 4.0.5
bitops 1.0-7 blob 1.2.4 boot 1.3-28
brew 1.0-10 brio 1.1.4 broom 1.0.5
bslib 0.6.1 cachem 1.0.8 callr 3.7.3
caret 6.0-94 cellranger 1.1.0 chron 2.3-61
class 7.3-22 cli 3.6.2 clipr 0.8.0
clock 0.7.0 cluster 2.1.4 codetools 0.2-19
colorspace 2.1-0 commonmark 1.9.1 compiler 4.3.2
config 0.3.2 conflicted 1.2.0 cpp11 0.4.7
crayon 1.5.2 credentials 2.0.1 curl 5.2.0
data.table 1.15.0 datasets 4.3.2 DBI 1.2.1
dbplyr 2.4.0 desc 1.4.3 devtools 2.4.5
diagram 1.6.5 diffobj 0.3.5 digest 0.6.34
downlit 0.4.3 dplyr 1.1.4 dtplyr 1.3.1
e1071 1.7-14 ellipsis 0.3.2 evaluate 0.23
fansi 1.0.6 farver 2.1.1 fastmap 1.1.1
fontawesome 0.5.2 forcats 1.0.0 foreach 1.5.2
foreign 0.8-85 forge 0.2.0 fs 1.6.3
future 1.33.1 future.apply 1.11.1 gargle 1.5.2
generics 0.1.3 gert 2.0.1 ggplot2 3.4.4
gh 1.4.0 git2r 0.33.0 gitcreds 0.1.2
glmnet 4.1-8 globals 0.16.2 glue 1.7.0
googledrive 2.1.1 googlesheets4 1.1.1 gower 1.0.1
graphics 4.3.2 grDevices 4.3.2 grid 4.3.2
gridExtra 2.3 gsubfn 0.7 gt 0.10.1
gtable 0.3.4 hardhat 1.3.1 haven 2.5.4
highr 0.10 hms 1.1.3 htmltools 0.5.7
htmlwidgets 1.6.4 httpuv 1.6.14 httr 1.4.7
httr2 1.0.0 ids 1.0.1 ini 0.3.1
ipred 0.9-14 isoband 0.2.7 iterators 1.0.14
jquerylib 0.1.4 jsonlite 1.8.8 juicyjuice 0.1.0
KernSmooth 2.23-21 knitr 1.45 labeling 0.4.3
later 1.3.2 lattice 0.21-8 lava 1.7.3
lifecycle 1.0.4 listenv 0.9.1 lubridate 1.9.3
magrittr 2.0.3 markdown 1.12 MASS 7.3-60
Matrix 1.5-4.1 memoise 2.0.1 methods 4.3.2
mgcv 1.8-42 mime 0.12 miniUI 0.1.1.1
mlflow 2.10.0 ModelMetrics 1.2.2.2 modelr 0.1.11
munsell 0.5.0 nlme 3.1-163 nnet 7.3-19
numDeriv 2016.8-1.1 openssl 2.1.1 parallel 4.3.2
parallelly 1.36.0 pillar 1.9.0 pkgbuild 1.4.3
pkgconfig 2.0.3 pkgdown 2.0.7 pkgload 1.3.4
plogr 0.2.0 plyr 1.8.9 praise 1.0.0
prettyunits 1.2.0 pROC 1.18.5 processx 3.8.3
prodlim 2023.08.28 profvis 0.3.8 progress 1.2.3
progressr 0.14.0 promises 1.2.1 proto 1.0.0
proxy 0.4-27 ps 1.7.6 purrr 1.0.2
R6 2.5.1 ragg 1.2.7 randomForest 4.7-1.1
rappdirs 0.3.3 rcmdcheck 1.4.0 RColorBrewer 1.1-3
Rcpp 1.0.12 RcppEigen 0.3.3.9.4 reactable 0.4.4
reactR 0.5.0 readr 2.1.5 readxl 1.4.3
recipes 1.0.9 rematch 2.0.0 rematch2 2.1.2
remotes 2.4.2.1 reprex 2.1.0 reshape2 1.4.4
rlang 1.1.3 rmarkdown 2.25 RODBC 1.3-23
roxygen2 7.3.1 rpart 4.1.21 rprojroot 2.0.4
Rserve 1.8-13 RSQLite 2.3.5 rstudioapi 0.15.0
rversions 2.1.2 rvest 1.0.3 sass 0.4.8
scales 1.3.0 selectr 0.4-2 sessioninfo 1.2.2
shape 1.4.6 shiny 1.8.0 sourcetools 0.1.7-1
sparklyr 1.8.4 spatial 7.3-15 splines 4.3.2
sqldf 0.4-11 SQUAREM 2021.1 stats 4.3.2
stats4 4.3.2 stringi 1.8.3 stringr 1.5.1
survival 3.5-5 swagger 3.33.1 sys 3.4.2
systemfonts 1.0.5 tcltk 4.3.2 testthat 3.2.1
textshaping 0.3.7 tibble 3.2.1 tidyr 1.3.1
tidyselect 1.2.0 tidyverse 2.0.0 timechange 0.3.0
timeDate 4032.109 tinytex 0.49 tools 4.3.2
tzdb 0.4.0 urlchecker 1.0.1 usethis 2.2.2
utf8 1.2.4 utils 4.3.2 uuid 1.2-0
V8 4.4.1 vctrs 0.6.5 viridisLite 0.4.2
vroom 1.6.5 waldo 0.5.2 whisker 0.4.1
withr 3.0.0 xfun 0.41 xml2 1.3.6
xopen 1.0.0 xtable 1.8-4 yaml 2.3.8
zeallot 0.1.0 zip 2.3.1

Installed Java and Scala libraries (Scala 2.12 cluster version)

Group ID Artifact ID Version
antlr antlr 2.7.7
com.amazonaws amazon-kinesis-client 1.12.0
com.amazonaws aws-java-sdk-autoscaling 1.12.390
com.amazonaws aws-java-sdk-cloudformation 1.12.390
com.amazonaws aws-java-sdk-cloudfront 1.12.390
com.amazonaws aws-java-sdk-cloudhsm 1.12.390
com.amazonaws aws-java-sdk-cloudsearch 1.12.390
com.amazonaws aws-java-sdk-cloudtrail 1.12.390
com.amazonaws aws-java-sdk-cloudwatch 1.12.390
com.amazonaws aws-java-sdk-cloudwatchmetrics 1.12.390
com.amazonaws aws-java-sdk-codedeploy 1.12.390
com.amazonaws aws-java-sdk-cognitoidentity 1.12.390
com.amazonaws aws-java-sdk-cognitosync 1.12.390
com.amazonaws aws-java-sdk-config 1.12.390
com.amazonaws aws-java-sdk-core 1.12.390
com.amazonaws aws-java-sdk-datapipeline 1.12.390
com.amazonaws aws-java-sdk-directconnect 1.12.390
com.amazonaws aws-java-sdk-directory 1.12.390
com.amazonaws aws-java-sdk-dynamodb 1.12.390
com.amazonaws aws-java-sdk-ec2 1.12.390
com.amazonaws aws-java-sdk-ecs 1.12.390
com.amazonaws aws-java-sdk-efs 1.12.390
com.amazonaws aws-java-sdk-elasticache 1.12.390
com.amazonaws aws-java-sdk-elasticbeanstalk 1.12.390
com.amazonaws aws-java-sdk-elasticloadbalancing 1.12.390
com.amazonaws aws-java-sdk-elastictranscoder 1.12.390
com.amazonaws aws-java-sdk-emr 1.12.390
com.amazonaws aws-java-sdk-glacier 1.12.390
com.amazonaws aws-java-sdk-glue 1.12.390
com.amazonaws aws-java-sdk-iam 1.12.390
com.amazonaws aws-java-sdk-importexport 1.12.390
com.amazonaws aws-java-sdk-kinesis 1.12.390
com.amazonaws aws-java-sdk-kms 1.12.390
com.amazonaws aws-java-sdk-lambda 1.12.390
com.amazonaws aws-java-sdk-logs 1.12.390
com.amazonaws aws-java-sdk-machinelearning 1.12.390
com.amazonaws aws-java-sdk-opsworks 1.12.390
com.amazonaws aws-java-sdk-rds 1.12.390
com.amazonaws aws-java-sdk-redshift 1.12.390
com.amazonaws aws-java-sdk-route53 1.12.390
com.amazonaws aws-java-sdk-s3 1.12.390
com.amazonaws aws-java-sdk-ses 1.12.390
com.amazonaws aws-java-sdk-simpledb 1.12.390
com.amazonaws aws-java-sdk-simpleworkflow 1.12.390
com.amazonaws aws-java-sdk-sns 1.12.390
com.amazonaws aws-java-sdk-sqs 1.12.390
com.amazonaws aws-java-sdk-ssm 1.12.390
com.amazonaws aws-java-sdk-storagegateway 1.12.390
com.amazonaws aws-java-sdk-sts 1.12.390
com.amazonaws aws-java-sdk-support 1.12.390
com.amazonaws aws-java-sdk-swf-libraries 1.11.22
com.amazonaws aws-java-sdk-workspaces 1.12.390
com.amazonaws jmespath-java 1.12.390
com.clearspring.analytics stream 2.9.6
com.databricks Rserve 1.8-3
com.databricks databricks-sdk-java 0.17.1
com.databricks jets3t 0.7.1-0
com.databricks.scalapb compilerplugin_2.12 0.4.15-10
com.databricks.scalapb scalapb-runtime_2.12 0.4.15-10
com.esotericsoftware kryo-shaded 4.0.2
com.esotericsoftware minlog 1.3.0
com.fasterxml classmate 1.3.4
com.fasterxml.jackson.core jackson-annotations 2.15.2
com.fasterxml.jackson.core jackson-core 2.15.2
com.fasterxml.jackson.core jackson-databind 2.15.2
com.fasterxml.jackson.dataformat jackson-dataformat-cbor 2.15.2
com.fasterxml.jackson.dataformat jackson-dataformat-yaml 2.15.2
com.fasterxml.jackson.datatype jackson-datatype-joda 2.15.2
com.fasterxml.jackson.datatype jackson-datatype-jsr310 2.16.0
com.fasterxml.jackson.module jackson-module-paranamer 2.15.2
com.fasterxml.jackson.module jackson-module-scala_2.12 2.15.2
com.github.ben-manes.caffeine caffeine 2.9.3
com.github.fommil jniloader 1.1
com.github.fommil.netlib native_ref-java 1.1
com.github.fommil.netlib native_ref-java 1.1-natives
com.github.fommil.netlib native_system-java 1.1
com.github.fommil.netlib native_system-java 1.1-natives
com.github.fommil.netlib netlib-native_ref-linux-x86_64 1.1-natives
com.github.fommil.netlib netlib-native_system-linux-x86_64 1.1-natives
com.github.luben zstd-jni 1.5.5-4
com.github.wendykierp JTransforms 3.1
com.google.code.findbugs jsr305 3.0.0
com.google.code.gson gson 2.10.1
com.google.crypto.tink tink 1.9.0
com.google.errorprone error_prone_annotations 2.10.0
com.google.flatbuffers flatbuffers-java 23.5.26
com.google.guava guava 15.0
com.google.protobuf protobuf-java 2.6.1
com.helger profiler 1.1.1
com.jcraft jsch 0.1.55
com.jolbox bonecp 0.8.0.RELEASE
com.lihaoyi sourcecode_2.12 0.1.9
com.microsoft.azure azure-data-lake-store-sdk 2.3.9
com.microsoft.sqlserver mssql-jdbc 11.2.2.jre8
com.ning compress-lzf 1.1.2
com.sun.mail javax.mail 1.5.2
com.sun.xml.bind jaxb-core 2.2.11
com.sun.xml.bind jaxb-impl 2.2.11
com.tdunning json 1.8
com.thoughtworks.paranamer paranamer 2.8
com.trueaccord.lenses lenses_2.12 0.4.12
com.twitter chill-java 0.10.0
com.twitter chill_2.12 0.10.0
com.twitter util-app_2.12 7.1.0
com.twitter util-core_2.12 7.1.0
com.twitter util-function_2.12 7.1.0
com.twitter util-jvm_2.12 7.1.0
com.twitter util-lint_2.12 7.1.0
com.twitter util-registry_2.12 7.1.0
com.twitter util-stats_2.12 7.1.0
com.typesafe config 1.4.3
com.typesafe.scala-logging scala-logging_2.12 3.7.2
com.uber h3 3.7.3
com.univocity univocity-parsers 2.9.1
com.zaxxer HikariCP 4.0.3
commons-cli commons-cli 1.5.0
commons-codec commons-codec 1.16.0
commons-collections commons-collections 3.2.2
commons-dbcp commons-dbcp 1.4
commons-fileupload commons-fileupload 1.5
commons-httpclient commons-httpclient 3.1
commons-io commons-io 2.13.0
commons-lang commons-lang 2.6
commons-logging commons-logging 1.1.3
commons-pool commons-pool 1.5.4
dev.ludovic.netlib arpack 3.0.3
dev.ludovic.netlib blas 3.0.3
dev.ludovic.netlib lapack 3.0.3
info.ganglia.gmetric4j gmetric4j 1.0.10
io.airlift aircompressor 0.25
io.delta delta-sharing-client_2.12 1.0.4
io.dropwizard.metrics metrics-annotation 4.2.19
io.dropwizard.metrics metrics-core 4.2.19
io.dropwizard.metrics metrics-graphite 4.2.19
io.dropwizard.metrics metrics-healthchecks 4.2.19
io.dropwizard.metrics metrics-jetty9 4.2.19
io.dropwizard.metrics metrics-jmx 4.2.19
io.dropwizard.metrics metrics-json 4.2.19
io.dropwizard.metrics metrics-jvm 4.2.19
io.dropwizard.metrics metrics-servlets 4.2.19
io.netty netty-all 4.1.96.Final
io.netty netty-buffer 4.1.96.Final
io.netty netty-codec 4.1.96.Final
io.netty netty-codec-http 4.1.96.Final
io.netty netty-codec-http2 4.1.96.Final
io.netty netty-codec-socks 4.1.96.Final
io.netty netty-common 4.1.96.Final
io.netty netty-handler 4.1.96.Final
io.netty netty-handler-proxy 4.1.96.Final
io.netty netty-resolver 4.1.96.Final
io.netty netty-tcnative-boringssl-static 2.0.61.Final-linux-aarch_64
io.netty netty-tcnative-boringssl-static 2.0.61.Final-linux-x86_64
io.netty netty-tcnative-boringssl-static 2.0.61.Final-osx-aarch_64
io.netty netty-tcnative-boringssl-static 2.0.61.Final-osx-x86_64
io.netty netty-tcnative-boringssl-static 2.0.61.Final-windows-x86_64
io.netty netty-tcnative-classes 2.0.61.Final
io.netty netty-transport 4.1.96.Final
io.netty netty-transport-classes-epoll 4.1.96.Final
io.netty netty-transport-classes-kqueue 4.1.96.Final
io.netty netty-transport-native-epoll 4.1.96.Final
io.netty netty-transport-native-epoll 4.1.96.Final-linux-aarch_64
io.netty netty-transport-native-epoll 4.1.96.Final-linux-x86_64
io.netty netty-transport-native-kqueue 4.1.96.Final-osx-aarch_64
io.netty netty-transport-native-kqueue 4.1.96.Final-osx-x86_64
io.netty netty-transport-native-unix-common 4.1.96.Final
io.prometheus simpleclient 0.7.0
io.prometheus simpleclient_common 0.7.0
io.prometheus simpleclient_dropwizard 0.7.0
io.prometheus simpleclient_pushgateway 0.7.0
io.prometheus simpleclient_servlet 0.7.0
io.prometheus.jmx collector 0.12.0
jakarta.annotation jakarta.annotation-api 1.3.5
jakarta.servlet jakarta.servlet-api 4.0.3
jakarta.validation jakarta.validation-api 2.0.2
jakarta.ws.rs jakarta.ws.rs-api 2.1.6
javax.activation activation 1.1.1
javax.el javax.el-api 2.2.4
javax.jdo jdo-api 3.0.1
javax.transaction jta 1.1
javax.transaction transaction-api 1.1
javax.xml.bind jaxb-api 2.2.11
javolution javolution 5.5.1
jline jline 2.14.6
joda-time joda-time 2.12.1
net.java.dev.jna jna 5.8.0
net.razorvine pickle 1.3
net.sf.jpam jpam 1.1
net.sf.opencsv opencsv 2.3
net.sf.supercsv super-csv 2.2.0
net.snowflake snowflake-ingest-sdk 0.9.6
net.sourceforge.f2j arpack_combined_all 0.1
org.acplt.remotetea remotetea-oncrpc 1.1.2
org.antlr ST4 4.0.4
org.antlr antlr-runtime 3.5.2
org.antlr antlr4-runtime 4.9.3
org.antlr stringtemplate 3.2.1
org.apache.ant ant 1.10.11
org.apache.ant ant-jsch 1.10.11
org.apache.ant ant-launcher 1.10.11
org.apache.arrow arrow-format 15.0.0
org.apache.arrow arrow-memory-core 15.0.0
org.apache.arrow arrow-memory-netty 15.0.0
org.apache.arrow arrow-vector 15.0.0
org.apache.avro avro 1.11.3
org.apache.avro avro-ipc 1.11.3
org.apache.avro avro-mapred 1.11.3
org.apache.commons commons-collections4 4.4
org.apache.commons commons-compress 1.23.0
org.apache.commons commons-crypto 1.1.0
org.apache.commons commons-lang3 3.12.0
org.apache.commons commons-math3 3.6.1
org.apache.commons commons-text 1.10.0
org.apache.curator curator-client 2.13.0
org.apache.curator curator-framework 2.13.0
org.apache.curator curator-recipes 2.13.0
org.apache.datasketches datasketches-java 3.1.0
org.apache.datasketches datasketches-memory 2.0.0
org.apache.derby derby 10.14.2.0
org.apache.hadoop hadoop-client-runtime 3.3.6
org.apache.hive hive-beeline 2.3.9
org.apache.hive hive-cli 2.3.9
org.apache.hive hive-jdbc 2.3.9
org.apache.hive hive-llap-client 2.3.9
org.apache.hive hive-llap-common 2.3.9
org.apache.hive hive-serde 2.3.9
org.apache.hive hive-shims 2.3.9
org.apache.hive hive-storage-api 2.8.1
org.apache.hive.shims hive-shims-0.23 2.3.9
org.apache.hive.shims hive-shims-common 2.3.9
org.apache.hive.shims hive-shims-scheduler 2.3.9
org.apache.httpcomponents httpclient 4.5.14
org.apache.httpcomponents httpcore 4.4.16
org.apache.ivy ivy 2.5.1
org.apache.logging.log4j log4j-1.2-api 2.22.1
org.apache.logging.log4j log4j-api 2.22.1
org.apache.logging.log4j log4j-core 2.22.1
org.apache.logging.log4j log4j-slf4j2-impl 2.22.1
org.apache.orc orc-core 1.9.2-shaded-protobuf
org.apache.orc orc-mapreduce 1.9.2-shaded-protobuf
org.apache.orc orc-shims 1.9.2
org.apache.thrift libfb303 0.9.3
org.apache.thrift libthrift 0.12.0
org.apache.ws.xmlschema xmlschema-core 2.3.0
org.apache.xbean xbean-asm9-shaded 4.23
org.apache.yetus audience-annotations 0.13.0
org.apache.zookeeper zookeeper 3.6.3
org.apache.zookeeper zookeeper-jute 3.6.3
org.checkerframework checker-qual 3.31.0
org.codehaus.jackson jackson-core-asl 1.9.13
org.codehaus.jackson jackson-mapper-asl 1.9.13
org.codehaus.janino commons-compiler 3.0.16
org.codehaus.janino janino 3.0.16
org.datanucleus datanucleus-api-jdo 4.2.4
org.datanucleus datanucleus-core 4.1.17
org.datanucleus datanucleus-rdbms 4.1.19
org.datanucleus javax.jdo 3.2.0-m3
org.eclipse.collections eclipse-collections 11.1.0
org.eclipse.collections eclipse-collections-api 11.1.0
org.eclipse.jetty jetty-client 9.4.52.v20230823
org.eclipse.jetty jetty-continuation 9.4.52.v20230823
org.eclipse.jetty jetty-http 9.4.52.v20230823
org.eclipse.jetty jetty-io 9.4.52.v20230823
org.eclipse.jetty jetty-jndi 9.4.52.v20230823
org.eclipse.jetty jetty-plus 9.4.52.v20230823
org.eclipse.jetty jetty-proxy 9.4.52.v20230823
org.eclipse.jetty jetty-security 9.4.52.v20230823
org.eclipse.jetty jetty-server 9.4.52.v20230823
org.eclipse.jetty jetty-servlet 9.4.52.v20230823
org.eclipse.jetty jetty-servlets 9.4.52.v20230823
org.eclipse.jetty jetty-util 9.4.52.v20230823
org.eclipse.jetty jetty-util-ajax 9.4.52.v20230823
org.eclipse.jetty jetty-webapp 9.4.52.v20230823
org.eclipse.jetty jetty-xml 9.4.52.v20230823
org.eclipse.jetty.websocket websocket-api 9.4.52.v20230823
org.eclipse.jetty.websocket websocket-client 9.4.52.v20230823
org.eclipse.jetty.websocket websocket-common 9.4.52.v20230823
org.eclipse.jetty.websocket websocket-server 9.4.52.v20230823
org.eclipse.jetty.websocket websocket-servlet 9.4.52.v20230823
org.fusesource.leveldbjni leveldbjni-all 1.8
org.glassfish.hk2 hk2-api 2.6.1
org.glassfish.hk2 hk2-locator 2.6.1
org.glassfish.hk2 hk2-utils 2.6.1
org.glassfish.hk2 osgi-resource-locator 1.0.3
org.glassfish.hk2.external aopalliance-repackaged 2.6.1
org.glassfish.hk2.external jakarta.inject 2.6.1
org.glassfish.jersey.containers jersey-container-servlet 2.40
org.glassfish.jersey.containers jersey-container-servlet-core 2.40
org.glassfish.jersey.core jersey-client 2.40
org.glassfish.jersey.core jersey-common 2.40
org.glassfish.jersey.core jersey-server 2.40
org.glassfish.jersey.inject jersey-hk2 2.40
org.hibernate.validator hibernate-validator 6.1.7.Final
org.ini4j ini4j 0.5.4
org.javassist javassist 3.29.2-GA
org.jboss.logging jboss-logging 3.3.2.Final
org.jdbi jdbi 2.63.1
org.jetbrains annotations 17.0.0
org.joda joda-convert 1.7
org.jodd jodd-core 3.5.2
org.json4s json4s-ast_2.12 3.7.0-M11
org.json4s json4s-core_2.12 3.7.0-M11
org.json4s json4s-jackson_2.12 3.7.0-M11
org.json4s json4s-scalap_2.12 3.7.0-M11
org.lz4 lz4-java 1.8.0
org.mlflow mlflow-spark_2.12 2.9.1
org.objenesis objenesis 2.5.1
org.postgresql postgresql 42.6.1
org.roaringbitmap RoaringBitmap 0.9.45
org.roaringbitmap shims 0.9.45
org.rocksdb rocksdbjni 8.3.2
org.rosuda.REngine REngine 2.1.0
org.scala-lang scala-compiler_2.12 2.12.15
org.scala-lang scala-library_2.12 2.12.15
org.scala-lang scala-reflect_2.12 2.12.15
org.scala-lang.modules scala-collection-compat_2.12 2.11.0
org.scala-lang.modules scala-parser-combinators_2.12 1.1.2
org.scala-lang.modules scala-xml_2.12 1.2.0
org.scala-sbt test-interface 1.0
org.scalacheck scalacheck_2.12 1.14.2
org.scalactic scalactic_2.12 3.2.15
org.scalanlp breeze-macros_2.12 2.1.0
org.scalanlp breeze_2.12 2.1.0
org.scalatest scalatest-compatible 3.2.15
org.scalatest scalatest-core_2.12 3.2.15
org.scalatest scalatest-diagrams_2.12 3.2.15
org.scalatest scalatest-featurespec_2.12 3.2.15
org.scalatest scalatest-flatspec_2.12 3.2.15
org.scalatest scalatest-freespec_2.12 3.2.15
org.scalatest scalatest-funspec_2.12 3.2.15
org.scalatest scalatest-funsuite_2.12 3.2.15
org.scalatest scalatest-matchers-core_2.12 3.2.15
org.scalatest scalatest-mustmatchers_2.12 3.2.15
org.scalatest scalatest-propspec_2.12 3.2.15
org.scalatest scalatest-refspec_2.12 3.2.15
org.scalatest scalatest-shouldmatchers_2.12 3.2.15
org.scalatest scalatest-wordspec_2.12 3.2.15
org.scalatest scalatest_2.12 3.2.15
org.slf4j jcl-over-slf4j 2.0.7
org.slf4j jul-to-slf4j 2.0.7
org.slf4j slf4j-api 2.0.7
org.slf4j slf4j-simple 1.7.25
org.threeten threeten-extra 1.7.1
org.tukaani xz 1.9
org.typelevel algebra_2.12 2.0.1
org.typelevel cats-kernel_2.12 2.1.1
org.typelevel spire-macros_2.12 0.17.0
org.typelevel spire-platform_2.12 0.17.0
org.typelevel spire-util_2.12 0.17.0
org.typelevel spire_2.12 0.17.0
org.wildfly.openssl wildfly-openssl 1.1.3.Final
org.xerial sqlite-jdbc 3.42.0.0
org.xerial.snappy snappy-java 1.1.10.3
org.yaml snakeyaml 2.0
oro oro 2.0.8
pl.edu.icm JLargeArrays 1.5
software.amazon.cryptools AmazonCorrettoCryptoProvider 1.6.1-linux-x86_64
software.amazon.ion ion-java 1.0.2
stax stax-api 1.0.1