Databricks Runtime 17.0

2025-06-25

The following release notes provide information about Databricks Runtime 17.0, powered by Apache Spark 4.0.0.

Databricks released this version in June 2025.

Tip

To see release notes for Databricks Runtime versions that have reached end-of-support (EoS), see End-of-support Databricks Runtime release notes. The EoS Databricks Runtime versions have been retired and might not be updated.

DBR 17.0 new and updated features

Spark ML support on standard compute
SQL procedure support
Set a default collation for SQL Functions
Recursive common table expressions (rCTE) support
ANSI SQL enabled by default
PySpark and Spark Connect now support the DataFrames df.mergeInto API
Support ALL CATALOGS in SHOW SCHEMAS
Liquid clustering now compacts deletion vectors more efficiently
Allow non-deterministic expressions in UPDATE/INSERT column values for MERGE operations
Ignore and rescue empty structs for AutoLoader ingestion (especially Avro)
Change Delta MERGE Python and Scala APIs to return DataFrame instead of Unit
Support VAR keyword for declaring and dropping SQL variables
Upgrade Apache Parquet libraries
Support workload identity federation for Google Cloud Pub/Sub

Spark ML support on standard compute

Standard access mode (formerly shared access mode) now supports Spark ML in PySpark (pyspark.ml) and MLflow for Spark (mlflow.spark). For hyperparameter tuning, Databricks recommends using Optuna and Joblib Spark on standard compute.

SQL procedure support

SQL scripts can now be encapsulated in a procedure stored as a reusable asset in Unity Catalog. You can create a procedure using the CREATE PROCEDURE command, and then call it using the CALL command.

Set a default collation for SQL Functions

Using the new DEFAULT COLLATION clause in the CREATE FUNCTION command defines the default collation used for STRING parameters, the return type, and STRING literals in the function body.

Recursive common table expressions (rCTE) support

Azure Databricks now supports navigation of hierarchical data using recursive common table expressions (rCTEs). Use a self-referencing CTE with UNION ALL to follow the recursive relationship.

ANSI SQL enabled by default

The default SQL dialect is now ANSI SQL. ANSI SQL is a well-established standard and will help protect users from unexpected or incorrect results. Read the Databricks ANSI enablement guide for more information.

PySpark and Spark Connect now support the DataFrames `df.mergeInto` API

PySpark and Spark Connect now support the df.mergeInto API, which was previously only available for Scala.

Support `ALL CATALOGS` in `SHOW` SCHEMAS

The SHOW SCHEMAS syntax is updated to accept the following syntax:

SHOW SCHEMAS [ { FROM | IN } { catalog_name | ALL CATALOGS } ] [ [ LIKE ] pattern ]

When ALL CATALOGS is specified in a a SHOW query, the execution iterates through all active catalogs that support namespaces using the catalog manager (DsV2). For each catalog, it includes the top-level namespaces.

The output attributes and schema of the command have been modified to add a catalog column indicating the catalog of the corresponding namespace. The new column is added to the end of the output attributes, as shown below:

Previous output

| Namespace        |
|------------------|
| test-namespace-1 |
| test-namespace-2 |

New output

| Namespace        | Catalog        |
|------------------|----------------|
| test-namespace-1 | test-catalog-1 |
| test-namespace-2 | test-catalog-2 |

Liquid clustering now compacts deletion vectors more efficiently

Delta tables with Liquid clustering now apply physical changes from deletion vectors more efficiently when OPTIMIZE is running. For more details, see Apply changes to Parquet data files.

Allow non-deterministic expressions in `UPDATE`/`INSERT` column values for `MERGE` operations

Azure Databricks now allows the use of non-deterministic expressions in updated and inserted column values of MERGE operations. However, non-deterministic expressions in the conditions of MERGE statements are not supported.

For example, you can now generate dynamic or random values for columns:

MERGE INTO target USING source
ON target.key = source.key
WHEN MATCHED THEN UPDATE SET target.value = source.value + rand()

This can be helpful for data privacy to obfuscate actual data while preserving the data properties (such as mean values or other computed columns).

Ignore and rescue empty structs for AutoLoader ingestion (especially Avro)

Auto Loader now rescues Avro data types with an empty schema since Delta table does not support ingestiom of empty struct-type data.

Change Delta MERGE Python and Scala APIs to return DataFrame instead of Unit

The Scala and Python MERGE APIs (such as DeltaMergeBuilder) now also return a DataFrame like the SQL API does, with the same results.

Support VAR keyword for declaring and dropping SQL variables

SQL syntax for declaring and dropping variables now supports the VAR keyword in addition to VARIABLE. This change unifies the syntax across all variable-related operations, which improves consistency and reduces confusion for users who already use VAR when setting variables.

Upgrade Apache Parquet libraries

Apache Parquet libraries have been upgraded from version 1.13.1 to 1.15.1 to ensure compatibility with Spark 4.0. This update includes performance improvements, bug fixes, and enhanced Parquet feature support across the following modules:

parquet-column
parquet-common
parquet-encoding
parquet-format-structures
parquet-hadoop
parquet-jackson

Support workload identity federation for Google Cloud Pub/Sub

You can now use workload identity federation to securely connect to Google Cloud Pub/Sub from Databricks without requiring service account keys. This simplifies authentication for streaming and data ingestion workloads that integrate with Pub/Sub.

Behavioral changes

Databricks File System (DBFS) custom CA certificates are no longer supported
Behavioral change for the Auto Loader incremental directory listing option
Removed the "True cache misses" section in Spark UI
Removed the "Cache Metadata Manager Peak Disk Usage" metric in the Spark UI
Removed the "Rescheduled cache miss bytes" section in the Spark UI
CREATE VIEW column-level clauses now throw errors when the clause would only apply to materialized views

Databricks File System (DBFS) custom CA certificates are no longer supported

As part of the ongoing effort to deprecate data storage in the DBFS root and DBFS mounts, DBFS custom CA certificates are not supported in Databricks Runtime 17.0 and above. For recommendations on working with files, see Work with files on Azure Databricks.

Behavioral change for the Auto Loader incremental directory listing option

The value of the deprecated Auto Loader cloudFiles.useIncrementalListing option is now set to a default value of false. As a result, this change causes Auto Loader to perform a full directory listing each time it's run. Previously, the default value of the cloudFiles.useIncrementalListing option was auto, instructing Auto Loader to make a best-effort attempt at detecting if an incremental listing can be used with a directory.

Databricks recommends against using this option. Instead, use file notification mode with file events. If you want to continue to use the incremental listing feature, set cloudFiles.useIncrementalListing to auto in your code. When you set this value to auto, Auto Loader makes a best-effort attempt to do a full listing once every seven incremental listings, which matches the behavior of this option before this change.

To learn more about Auto Loader directory listing, see Auto Loader streams with directory listing mode.

Removed the "True cache misses" section in Spark UI

This changes removes support for the "Cache true misses size" metric (for both compressed and uncompressed caches). The "Cache writes misses" metric measures the same information.

Use the numLocalScanTasks as a viable proxy for this metric, when your intention is to see how the cache performs when files are assigned to the right executor.

Removed the "Cache Metadata Manager Peak Disk Usage" metric in the Spark UI

This change removes support for the cacheLocalityMgrDiskUsageInBytes and cacheLocalityMgrTimeMs metrics from the Databricks Runtime and the Spark UI.

Removed the "Rescheduled cache miss bytes" section in the Spark UI

Removed the cache rescheduled misses size and cache rescheduled misses size (uncompressed) metrics from DBR. This is done because this measures how the cache performs when files are assigned to non-preferred executors. numNonLocalScanTasks is a good proxy for this metric.

`CREATE VIEW` column-level clauses now throw errors when the clause would only apply to materialized views

CREATE VIEW commands which specify a column-level clause that is only valid for MATERIALIZED VIEWs now throw an error. The affected clauses for CREATE VIEW commands are:

NOT NULL
A specified datatype, such as FLOAT or STRING
DEFAULT
COLUMN MASK

Library upgrades

Upgraded Python libraries:
- azure-core from 1.31.0 to 1.34.0
- black from 24.4.2 to 24.10.0
- boto3 from 1.34.69 to 1.36.2
- botocore from 1.34.69 to 1.36.3
- cachetools from 5.3.3 to 5.5.1
- certifi from 2024.6.2 to 2025.1.31
- cffi from 1.16.0 to 1.17.1
- charset-normalizer from 2.0.4 to 3.3.2
- cloudpickle from 2.2.1 to 3.0.0
- contourpy from 1.2.0 to 1.3.1
- cryptography from 42.0.5 to 43.0.3
- Cython from 3.0.11 to 3.0.12
- databricks-sdk from 0.30.0 to 0.49.0
- debugpy from 1.6.7 to 1.8.11
- Deprecated from 1.2.14 to 1.2.13
- distlib from 0.3.8 to 0.3.9
- filelock from 3.15.4 to 3.18.0
- fonttools from 4.51.0 to 4.55.3
- GitPython from 3.1.37 to 3.1.43
- google-auth from 2.35.0 to 2.40.0
- google-cloud-core from 2.4.1 to 2.4.3
- google-cloud-storage from 2.18.2 to 3.1.0
- google-crc32c from 1.6.0 to 1.7.1
- grpcio from 1.60.0 to 1.67.0
- grpcio-status from 1.60.0 to 1.67.0
- importlib-metadata from 6.0.0 to 6.6.0
- ipyflow-core from 0.0.201 to 0.0.209
- ipykernel from 6.28.0 to 6.29.5
- ipython from 8.25.0 to 8.30.0
- ipywidgets from 7.7.2 to 7.8.1
- jedi from 0.19.1 to 0.19.2
- jupyter_client from 8.6.0 to 8.6.3
- kiwisolver from 1.4.4 to 1.4.8
- matplotlib from 3.8.4 to 3.10.0
- matplotlib-inline from 0.1.6 to 0.1.7
- mlflow-skinny from 2.19.0 to 2.22.0
- numpy from 1.26.4 to 2.1.3
- opentelemetry-api from 1.27.0 to 1.32.1
- opentelemetry-sdk from 1.27.0 to 1.32.1
- opentelemetry-semantic-conventions from 0.48b0 to 0.53b1
- pandas from 1.5.3 to 2.2.3
- parso from 0.8.3 to 0.8.4
- patsy from 0.5.6 to 1.0.1
- pillow from 10.3.0 to 11.1.0
- plotly from 5.22.0 to 5.24.1
- pluggy from 1.0.0 to 1.5.0
- proto-plus from 1.24.0 to 1.26.1
- protobuf from 4.24.1 to 5.29.4
- pyarrow from 15.0.2 to 19.0.1
- pyccolo from 0.0.65 to 0.0.71
- pydantic from 2.8.2 to 2.10.6
- pydantic_core from 2.20.1 to 2.27.2
- PyJWT from 2.7.0 to 2.10.1
- pyodbc from 5.0.1 to 5.2.0
- pyparsing from 3.0.9 to 3.2.0
- pyright from 1.1.294 to 1.1.394
- python-lsp-server from 1.10.0 to 1.12.0
- PyYAML from 6.0.1 to 6.0.2
- pyzmq from 25.1.2 to 26.2.0
- requests from 2.32.2 to 2.32.3
- rsa from 4.9 to 4.9.1
- s3transfer from 0.10.2 to 0.11.3
- scikit-learn from 1.4.2 to 1.6.1
- scipy from 1.13.1 to 1.15.1
- sqlparse from 0.5.1 to 0.5.3
- statsmodels from 0.14.2 to 0.14.4
- tenacity from 8.2.2 to 9.0.0
- threadpoolctl from 2.2.0 to 3.5.0
- tornado from 6.4.1 to 6.4.2
- typing_extensions from 4.11.0 to 4.12.2
- urllib3 from 1.26.16 to 2.3.0
- virtualenv from 20.26.2 to 20.29.3
- wheel from 0.43.0 to 0.45.1
- wrapt from 1.14.1 to 1.17.0
- yapf from 0.33.0 to 0.40.2
- zipp from 3.17.0 to 3.21.0
Upgraded R libraries:
- arrow from 16.1.0 to 19.0.1
- askpass from 1.2.0 to 1.2.1
- base from 4.4.0 to 4.4.2
- bigD from 0.2.0 to 0.3.0
- bit from 4.0.5 to 4.6.0
- bit64 from 4.0.5 to 4.6.0-1
- bitops from 1.0-8 to 1.0-9
- broom from 1.0.6 to 1.0.7
- bslib from 0.8.0 to 0.9.0
- caret from 6.0-94 to 7.0-1
- chron from 2.3-61 to 2.3-62
- cli from 3.6.3 to 3.6.4
- clock from 0.7.1 to 0.7.2
- commonmark from 1.9.1 to 1.9.5
- compiler from 4.4.0 to 4.4.2
- cpp11 from 0.4.7 to 0.5.2
- credentials from 2.0.1 to 2.0.2
- curl from 5.2.1 to 6.2.1
- data.table from 1.15.4 to 1.17.0
- datasets from 4.4.0 to 4.4.2
- digest from 0.6.36 to 0.6.37
- e1071 from 1.7-14 to 1.7-16
- evaluate from 0.24.0 to 1.0.3
- fontawesome from 0.5.2 to 0.5.3
- fs from 1.6.4 to 1.6.5
- future.apply from 1.11.2 to 1.11.3
- gert from 2.1.0 to 2.1.4
- git2r from 0.33.0 to 0.35.0
- glue from 1.7.0 to 1.8.0
- gower from 1.0.1 to 1.0.2
- graphics from 4.4.0 to 4.4.2
- grDevices from 4.4.0 to 4.4.2
- grid from 4.4.0 to 4.4.2
- gt from 0.11.0 to 0.11.1
- gtable from 0.3.5 to 0.3.6
- hardhat from 1.4.0 to 1.4.1
- httr2 from 1.0.2 to 1.1.1
- jsonlite from 1.8.8 to 1.9.1
- knitr from 1.48 to 1.50
- later from 1.3.2 to 1.4.1
- lava from 1.8.0 to 1.8.1
- lubridate from 1.9.3 to 1.9.4
- methods from 4.4.0 to 4.4.2
- mime from 0.12 to 0.13
- mlflow from 2.14.1 to 2.20.4
- nlme from 3.1-165 to 3.1-164
- openssl from 2.2.0 to 2.3.2
- parallel from 4.4.0 to 4.4.2
- parallelly from 1.38.0 to 1.42.0
- pillar from 1.9.0 to 1.10.1
- pkgbuild from 1.4.4 to 1.4.6
- pkgdown from 2.1.0 to 2.1.1
- processx from 3.8.4 to 3.8.6
- profvis from 0.3.8 to 0.4.0
- progressr from 0.14.0 to 0.15.1
- promises from 1.3.0 to 1.3.2
- ps from 1.7.7 to 1.9.0
- purrr from 1.0.2 to 1.0.4
- R6 from 2.5.1 to 2.6.1
- ragg from 1.3.2 to 1.3.3
- randomForest from 4.7-1.1 to 4.7-1.2
- Rcpp from 1.0.13 to 1.0.14
- RcppEigen from 0.3.4.0.0 to 0.3.4.0.2
- reactR from 0.6.0 to 0.6.1
- readxl from 1.4.3 to 1.4.5
- recipes from 1.1.0 to 1.2.0
- rlang from 1.1.4 to 1.1.5
- rmarkdown from 2.27 to 2.29
- RODBC from 1.3-23 to 1.3-26
- Rserve from 1.8-13 to 1.8-15
- RSQLite from 2.3.7 to 2.3.9
- rstudioapi from 0.16.0 to 0.17.1
- sessioninfo from 1.2.2 to 1.2.3
- shiny from 1.9.1 to 1.10.0
- sparklyr from 1.8.6 to 1.9.0
- SparkR from 3.5.2 to 4.0.0
- splines from 4.4.0 to 4.4.2
- stats from 4.4.0 to 4.4.2
- stats4 from 4.4.0 to 4.4.2
- survival from 3.6-4 to 3.5-8
- sys from 3.4.2 to 3.4.3
- systemfonts from 1.1.0 to 1.2.1
- tcltk from 4.4.0 to 4.4.2
- testthat from 3.2.1.1 to 3.2.3
- textshaping from 0.4.0 to 1.0.0
- timeDate from 4032.109 to 4041.110
- tinytex from 0.52 to 0.56
- tools from 4.4.0 to 4.4.2
- tzdb from 0.4.0 to 0.5.0
- usethis from 3.0.0 to 3.1.0
- utils from 4.4.0 to 4.4.2
- V8 from 4.4.2 to 6.0.2
- waldo from 0.5.2 to 0.6.1
- withr from 3.0.1 to 3.0.2
- xfun from 0.46 to 0.51
- xml2 from 1.3.6 to 1.3.8
- zip from 2.3.1 to 2.3.2
Upgraded Java libraries:
- com.clearspring.analytics.stream from 2.9.6 to 2.9.8
- com.esotericsoftware.kryo-shaded from 4.0.2 to 4.0.3
- com.fasterxml.classmate from 1.3.4 to 1.5.1
- com.fasterxml.jackson.core.jackson-annotations from 2.15.2 to 2.18.2
- com.fasterxml.jackson.core.jackson-core from 2.15.2 to 2.18.2
- com.fasterxml.jackson.core.jackson-databind from 2.15.2 to 2.18.2
- com.fasterxml.jackson.dataformat.jackson-dataformat-cbor from 2.15.2 to 2.18.2
- com.fasterxml.jackson.datatype.jackson-datatype-joda from 2.15.2 to 2.18.2
- com.fasterxml.jackson.datatype.jackson-datatype-jsr310 from 2.16.0 to 2.18.2
- com.fasterxml.jackson.module.jackson-module-paranamer from 2.15.2 to 2.18.2
- com.github.luben.zstd-jni from 1.5.5-4 to 1.5.6-10
- com.google.code.gson.gson from 2.10.1 to 2.11.0
- com.google.crypto.tink.tink from 1.9.0 to 1.16.0
- com.google.errorprone.error_prone_annotations from 2.10.0 to 2.36.0
- com.google.flatbuffers.flatbuffers-java from 23.5.26 to 24.3.25
- com.google.guava.guava from 15.0 to 33.4.0-jre
- com.google.protobuf.protobuf-java from 3.25.1 to 3.25.5
- com.microsoft.azure.azure-data-lake-store-sdk from 2.3.9 to 2.3.10
- com.microsoft.sqlserver.mssql-jdbc from 11.2.3.jre8 to 12.8.0.jre8
- commons-cli.commons-cli from 1.5.0 to 1.9.0
- commons-codec.commons-codec from 1.16.0 to 1.17.2
- commons-io.commons-io from 2.13.0 to 2.18.0
- io.airlift.aircompressor from 0.27 to 2.0.2
- io.dropwizard.metrics.metrics-annotation from 4.2.19 to 4.2.30
- io.dropwizard.metrics.metrics-core from 4.2.19 to 4.2.30
- io.dropwizard.metrics.metrics-graphite from 4.2.19 to 4.2.30
- io.dropwizard.metrics.metrics-healthchecks from 4.2.19 to 4.2.30
- io.dropwizard.metrics.metrics-jetty9 from 4.2.19 to 4.2.30
- io.dropwizard.metrics.metrics-jmx from 4.2.19 to 4.2.30
- io.dropwizard.metrics.metrics-json from 4.2.19 to 4.2.30
- io.dropwizard.metrics.metrics-jvm from 4.2.19 to 4.2.30
- io.dropwizard.metrics.metrics-servlets from 4.2.19 to 4.2.30
- io.netty.netty-all from 4.1.108.Final to 4.1.118.Final
- io.netty.netty-buffer from 4.1.108.Final to 4.1.118.Final
- io.netty.netty-codec from 4.1.108.Final to 4.1.118.Final
- io.netty.netty-codec-http from 4.1.108.Final to 4.1.118.Final
- io.netty.netty-codec-http2 from 4.1.108.Final to 4.1.118.Final
- io.netty.netty-codec-socks from 4.1.108.Final to 4.1.118.Final
- io.netty.netty-common from 4.1.108.Final to 4.1.118.Final
- io.netty.netty-handler from 4.1.108.Final to 4.1.118.Final
- io.netty.netty-handler-proxy from 4.1.108.Final to 4.1.118.Final
- io.netty.netty-resolver from 4.1.108.Final to 4.1.118.Final
- io.netty.netty-tcnative-boringssl-static from 2.0.61.Final-db-r16-windows-x86_64 to 2.0.70.Final-db-r0-windows-x86_64
- io.netty.netty-tcnative-classes from 2.0.61.Final to 2.0.70.Final
- io.netty.netty-transport from 4.1.108.Final to 4.1.118.Final
- io.netty.netty-transport-classes-epoll from 4.1.108.Final to 4.1.118.Final
- io.netty.netty-transport-classes-kqueue from 4.1.108.Final to 4.1.118.Final
- io.netty.netty-transport-native-epoll from 4.1.108.Final-linux-x86_64 to 4.1.118.Final-linux-x86_64
- io.netty.netty-transport-native-kqueue from 4.1.108.Final-osx-x86_64 to 4.1.118.Final-osx-x86_64
- io.netty.netty-transport-native-unix-common from 4.1.108.Final to 4.1.118.Final
- io.prometheus.jmx.collector from 0.12.0 to 0.18.0
- io.prometheus.simpleclient from 0.7.0 to 0.16.1-databricks
- io.prometheus.simpleclient_common from 0.7.0 to 0.16.1-databricks
- io.prometheus.simpleclient_dropwizard from 0.7.0 to 0.16.1-databricks
- io.prometheus.simpleclient_pushgateway from 0.7.0 to 0.16.1-databricks
- io.prometheus.simpleclient_servlet from 0.7.0 to 0.16.1-databricks
- joda-time.joda-time from 2.12.1 to 2.13.0
- net.razorvine.pickle from 1.3 to 1.5
- org.antlr.antlr4-runtime from 4.9.3 to 4.13.1
- org.apache.arrow.arrow-format from 15.0.0 to 18.2.0
- org.apache.arrow.arrow-memory-core from 15.0.0 to 18.2.0
- org.apache.arrow.arrow-memory-netty from 15.0.0 to 18.2.0
- org.apache.arrow.arrow-vector from 15.0.0 to 18.2.0
- org.apache.avro.avro from 1.11.4 to 1.12.0
- org.apache.avro.avro-ipc from 1.11.4 to 1.12.0
- org.apache.avro.avro-mapred from 1.11.4 to 1.12.0
- org.apache.commons.commons-compress from 1.23.0 to 1.27.1
- org.apache.commons.commons-lang3 from 3.12.0 to 3.17.0
- org.apache.commons.commons-text from 1.10.0 to 1.13.0
- org.apache.curator.curator-client from 2.13.0 to 5.7.1
- org.apache.curator.curator-framework from 2.13.0 to 5.7.1
- org.apache.curator.curator-recipes from 2.13.0 to 5.7.1
- org.apache.datasketches.datasketches-java from 3.1.0 to 6.1.1
- org.apache.datasketches.datasketches-memory from 2.0.0 to 3.0.2
- org.apache.hadoop.hadoop-client-runtime from 3.3.6 to 3.4.1
- org.apache.hive.hive-beeline from 2.3.9 to 2.3.10
- org.apache.hive.hive-cli from 2.3.9 to 2.3.10
- org.apache.hive.hive-jdbc from 2.3.9 to 2.3.10
- org.apache.hive.hive-llap-client from 2.3.9 to 2.3.10
- org.apache.hive.hive-llap-common from 2.3.9 to 2.3.10
- org.apache.hive.hive-serde from 2.3.9 to 2.3.10
- org.apache.hive.hive-shims from 2.3.9 to 2.3.10
- org.apache.hive.shims.hive-shims-0.23 from 2.3.9 to 2.3.10
- org.apache.hive.shims.hive-shims-common from 2.3.9 to 2.3.10
- org.apache.hive.shims.hive-shims-scheduler from 2.3.9 to 2.3.10
- org.apache.ivy.ivy from 2.5.2 to 2.5.3
- org.apache.logging.log4j.log4j-1.2-api from 2.22.1 to 2.24.3
- org.apache.logging.log4j.log4j-api from 2.22.1 to 2.24.3
- org.apache.logging.log4j.log4j-core from 2.22.1 to 2.24.3
- org.apache.logging.log4j.log4j-layout-template-json from 2.22.1 to 2.24.3
- org.apache.logging.log4j.log4j-slf4j2-impl from 2.22.1 to 2.24.3
- org.apache.orc.orc-core from 1.9.2-shaded-protobuf to 2.1.1-shaded-protobuf
- org.apache.orc.orc-mapreduce from 1.9.2-shaded-protobuf to 2.1.1-shaded-protobuf
- org.apache.orc.orc-shims from 1.9.2 to 2.1.1
- org.apache.thrift.libthrift from 0.12.0 to 0.16.0
- org.apache.ws.xmlschema.xmlschema-core from 2.3.0 to 2.3.1
- org.apache.xbean.xbean-asm9-shaded from 4.23 to 4.26
- org.apache.zookeeper.zookeeper from 3.9.2 to 3.9.3
- org.apache.zookeeper.zookeeper-jute from 3.9.2 to 3.9.3
- org.checkerframework.checker-qual from 3.31.0 to 3.43.0
- org.eclipse.jetty.jetty-client from 9.4.52.v20230823 to 9.4.53.v20231009
- org.eclipse.jetty.jetty-continuation from 9.4.52.v20230823 to 9.4.53.v20231009
- org.eclipse.jetty.jetty-http from 9.4.52.v20230823 to 9.4.53.v20231009
- org.eclipse.jetty.jetty-io from 9.4.52.v20230823 to 9.4.53.v20231009
- org.eclipse.jetty.jetty-jndi from 9.4.52.v20230823 to 9.4.53.v20231009
- org.eclipse.jetty.jetty-plus from 9.4.52.v20230823 to 9.4.53.v20231009
- org.eclipse.jetty.jetty-proxy from 9.4.52.v20230823 to 9.4.53.v20231009
- org.eclipse.jetty.jetty-security from 9.4.52.v20230823 to 9.4.53.v20231009
- org.eclipse.jetty.jetty-server from 9.4.52.v20230823 to 9.4.53.v20231009
- org.eclipse.jetty.jetty-servlet from 9.4.52.v20230823 to 9.4.53.v20231009
- org.eclipse.jetty.jetty-servlets from 9.4.52.v20230823 to 9.4.53.v20231009
- org.eclipse.jetty.jetty-util from 9.4.52.v20230823 to 9.4.53.v20231009
- org.eclipse.jetty.jetty-util-ajax from 9.4.52.v20230823 to 9.4.53.v20231009
- org.eclipse.jetty.jetty-webapp from 9.4.52.v20230823 to 9.4.53.v20231009
- org.eclipse.jetty.jetty-xml from 9.4.52.v20230823 to 9.4.53.v20231009
- org.eclipse.jetty.websocket.websocket-api from 9.4.52.v20230823 to 9.4.53.v20231009
- org.eclipse.jetty.websocket.websocket-client from 9.4.52.v20230823 to 9.4.53.v20231009
- org.eclipse.jetty.websocket.websocket-common from 9.4.52.v20230823 to 9.4.53.v20231009
- org.eclipse.jetty.websocket.websocket-server from 9.4.52.v20230823 to 9.4.53.v20231009
- org.eclipse.jetty.websocket.websocket-servlet from 9.4.52.v20230823 to 9.4.53.v20231009
- org.glassfish.jersey.containers.jersey-container-servlet from 2.40 to 2.41
- org.glassfish.jersey.containers.jersey-container-servlet-core from 2.40 to 2.41
- org.glassfish.jersey.core.jersey-client from 2.40 to 2.41
- org.glassfish.jersey.core.jersey-common from 2.40 to 2.41
- org.glassfish.jersey.core.jersey-server from 2.40 to 2.41
- org.glassfish.jersey.inject.jersey-hk2 from 2.40 to 2.41
- org.hibernate.validator.hibernate-validator from 6.1.7.Final to 6.2.5.Final
- org.jboss.logging.jboss-logging from 3.3.2.Final to 3.4.1.Final
- org.objenesis.objenesis from 2.5.1 to 3.3
- org.roaringbitmap.RoaringBitmap from 0.9.45-databricks to 1.2.1
- org.rocksdb.rocksdbjni from 9.2.1 to 9.8.4
- org.scalatest.scalatest-compatible from 3.2.16 to 3.2.19
- org.slf4j.jcl-over-slf4j from 2.0.7 to 2.0.16
- org.slf4j.jul-to-slf4j from 2.0.7 to 2.0.16
- org.slf4j.slf4j-api from 2.0.7 to 2.0.16
- org.threeten.threeten-extra from 1.7.1 to 1.8.0
- org.tukaani.xz from 1.9 to 1.10

Apache Spark

Many of its features were already available in Databricks Runtime 14.x, 15.x and 16.x, and now they ship out of the box with Runtime 17.0.

SPARK-52311 Redefine UnionLoop output to not be duplicated if the anchor output is duplicated
SPARK-50104 Support SparkSession.executeCommand in Connect
SPARK-51085 Restore SQLContext Companion
SPARK-49698 Add ClassicOnly annotation for classic only methods.
SPARK-52026 Block pandas API on Spark on ANSI mode by default
SPARK-43415 Implement KVGDS.agg with custom mapValues function
SPARK-50979 Remove .expr/.typedExpr implicits
SPARK-49961 Correct transform type signature for both Scala and Java
SPARK-51012 Remove SparkStrategy from Connect Shims.
SPARK-50915 Add getCondition and deprecate getErrorClass in PySparkException
SPARK-51821 Call interrupt() without holding uninterruptibleLock to avoid possible deadlock
SPARK-52192 MLCache loading path check
SPARK-52122 Fix DefaultParamsReader RCE vulnerability
SPARK-52191 Remove Java deserializer in model local path loader
SPARK-52051 Backport "Enable model summary when memory control is enabled" and "Refine error message, and hide internal spark config" commits to DBR 17.0.0
SPARK-51391 Fix SparkConnectClient to respect SPARK_USER and user.name
SPARK-51416 Remove SPARK_CONNECT_MODE when starting Spark Connect server
SPARK-51156 Static token authentication support in Spark Connect
SPARK-51279 Avoid constant sleep for waiting Spark Connect server in Scala
SPARK-51254 Disallow --master with Spark Connect URL
SPARK-51267 Match local Spark Connect server logic between Python and Scala
SPARK-51212 Add a separated PySpark package for Spark Connect by default
SPARK-52017 Enable multiple self-references and self-references from a Subquery inside rCTEs
SPARK-52035 Decouple LinearRegressionTrainingSummary and LinearRegressionModel
SPARK-50511 Avoid wrapping Python data source error messages
SPARK-51974 Limit model size and per-session model cache size
SPARK-51947 Spark connect model cache offloading
SPARK-49751 Fix deserialization of SparkListenerConnectServiceStarted event
SPARK-47952 Support retrieving the real SparkConnectService GRPC address and port programmatically when running on Yarn
SPARK-47587 Hive module: Migrate logWarn with variables to structured logging framework
SPARK-50768 Introduce TaskContext.createResourceUninterruptibly to avoid stream leak by task interruption
SPARK-51818 Move QueryExecution creation to AnalyzeHandler and don't Execute for AnalyzePlanRequests
SPARK-51609 Optimize Recursive CTE execution for simple queries
SPARK-49748 Add getCondition and deprecate getErrorClass in SparkThrowable
SPARK-51867 Make scala model supporting save / load methods against local filesystem path
SPARK-51856 Update model size API to count distributed DataFrame size
SPARK-51941 Fix the precision problem for convertToCatalyst when the Input is BigDecimal between -1.0 and 1.0
SPARK-50605 Support SQL API mode for easier migration to Spark Connect
SPARK-51849 Refactoring ResolveDDLCommandStringTypes
SPARK-51673 Apply default collation to alter view query
SPARK-51880 Fix ML cache object python client references
SPARK-51873 For OneVsRest algorithm, allow using save / load to replace cache
SPARK-51072 CallerContext to set Hadoop cloud audit context
SPARK-51790 Register UTF8String to KryoSerializer
SPARK-51022 Remove unused tableSampleClause in build() method of MsSqlServerSQLQueryBuilder and DB2SQLQueryBuilder
SPARK-51219 Fix ShowTablesExec.isTempView to work with non-V2SessionCatalog catalogs
SPARK-49700 Unified Scala Interface for Connect and Classic
SPARK-50458 Proper error handling for unsupported file system when reading files
SPARK-50666 Support hint for reading in JDBC data source
SPARK-50131 Reapply "Add IN Subquery DataFram…"
SPARK-51899 Implement error handling rules for spark.catalog.listTables()
SPARK-51820 Address remaining issues for new group/order by ordinal approach
SPARK-48585 Make built-in JdbcDialect's method classifyException throw out the original exception
SPARK-48387 Postgres: Map TimestampType to TIMESTAMP WITH TIME ZONE
SPARK-51820 Move UnresolvedOrdinal construction before analysis to avoid issue with group by ordinal
SPARK-48337 Fix precision loss for JDBC TIME values
SPARK-51711 Propagates the active remote spark session to new threads to fix CrossValidator
SPARK-47515 Save TimestampNTZType as DATETIME in MySQL
SPARK-48439 Derby: Calculate suitable precision and scale for DECIMAL type
SPARK-51820 Prepare redaction logic for SPARK-51820 codesync
SPARK-48323 DB2: Map BooleanType to BOOLEAN instead of CHAR(1)
SPARK-51635 Merge PushProjectionThroughLimit and PushProjectionThroughOffset
SPARK-48211 DB2: Read SMALLINT as ShortType
SPARK-51803 Store external engine JDBC type in the metadata of StructField
SPARK-51845 Add proto messages CleanCache and GetCacheInfo
SPARK-49511 Apply formatting rules to sql/api
SPARK-47968 MsSQLServer: Map datatimeoffset to TimestampType
SPARK-51726 Use TableInfo for Stage CREATE/REPLACE/CREATE OR REPLACE table
SPARK-47967 Make JdbcUtils.makeGetter handle reading time type as NTZ correctly
SPARK-47989 MsSQLServer: Fix the scope of spark.sql.legacy.mssqlserver.numericMapping.enabled
SPARK-51193 Upgrade Netty to 4.1.118.Final and netty-tcnative to 2.0.70.Final
SPARK-47882 createTableColumnTypes need to be mapped to database types instead of using directly
SPARK-47879 Oracle: Use VARCHAR2 instead of VARCHAR for VarcharType mapping
SPARK-51372 Introduce TableInfo for table creations
SPARK-47791 Truncate exceed decimals with scale first instead of precision from JDBC datasource
SPARK-51404 Parse the time(n) type as TimeType(n)
SPARK-50350 Avro: add new function schema_of_avro (scala side)
SPARK-51136 Set CallerContext for History Server
SPARK-50641 Move GetJsonObjectEvaluator to JsonExpressionEvalUtils
SPARK-50083 Integrate _LEGACY_ERROR_TEMP_1231 into PARTITIONS_NOT_FOUND
SPARK-51556 Add the try_to_time function
SPARK-47684 Postgres: Map length unspecified bpchar to StringType
SPARK-48688 Return reasonable error when calling SQL to_avro and from_avro functions but Avro is not loaded by default
SPARK-49839 SPJ: Skip shuffles if possible for sorts
SPARK-45534 Use java.lang.ref.Cleaner instead of finalize for RemoteBlockPushResolver
SPARK-51816 Simplify StatFunctions.multipleApproxQuantiles with dataframe APIs
SPARK-49179 Fix v2 multi bucketed inner joins throw AssertionError
SPARK-47456 Support ORC Brotli codec
SPARK-51542 Add a scroll-button for addressing top and bottom
SPARK-51541 Support the TIME data type in Literal methods
SPARK-51615 Refactor ShowNamespaces to use RunnableCommand
SPARK-51191 Validate default values handling in DELETE, UPDATE, MERGE
SPARK-51829 Client side should update client.thread_local.ml_caches after deletion
SPARK-51358 Introduce snapshot upload lag detection through StateStoreCoordinator
SPARK-51686 Link the execution IDs of sub-executions for current execution if any
SPARK-51456 Add the to_time function
SPARK-51773 Turn file formats into case classes to properly compare them
SPARK-51777 Register sql.columnar.* classes to KryoSerializer
SPARK-51432 Throw a proper exception when Arrow schemas are mismatched
SPARK-51395 Refine handling of default values in procedures
SPARK-50582 Add quote builtin function
SPARK-51684 Fix test failure in test_pandas_transform_with_state
SPARK-51213 Keep Expression class info when resolving hint parameters
SPARK-51651 Link the root execution id for current execution if any
SPARK-50947 Assign appropriate error class and SparkException for duplicated artifacts
SPARK-51574 Filter serialization for Python Data Source filter pushdown
SPARK-51608 Log exception on Python runner termination
SPARK-51266 Remove the unused definition of private[spark] object TaskDetailsClassNames
SPARK-51011 Add logging for whether a task is going to be interrupted when killed
SPARK-49646 Add spark config for fixing subquery decorrelation
SPARK-51107 Refactor CommandBuilderUtils#join to reuse lines and reduce redundancy
SPARK-51758 Fix test case related to extra batch causing empty df due to watermark
SPARK-51664 Support the TIME data type in the Hash expression
SPARK-51819 Update pyspark-errors test module to include missing tests
SPARK-50751 Assign appropriate error condition for _LEGACY_ERROR_TEMP_1305
SPARK-50973 Cleanup deprecated api usage related to avro.Schema#toString(boolean)
SPARK-50908 Disable flaky TTL test in test_pandas_transform_with_state.py
SPARK-50811 Support enabling JVM profiler on driver
SPARK-50808 Fix issue in writeAll with mixed types not getting written properly
SPARK-51780 Implement Describe Procedure
SPARK-50370 Codegen Support for json_tuple
SPARK-50756 Use error class for exceptions in SparkConf.validateSettings
SPARK-50805 Move method nameForAppAndAttempt to o.a.s.u.Utils
SPARK-51812 Remove redundant parameters of some methods in QueryExecution
SPARK-50819 Refactor Spark profiler module
SPARK-51547 Assign name to the error condition: _LEGACY_ERROR_TEMP_2130
SPARK-48538 Avoid HMS memory leak caused by bonecp
SPARK-51176 Meet consistency for unexpected errors PySpark Connect <> Classic
SPARK-50773 Disable structured logging by default
SPARK-50616 Add File Extension Option to CSV DataSource Writer
SPARK-50624 Add TimestampNTZType to ColumnarRow/MutableColumnarRow
SPARK-51590 Disable TIME in builtin file-based datasources
SPARK-49886 Query level failure test for rocksDB format V2
SPARK-50823 Upgrade cloudpickle from 3.1.0 to 3.1.1
SPARK-50780 Use overrideStdFeatures instead of setFeatureMask in JacksonParser
SPARK-50621 Upgrade Cloudpickle to 3.1.0
SPARK-50719 Support interruptOperation for PySpark
SPARK-50545 AccessControlException should be thrown even if ignoreCorruptFiles is enabled
SPARK-51517 Support the TIME data type in Hive results
SPARK-47856 Document Mapping Spark SQL Data Types from Oracle and add tests
SPARK-46066 Use the Separators API instead of the String API to construct the DefaultPrettyPrinter
SPARK-50718 Support addArtifact(s) for PySpark
SPARK-51497 Add the default time formatter
SPARK-51488 Support the TIME keyword as a data type
SPARK-51273 Spark Connect Call Procedure runs the procedure twice
SPARK-51092 Skip the v1 FlatMapGroupsWithState tests with timeout on big endian platforms
SPARK-50606 Fix NPE on uninitiated SessionHolder
SPARK-49530 Support pie subplots in pyspark plotting
SPARK-50357 Support Interrupt(Tag|All) APIs for PySpark
SPARK-51290 Enable filling default values in DSv2 writes
SPARK-50485 Unwrap SparkThrowable in (Unchecked)ExecutionException thrown by tableRelationCache
SPARK-51513 Fix RewriteMergeIntoTable rule produces unresolved plan
SPARK-51482 Support cast from string to time
SPARK-51462 Support typed literals of the TIME data type
SPARK-51454 Support cast from time to string
SPARK-51447 Add stringToTime and stringToTimeAnsi
SPARK-51775 Normalize LogicalRelation and HiveTableRelation by NormalizePlan
SPARK-51791 ImputerModel stores coefficients with arrays instead of dataframe
SPARK-51442 Add time formatters
SPARK-51384 Support java.time.LocalTime as the external type of TimeType
SPARK-51747 Data source cached plan should respect options
SPARK-51774 Add GRPC Status code to Python Connect GRPC Exception
SPARK-51660 Gracefully handle when MDC is not supported
SPARK-51296 Support collecting corrupt data in singleVariantColumn mode.
SPARK-45907 Use Java9+ ProcessHandle APIs to computeProcessTree in ProcfsMetricsGetter
SPARK-51342 Add TimeType
SPARK-51769 Add maxRecordsPerOutputBatch to limit the number of record of Arrow output batch
SPARK-51350 Implement Show Procedures
SPARK-51711 Memory based MLCache eviction policy
SPARK-51178 Raise proper PySpark error instead of SparkConnectGrpcException
SPARK-51738 IN subquery with struct type
SPARK-51714 Add Failure Ingestion test to test state store checkpoint format V2
SPARK-51704 Eliminate unnecessary collect operation
SPARK-51512 Filter out null MapStatus when cleaning up shuffle data with ExternalShuffleService
SPARK-49308 Support UserDefinedAggregateFunction in Spark Connect Scala Client
SPARK-50091 Handle case of aggregates in left-hand operand of IN-subquery
SPARK-50265 Support spark.udf.registerJavaUdf in Connect
SPARK-49273 Origin support for Spark Connect Scala client
SPARK-51187 Implement the graceful deprecation of incorrect config introduced in SPARK-49699
SPARK-51650 Support delete ml cached objects in batch
SPARK-51619 Support UDT input / output in Arrow-optimized Python UDF
SPARK-51333 Unwrap InvocationTargetException thrown in MLUtils.loadOperator
SPARK-51566 Python UDF traceback improvement
SPARK-51393 Fallback to regular Python UDF when Arrow is not found but Arrow-optimized Python UDFs enabled
SPARK-49960 Custom ExpressionEncoder support and TransformingEncoder fixes
SPARK-51380 Add visitSQLFunction and visitAggregateFunction to improve the flexibility of V2ExpressionSQLBuilder
SPARK-51600 Prepend classes of sql/hive and sql/hive-thriftserver when isTesting || isTestingSql is true
SPARK-51070 Use scala.collection.Set instead of Set in ValidateExternalType
SPARK-50759 Deprecate a few legacy Catalog APIs
SPARK-50994 Perform RDD conversion under tracked execution
SPARK-51466 Eliminate Hive built-in UDFs initialization on Hive UDF evaluation
SPARK-51491 Simplify boxplot with subquery APIs
SPARK-51175 Make Master show elapsed time when removing drivers
SPARK-50334 Extract common logic for reading the descriptor of PB file
SPARK-50483 BlockMissingException should be thrown even if ignoreCorruptFiles is enabled
SPARK-50286 Correctly propagate SQL options to WriteBuilder
SPARK-51023 Log remote address on RPC exception
SPARK-47611 Cleanup dead code in MySQLDialect.getCatalystType
SPARK-49229 Deduplicate Scala UDF handling in the SparkConnectPlanner
SPARK-50557 Support RuntimeConfig.contains(..) in Scala SQL Interface
SPARK-51471 RatePerMicroBatchStream - classify the ASSERT error when offset/timestamp in startOffset is larger than the endOffset
SPARK-50473 Simplify classic Column handling
SPARK-49286 Move Avro/Protobuf functions to sql/api
SPARK-49087 Distinguish UnresolvedFunction calling internal functions
SPARK-50422 Make Parameterized SQL queries of SparkSession.sql API GA
SPARK-49249 Companion PR for "Add the new default artifact manager config in PySpark to allowlist"
SPARK-50366 Isolate user-defined tags on thread level for SparkSession in Classic
SPARK-49436 Common interface for SQLContext
SPARK-51551 For tuning algorithm, allow using save / load to replace cache
SPARK-51599 Optimize ps.read_excel for large excel file
SPARK-51118 Fix ExtractPythonUDFs to check the chained UDF input types for fallback
SPARK-50395 Fix malformed URI syntax in Windows
SPARK-50708 Delete Artifact resources on GC of ArtifactManager instance
SPARK-51076 Arrow Python UDF fallback for UDT input and output types
SPARK-50243 Cached classloader for ArtifactManager
SPARK-49249 Artifact isolation in Spark Classic
SPARK-50821 Upgrade Py4J from 0.10.9.8 to 0.10.9.9
SPARK-51591 Fix ThreadPoolExecutor failure in python 3.13 daily test
SPARK-40353 Fix index nullable mismatch in ps.read_excel
SPARK-42746 Implement LISTAGG function
SPARK-50102 Add shims need for missing public SQL methods.
SPARK-50513 Split EncoderImplicits from SQLImplicits and provide helper object within StatefulProcessor
SPARK-51567 Fix DistributedLDAModel.vocabSize
SPARK-49569 Add shims to support SparkContext and RDD
SPARK-51473 ML transformed dataframe keep a reference to the model
SPARK-51340 Model size estimation
SPARK-51474 Don't insert redundant ColumnarToRowExec for node supporting both columnar and row output
SPARK-51445 Change the never changed var to val
SPARK-50618 Make DataFrameReader and DataStreamReader leverage the analyzer more
SPARK-51097 Re-introduce RocksDB state store's last uploaded snapshot version instance metrics
SPARK-49418 Shared Session Thread Locals
SPARK-50096 Assign appropriate error condition for _LEGACY_ERROR_TEMP_2150: TUPLE_SIZE_EXCEEDS_LIMIT
SPARK-50264 Add missing methods to DataStreamWriter
SPARK-49434 Move aggregators to sql/api
SPARK-51451 Fix ExtractGenerator to wait for UnresolvedStarWithColumns to be resolved
SPARK-49416 Add Shared DataStreamReader interface
SPARK-49429 Add Shared DataStreamWriter interface
SPARK-49282 Create a shared SparkSessionBuilder interface.
SPARK-49415 Move SQLImplicits to sql/api
SPARK-51443 Fix singleVariantColumn in DSv2 and readStream.
SPARK-49369 Add implicit Column conversions
SPARK-49417 Add Shared StreamingQueryManager interface
SPARK-51079 Support large variable types in pandas UDF, createDataFrame and toPandas with Arrow
SPARK-51277 Implement 0-arg implementation in Arrow-optimized Python UDF
SPARK-50601 Support withColumns / withColumnsRenamed in subqueries
SPARK-49479 Cancel the Timer non-daemon thread on stopping the BarrierCoordinator
SPARK-51379 Move treeAggregate's final aggregation from driver to executor
SPARK-49712 Remove encoderFor from connect-client-jvm
SPARK-49424 Consolidate Encoders.scala
SPARK-49574 Disable broken delta-sharing tests for master
SPARK-51409 Add error classification in the changelog writer creation path
SPARK-49568 Remove self type from Dataset
SPARK-51433 Change release script to release pyspark-client
SPARK-51422 Eliminate the JVM-Python data exchange in CrossValidator
SPARK-51425 Add client API to set custom operation_id
SPARK-49284 Create a shared Catalog interface
SPARK-50855 Spark Connect Support for TransformWithState In Scala
SPARK-50694 Support renames in subqueries
SPARK-50880 Add a new visitBinaryComparison method to V2ExpressionSQLBuilder
SPARK-51282 Optimize OneVsRestModel transform by eliminating the JVM-Python data exchange
SPARK-51079 Support large variable types in pandas UDF, createDataFrame and toPandas with Arrow
SPARK-51383 Avoid making RPC calls if clients are already known as stopped
SPARK-51227 Fix PySpark Connect _minimum_grpc_version to 1.67.0
SPARK-51362 Change toJSON to use NextIterator API to eliminate adjacent record dependency
SPARK-51375 Suppress SparkConnect(Execution|Session)Manager.periodicMaintenance log messages
SPARK-50393 Introduce common TableArg for Spark Classic and Spark Connect
SPARK-50133 Support DataFrame conversion to table argument in Spark Connect Python Client
SPARK-49574 ExpressionEncoder tracks the AgnosticEncoder that created it
SPARK-49422 Add groupByKey to sql/api
SPARK-51381 Show Session ID in Spark Connect Session page
SPARK-51316 Allow Arrow batches in bytes instead of number of rows
SPARK-50134 Support DataFrame API for SCALAR and EXISTS subqueries in Spark Connect
SPARK-50392 DataFrame conversion to table argument in Spark Classic
SPARK-50553 Throw InvalidPlanInput for invalid plan message
SPARK-51322 Better error message for streaming subquery expression
SPARK-51281 DataFrameWriterV2 should respect the path option
SPARK-50856 Spark Connect Support for TransformWithStateInPandas In Python
SPARK-51333 Unwrap InvocationTargetException thrown by invokeMethod
SPARK-50134 Support DataFrame API for Lateral Join in Spark Connect
SPARK-51083 Modify JavaUtils to not swallow InterruptedExceptions
SPARK-49413 Create a shared RuntimeConfig interface (follow-up)
SPARK-49413 Create a shared RuntimeConfig interface
SPARK-50993 Move nullDataSourceOption from QueryCompilationErrors into QueryExecutionErrors
SPARK-51329 Add numFeatures for clustering models
SPARK-51305 Improve SparkConnectPlanExecution.createObservedMetricsResponse
SPARK-51097 Adding state store instance metrics for last uploaded snapshot version in RocksDB
SPARK-49425 Create a shared DataFrameWriter
SPARK-50655 Move virtual col family related mapping into db layer instead of encoder
SPARK-48530 Support for local variables in SQL Scripting
SPARK-51284 Fix SQL Script execution for empty result
SPARK-49085 Remove special casing for Protobuf functions in Connect
SPARK-50881 Use cached schema where possible in connect dataframe.py
SPARK-51275 Session propagation in python readwrite
SPARK-51109 CTE in subquery expression as grouping column
SPARK-50598 Add parameters to enable later implementation of recursive CTEs
SPARK-51202 Pass the session in meta algorithm python writers
SPARK-51215 Add a helper function to invoke helper model attr
SPARK-51214 Don't eagerly remove the cached models for fit_transform
SPARK-51237 Add API details for new transformWithState helper APIs
SPARK-51192 Expose processWithoutResponseObserverForTesting in SparkConnectPlanner
SPARK-51217 ML model helper constructor clean up
SPARK-51218 Avoid map/flatMap in NondeterministicExpressionCollection
SPARK-50953 Add support for non-literal paths in VariantGet
SPARK-50132 Add DataFrame API for Lateral Joins
SPARK-51190 Fix TreeEnsembleModel.treeWeights
SPARK-50945 Support Summarizer and SummaryBuilder on Connect
SPARK-51142 ML protobufs clean up
SPARK-51139 Refine error class MLAttributeNotAllowedException
SPARK-51080 Fix save/load for PowerIterationClustering
SPARK-51100 Replace transformer wrappers with helper model attribute relations
SPARK-51091 Fix the default params of StopWordsRemover
SPARK-51089 Support VectorIndexerModel.categoryMaps on connect
SPARK-50954 Support client side model path overwrite for meta algorithms
SPARK-50975 Support CountVectorizerModel.from_vocabulary on connect
SPARK-50958 Support Word2VecModel.findSynonymsArray on connect
SPARK-50930 Support PowerIterationClustering on Connect
SPARK-51157 Add missing @varargs Scala annotation for Scala function APIs
SPARK-51155 Make SparkContext show total runtime after stopping
SPARK-51143 Pin plotly<6.0.0 and torch<2.6.0
SPARK-50949 Introduce a helper model to support StringIndexModel.from_labels_xxx
SPARK-51131 Throw exception when SQL Script is found inside EXECUTE IMMEDIATE command
SPARK-51078 Fix the py4j call in StopWordsRemover
SPARK-50944 Support KolmogorovSmirnovTest on Connect
SPARK-50602 Fix transpose to show a proper error message when invalid index columns are specified
SPARK-50943 Support Correlation on Connect
SPARK-50234 Improve error message and test for transpose DataFrame API
SPARK-50942 Support ChiSquareTest on Connect
SPARK-48353 Introduction of Exception Handling mechanism in SQL Scripting
SPARK-51043 Fine grained user logging for Spark Connect foreachBatch
SPARK-50799 Refine the docstring of rlike, length, octet_length, bit_length, and transform
SPARK-51015 Support RFormulaModel.toString on Connect
SPARK-50843 Support return a new model from existing one
SPARK-50969 Fix GaussianMixtureModel.gaussians on connect
SPARK-50899 Support PrefixSpan on connect
SPARK-51060 Support QuantileDiscretizer on Connect
SPARK-50974 Add support foldCol for CrossValidator on connect
SPARK-50922 Support OneVsRest on Connect
SPARK-50812 Add support PolynomialExpansion
SPARK-50923 Support FMClassifier and FMRegressor on Connect
SPARK-50918 Refactor read/write for Pipeline
SPARK-50938 Support VectorSizeHint and VectorSlicer on Connect
SPARK-51005 Support VectorIndexer and ElementwiseProduct on Connect
SPARK-51014 Support RFormula on connect
SPARK-50941 Add supports for TrainValidationSplit
SPARK-51004 Add supports for IndexString
SPARK-51003 Support LSH models on Connect
SPARK-50924 Support AFTSurvivalRegression and IsotonicRegression on Connect
SPARK-50921 Support MultilayerPerceptronClassifier on Connect
SPARK-50995 Support clusterCenters for KMeans and BisectingKMeans
SPARK-50940 Adds support CrossValidator/CrossValidatorModel on connect
SPARK-50929 Support LDA on Connect
SPARK-50925 Support GeneralizedLinearRegression on Connect
SPARK-50988 Fix uid inconsistencies for estimator and model
SPARK-50989 Support NGram, Normalizer and Interaction on connect
SPARK-50937 Support Imputer on Connect
SPARK-51049 Increase S3A Vector IO threshold for range merge
SPARK-50812 Support TargetEncoderModel on Connect
SPARK-50920 Support NaiveBayes on Connect
SPARK-50936 Support HashingTF, IDF and FeatureHasher on connect
SPARK-50934 Support CountVectorizer and OneHotEncoder on Connect
SPARK-49287 Move streaming classes to sql/api
SPARK-50932 Support Bucketizer on Connect
SPARK-50933 Support Feature Selectors on Connect
SPARK-50931 Support Binarizer on connect
SPARK-50935 Support DCT on connect
SPARK-50963 Support Tokenizers, SQLTransform and StopWordsRemover on Connect
SPARK-50928 Support GaussianMixture on Connect
SPARK-49383 Support Transpose DataFrame API
SPARK-50939 Support Word2Vec on Connect
SPARK-49249 Add new tag-related APIs in Connect back to Spark Core
SPARK-50919 Support LinearSVC on connect
SPARK-50883 Support altering multiple columns in the same command
SPARK-50918 Support Pipeline on connect
SPARK-50826 Refactor the way handling ALLOWED_ATTRIBUTES
SPARK-49427 Create a shared interface for MergeIntoWriter
SPARK-49414 Add Shared DataFrameReader interface
SPARK-50948 Add support for StringIndexer/PCA on Connect
SPARK-50901 Support Transformer VectorAssembler
SPARK-50879 Support feature scalers on Connect
SPARK-50130 Add DataFrame APIs for scalar and exists subqueries
SPARK-50075 Add DataFrame APIs for table-valued functions
SPARK-49426 Create a shared interface for DataFrameWriterV2
SPARK-50898 Support FPGrowth on connect
SPARK-50844 Make model be loaded by ServiceLoader when loading
SPARK-50884 Support isLargerBetter in Evaluator
SPARK-50959 Swallow the exception of JavaWrapper
SPARK-50558 Introduce simpleString for ExpressionSet
SPARK-49422 Create a shared interface for KeyValueGroupedDataset
SPARK-50878 Support ALS on Connect
SPARK-50897 Avoiding instance creation in ServiceLoader
SPARK-50877 Support KMeans & BisectingKMeans on Connect
SPARK-50876 Support Tree Regressors on Connect
SPARK-50874 Support LinearRegression on connect
SPARK-50869 Support evaluators on ML Connect
SPARK-50851 Express ML params with proto.Expression.Literal
SPARK-50825 Support Tree Classifiers in ML Connect
SPARK-50827 Support plugin
SPARK-49907 Support spark.ml on Connect
SPARK-50968 Fix the usage of Column.__new__
SPARK-49028 Create a shared SparkSession
SPARK-49421 Create a shared RelationalGroupedDataset interface
SPARK-50804 to_protobuf() should not throw MatchError
SPARK-50900 Add VectorUDT and MatrixUDT to ProtoDataTypes
SPARK-50579 Fix truncatedString
SPARK-50875 Add RTRIM collations to TVF
SPARK-49420 Add shared interface for DataFrameNaFunctions
SPARK-50669 Change the signature of TimestampAdd expression
SPARK-46615 Support s.c.immutable.ArraySeq in ArrowDeserializers
SPARK-49423 Consolidate Observation in sql/api
SPARK-49086 Move ML function registration to SparkSessionExtensions
SPARK-49419 Create shared DataFrameStatFunctions
SPARK-50735 Failure in ExecuteResponseObserver results in infinite reattaching requests
SPARK-50522 Support for indeterminate collation
SPARK-50893 Mark UDT.DataType optional
SPARK-50685 Improve Py4J performance by leveraging getattr
SPARK-50742 Remove spark.hadoop.fs.s3a.connection.establish.timeout setting
SPARK-50714 Enable schema evolution for TransformWithState when Avro encoding is used
SPARK-49029 Create shared Dataset interface
SPARK-50263 Replace System.currentTimeMillis with System.nanoTime
SPARK-50525 Define InsertMapSortInRepartitionExpressions Optimizer Rule
SPARK-50561 Improve type coercion and boundary checking for UNIFORM SQL function
SPARK-50707 Enable casting to/from char/varchar
SPARK-49027 Share Column API between Class and Connect
SPARK-49632 Remove the ANSI config suggestion in CANNOT_PARSE_TIMESTAMP
SPARK-50529 Change char/varchar behavior under the config
SPARK-50600 Set analyzed on analysis failure
SPARK-50789 The inputs for typed aggregations should be analyzed
SPARK-49025 Make Column implementation agnostic
SPARK-50738 Upgrade black to 23.12.1
SPARK-49883 State Store Checkpoint Structure V2 Integration with RocksDB
SPARK-50778 Add metadataColumn to PySpark DataFrame
SPARK-49565 Improve auto-generated expression aliases with pipe SQL operators
SPARK-50772 Retain table aliases after SET, EXTEND, DROP operators
SPARK-50690 Fix discrepancy in DESCRIBE TABLE view query output quoting
SPARK-50675 Table and view level collations support
SPARK-50480 Extend CharType and VarcharType from StringType
SPARK-50715 SparkSession.Builder sets the configs in batch
SPARK-50693 The inputs for TypedScalaUdf should be analyzed
SPARK-50710 Add support for optional client reconnection to sessions after release
SPARK-50596 Upgrade Py4J from 0.10.9.7 to 0.10.9.8
SPARK-50661 Add backward-compatibility for old client FEB.
SPARK-50515 Add read-only interface to SparkConf
SPARK-50642 Fix the state schema for FlatMapGroupsWithState in spark connect when there is no initial state
SPARK-50702 Refine the docstring of regexp_count, regexp_extract and regexp_extract_all
SPARK-50692 Add RPAD pushdown support
SPARK-50699 Parse and generate DDL string with a specified session
SPARK-50573 Adding State Schema ID to State Rows to schema evolution
SPARK-50311 Support (add|remove|get|clear)Tag(s) APIs for PySpark
SPARK-50661 Fix Spark Connect Scala foreachBatch impl. to support Dataset[T].
SPARK-50696 Optimize Py4J call for DDL parse method
SPARK-50687 Optimize the logic to get stack traces for DataFrameQueryContext
SPARK-50681 Cache the parsed schema for MapInXXX and ApplyInXXX
SPARK-50578 Add support for new version of state metadata for TransformWithStateInPandas
SPARK-50405 Handle collation type coercion of complex data types properly
SPARK-50615 Push variant into scan.
SPARK-50599 Create the DataEncoder trait that allows for Avro and UnsafeRow encoding
SPARK-50076 Fix logkeys
SPARK-50339 Enable changelog to store lineage information
SPARK-50540 Fix string schema for StatefulProcessorHandle
SPARK-50544 Implement StructType.toDDL
SPARK-50528 Move InvalidCommandInput to common module
SPARK-50063 Add support for Variant in the Spark Connect Scala client
SPARK-50310 Add a flag to disable DataFrameQueryContext for PySpark
SPARK-50310 Add a flag to disable DataFrameQueryContext for PySpark
SPARK-50032 Allow use of fully qualified collation name
SPARK-50466 Refine the docstring for string functions - part 1
SPARK-49676 Add Support for Chaining of Operators in transformWithStateInPandas API
SPARK-50081 Codegen Support for XPath*(by Invoke & RuntimeReplaceable)
SPARK-46725 Add DAYNAME function
SPARK-50067 Codegen Support for SchemaOfCsv(by Invoke & RuntimeReplaceable)
SPARK-49873 fix post merge failure on error testing
SPARK-50270 Added custom state metrics for TransformWithStateInPandas
SPARK-50381 Support spark.master.rest.maxThreads
SPARK-50427 Expose configure_logging as a public API
SPARK-50173 Make pandas expressions accept more datatypes
SPARK-50169 Improve performance of RegExpReplace
SPARK-50238 Add Variant Support in PySpark UDFs/UDTFs/UDAFs and Python UC UDFs
SPARK-50190 Remove direct dependency of Numpy from Histogram
SPARK-50183 Unify the internal functions for Pandas API and PySpark Plotting
SPARK-50170 Move _invoke_internal_function_over_columns to pyspark.sql.utils
SPARK-50036 Include SPARK_LOG_SCHEMA in the context of REPL shell
SPARK-50141 Make lpad and rpad accept Column type arguments
SPARK-49954 Codegen Support for SchemaOfJson (by Invoke & RuntimeReplaceable)
SPARK-50098 Upgrade the minimum version of googleapis-common-protos to 1.65.0
SPARK-50059 API compatibility check for Structured Streaming I/O
SPARK-50241 Replace NullIntolerant Mixin with Expression.nullIntolerant method
SPARK-49849 API compatibility check for Structured Streaming Query Management
SPARK-49851 API compatibility check for Protobuf
SPARK-49850 API compatibility check for Avro
SPARK-50039 API compatibility check for Grouping
SPARK-50023 API compatibility check for Functions
SPARK-50030 API compatibility check for Window
SPARK-50002 API compatibility check for I/O
SPARK-49848 API compatibility check for Catalog
SPARK-50022 Fix MasterPage to hide App UI links when UI is disabled
SPARK-50021 Fix ApplicationPage to hide App UI links when UI is disabled
SPARK-49990 Improve performance of randStr
SPARK-50380 ReorderAssociativeOperator should respect the contract in ConstantFolding
SPARK-50330 Add hints to Sort and Window nodes
SPARK-49609 Add API compatibility check between Classic and Connect
SPARK-49773 Uncaught Java exception from make_timestamp() with bad timezone
SPARK-49345 Make sure using the current running Spark Session
SPARK-49368 Avoid accessing protobuf lite classes directly
SPARK-50056 Codegen Support for ParseUrl (by Invoke & RuntimeReplaceable)
SPARK-49119 Fix the inconsistency of syntax show columns between v1 and v2
SPARK-50144 Address the limitation of metrics calculation with DSv1 streaming sources
SPARK-49962 Simplify AbstractStringTypes class hierarchy
SPARK-50327 Factor out function resolution to be reused in the single-pass Analyzer
SPARK-48775 Replace SQLContext with SparkSession in STS
SPARK-50325 Factor out alias resolution to be reused in the single-pass Analyzer
SPARK-48123 Provide a constant table schema for querying structured logs
SPARK-50055 Add TryMakeInterval alternative
SPARK-49312 Improve error message for assertSchemaEqual
SPARK-38912 Remove the comment related to classmethod and property
SPARK-50112 Allowing the TransformWithState operator to use Avro encoding
SPARK-50260 Refactor and optimize Spark Connect execution and session management
SPARK-50196 Fix Python error context to use a proper context
SPARK-50167 Improve PySpark plotting error messages and imports
SPARK-50085 Make lit(ndarray) with np.int8 respect the numpy datatype
SPARK-50273 Improve logging for RocksDB lock acquire/release cases
SPARK-50163 Fix the RocksDB extra acquireLock release due to the completion listener
SPARK-49770 Improve RocksDB SST file mapping management, and fix issue with reloading same version with existing snapshot
SPARK-50232 Add 'protobuf==5.28.3' in dev/requirements.txt
SPARK-50231 Make function instr accept Column substring
SPARK-50028 Replace global locks in Spark Connect server listener with fine-grained locks
SPARK-50077 Introduce a new pattern object for LogicalRelation to help avoiding default full params pattern
SPARK-50128 Add stateful processor handle APIs using implicit encoders in Scala
SPARK-49411 Communicate State Store Checkpoint ID between driver and stateful operators
SPARK-50054 Support histogram plots
SPARK-49854 Do not copy cluster libraries when cloning Artifact manager
SPARK-50071 Add try_make_timestamp(_ltz and _ntz) and related tests
SPARK-50024 Switch to use logger instead of warnings module in client
SPARK-50174 Factor out UnresolvedCatalogRelation resolution
SPARK-49734 Add seed argument for function shuffle
SPARK-49943 Remove timestamp_ntz_to_long from PythonSQLUtils
SPARK-49945 Add alias for distributed_id
SPARK-49755 Remove special casing for avro functions in Connect
SPARK-49805 Remove private[xxx] functions from function.scala
SPARK-49929 Support box plots
SPARK-49767 Refactor the internal function invocation
SPARK-49939 Codegen Support for json_object_keys (by Invoke & RuntimeReplaceable)
SPARK-49854 Clone artifact manager during session clone
SPARK-49766 Codegen Support for json_array_length (by Invoke & RuntimeReplaceable)
SPARK-49540 Unify the usage of distributed_sequence_id
SPARK-50046 Use stable order of EventTimeWatermark node to calculate watermark
SPARK-50031 Add the TryParseUrl expression
SPARK-49202 Apply ArrayBinarySearch for histogram
SPARK-49811 Rename StringTypeAnyCollation
SPARK-50106 Upgrade protobuf Python package to 5.28.3
SPARK-49203 Add expression for java.util.Arrays.binarySearch
SPARK-50090 Refactor ResolveBinaryArithmetic to separate single-node transformation
SPARK-49103 Support spark.master.rest.filters
SPARK-49222 Clear useless methods in QueryCompilationErrors
SPARK-50062 Support collations by InSet
SPARK-50035 Add support for explicit handleExpiredTimer function part of the stateful processor
SPARK-50050 Make lit accept str and bool type numpy ndarray
SPARK-50051 Make lit works with empty numpy ndarray
SPARK-49857 Add storageLevel to Dataset localCheckpoint API
SPARK-48749 Simplify UnaryPositive and eliminate its Catalyst Rules with RuntimeReplaceable
SPARK-50058 Factor out plan normalization functions to later use those in single-pass Analyzer testing
SPARK-50042 Upgrade numpy 2 for python linter
SPARK-50052 Make NumpyArrayConverter support empty str ndarray
SPARK-49126 Move spark.history.ui.maxApplications config definition to History.scala
SPARK-50044 Refine the docstring of multiple math functions
SPARK-48782 Add support for executing procedures in catalogs
SPARK-48773 Document config "spark.default.parallelism" by config builder framework
SPARK-49876 Get rid of global locks from Spark Connect Service
SPARK-48480 StreamingQueryListener should not be affected by spark.interrupt()
SPARK-49978 Move sparkR deprecation warning to package attach time
SPARK-48549 Improve SQL function sentences
SPARK-49956 Disabled collations with collect_set expression
SPARK-49974 Move resolveRelations(...) out of the Analyzer.scala
SPARK-49067 Move utf-8 literal into internal methods of UrlCodec class
SPARK-49393 Fail by default in deprecated catalog plugin APIs
SPARK-49918 Use read-only access to conf in SparkContext where appropriate
SPARK-49924 Keep containsNull after ArrayCompact replacement
SPARK-49895 Improve error when encountering trailing comma in SELECT clause
SPARK-49890 Extract the preparation of df.sample to parent class
SPARK-49810 Extract the preparation of DataFrame.sort to parent class
SPARK-49405 Restrict charsets in JsonOptions
SPARK-49542 Partition transform exception evaluate error
SPARK-47172 Add support for AES-GCM for RPC encryption
SPARK-44914 Fix HadoopConfUtilsSuite after dropping xercesImpl
SPARK-47496 Java SPI Support for dynamic JDBC dialect registering
SPARK-48961 Make the parameter naming of PySparkException consistent with JVM
SPARK-47390 Handle SQL timestamps mapping for Postgres and MySQL
SPARK-49824 Improve logging in SparkConnectStreamingQueryCache
SPARK-49894 Refine the string representation of column field operations
SPARK-49836 Fix possibly broken query when window is provided to window/session_window fn
SPARK-49531 Support line plot with plotly backend
SPARK-48780 Make errors in NamedParametersSupport generic to handle functions and procedures
SPARK-49026 Add ColumnNode to Proto conversion
SPARK-49814 When Spark Connect Client starts, show the spark version of the connect server
SPARK-49246 TableCatalog#loadTable should indicate if it's for writing
SPARK-49749 Change log level to debug in BlockManagerInfo
SPARK-48303 Reorganize LogKeys
SPARK-48112 Expose session in SparkConnectPlanner to plugins
SPARK-45919 Use Java 16 record to simplify Java class definition
SPARK-48126 Make spark.log.structuredLogging.enabled effective
SPARK-49656 Add support for state variables with value state collection types and read change feed options
SPARK-49323 Move MockObserver from Spark Connect Server's test folder to the Server's main folder
SPARK-49772 Remove ColumnFamilyOptions and add configs directly to dbOptions in RocksDB
SPARK-49688 Fix a data race between interrupt and execute plan
SPARK-49585 Replace executions map in SessionHolder with operationID set
SPARK-49684 Minimise the lifetime of the session restoration lock
SPARK-48857 Restrict charsets in CSVOptions
SPARK-48615 Perf improvement for parsing hex string
SPARK-49719 Make UUID and SHUFFLE accept integer seed
SPARK-49713 Make function count_min_sketch accept number arguments
SPARK-48623 Structured logging migrations [Part 3]
SPARK-48541 Add a new exit code for executors killed by TaskReaper
SPARK-48627 Perf improvement for binary to HEX_DISCRETE string
SPARK-49226 Clean-up UDF code generation
SPARK-49673 Increase CONNECT_GRPC_ARROW_MAX_BATCH_SIZE to 0.7 * CONNECT_GRPC_MAX_MESSAGE_SIZE
SPARK-49307 Add Kryo serialization to agnostic encoder framework
SPARK-48601 Give a more user friendly error message when setting a null value for JDBC Option
SPARK-42252 Add spark.shuffle.localDisk.file.output.buffer and deprecate spark.shuffle.unsafe.file.output.buffer
SPARK-49505 Create new SQL functions "randstr" and "uniform" to generate random strings or numbers within ranges
SPARK-48341 Allow plugins to use QueryTest in their tests
SPARK-48374 Support additional PyArrow Table column types
SPARK-49412 Compute all box plot metrics in single job
SPARK-49684 Remove global locks from session and execution managers
SPARK-49225 Add ColumnNode sql & normalize
SPARK-49274 Support java serialization based encoders
SPARK-49089 Move hardcoded Catalyst Expressions to the internal function registry
SPARK-48185 Fix 'symbolic reference class is not accessible: class sun.util.calendar.ZoneInfo'
SPARK-48037 Fix SortShuffleWriter lacks shuffle write related metrics resulting in potentially inaccurate data
SPARK-49534 No longer prepend sql/hive and sql/hive-thriftserver when spark-hive_xxx.jar is not in the classpath
SPARK-49502 Avoid NPE in SparkEnv.get.shuffleManager.unregisterShuffle
SPARK-49567 Use classic instead of vanilla from PySpark code base
SPARK-49582 Improve "dispatch_window_method" utility and docstring
SPARK-49478 Handle null metrics in ConnectProgressExecutionListener
SPARK-49525 Minor log improvement to Server Side Streaming Query ListenerBus Listener
SPARK-49544 Replace coarse-locking in SparkConnectExecutionManager with ConcurrentMap
SPARK-49548 Replace coarse-locking in SparkConnectSessionManager with ConcurrentMap
SPARK-49004 Use separate registry for Column API internal functions
SPARK-49443 Implement to_variant_object expression and make schema_of_variant expressions print OBJECT for Variant Objects
SPARK-49595 Fix DataFrame.unpivot/melt in Spark Connect Scala Client
SPARK-49526 Support Windows-style paths in ArtifactManager
SPARK-49396 Modify nullability check for CaseWhen expression
SPARK-49024 Add support for functions to column node
SPARK-48985 Connect Compatible Expression Constructors
SPARK-49083 Allow from_xml and from_json to natively work with json schemas
SPARK-48986 Add ColumnNode Intermediate Representation
SPARK-48960 Makes spark-submit work with Spark connect
SPARK-49492 Reattach attempted on inactive ExecutionHolder
SPARK-47307 Add a config to optionally chunk base64 strings
SPARK-49451 Allow duplicate keys in parse_json
SPARK-49021 Add support for reading transformWithState value state variables with state data source reader
SPARK-49249 Add addArtifact API to the Spark SQL Core
SPARK-48693 Simplify and unify toString of Invoke and StaticInvoke
SPARK-41982 Partitions of type string should not be treated as numeric types
SPARK-49216 Fix to not log message context with explicitly LogEntry constructed when Structured Logging conf is off
SPARK-49459 Support CRC32C for Shuffle Checksum
SPARK-49409 Adjust the default value of CONNECT_SESSION_PLAN_CACHE_SIZE
SPARK-49164 Fix not NullSafeEqual in predicate of SQL query in JDBC Relation
SPARK-48344 SQL Scripting execution (including Spark Connect)
SPARK-49260 No longer prepend the classes path of sql/core module in Spark Connect Shell
SPARK-49041 Raise proper error for dropDuplicates when wrong subset is given
SPARK-49300 Fix Hadoop delegation token leak when tokenRenewalInterval is not set
SPARK-48796 Load Column Family Id from RocksDBCheckpointMetadata for VCF when restarting
SPARK-49269 Eagerly evaluate VALUES() list in AstBuilder
SPARK-49336 Limit the nesting level when truncating a protobuf message
SPARK-49245 Refactor some analyzer rules
SPARK-48755 transformWithState pyspark base implementation and ValueState support
SPARK-48762 Introduce clusterBy DataFrameWriter API for Python
SPARK-48967 Improve performance and memory footprint of "INSERT INTO ... VALUES" Statements
SPARK-49195 Embed script level parsing logic into SparkSubmitCommandBuilder
SPARK-49173 Change Spark Connect shell prompt from @ to scala>
SPARK-49198 Prune more jars required for Spark Connect shell
SPARK-48936 Makes spark-shell work with Spark connect
SPARK-49201 Reimplement hist plot with Spark SQL
SPARK-49111 Move withProjectAndFilter to the companion object of DataSourceV2Strategy
SPARK-49185 Reimplement kde plot with Spark SQL
SPARK-48761 Introduce clusterBy DataFrameWriter API for Scala
SPARK-48628 Add task peak on/off heap memory metrics
SPARK-48900 Add reason field for all internal calls for job/stage cancellation
SPARK-49076 Fix the outdated logical plan name in AstBuilder's comments
SPARK-49059 Move SessionHolder.forTesting(...) to the test package
SPARK-48658 Encode/Decode functions report coding errors instead of mojibake
SPARK-45891 Add support for interval types in the Variant Spec
SPARK-49032 Add schema path in metadata table entry and related test for operator metadata format v2
SPARK-49009 Make Column APIs and functions accept Enums
SPARK-49035 Eliminate TypeVar ColumnOrName_
SPARK-48849 Create OperatorStateMetadataV2 for the TransformWithStateExec operator
SPARK-48974 Use SparkSession.implicits instead of SQLContext.implicits
SPARK-48996 Allow bare literals for __and__ and __or__ of Column
SPARK-48928 Log Warning for Calling .unpersist() on Locally Checkpointed RDDs
SPARK-48972 Unify the literal string handling in functions
SPARK-48891 Refactor StateSchemaCompatibilityChecker to unify all state schema formats
SPARK-48841 Include collationName to sql() of Collate
SPARK-48944 Unify the JSON-format schema handling in Connect Server
SPARK-48945 Simplify regex functions with lit
SPARK-48865 Add try_url_decode function
SPARK-48851 Change the value of SCHEMA_NOT_FOUND from namespace to catalog.namespace
SPARK-48510 Fix for UDAF toColumn API when running tests in Maven
SPARK-45190 Make from_xml support StructType schema
SPARK-48900 Add reason field for cancelJobGroup and cancelJobsWithTag
SPARK-48909 Use SparkSession over SparkContext when writing metadata
SPARK-48510 Support UDAF toColumn API in Spark Connect
SPARK-45155 Add API Docs for Spark Connect JVM/Scala Client
SPARK-48794 df.mergeInto support for Spark Connect (Scala and Python)
SPARK-48714 Implement DataFrame.mergeInto in PySpark
SPARK-48726 Create the StateSchemaV3 file format for TransformWithStateExec operator
SPARK-48834 Disable variant input/output to python UDFs, UDTFs, UDAFs during query compilation
SPARK-48716 Add jobGroupId to SparkListenerSQLExecutionStart
SPARK-48888 Remove snapshot creation based on changelog ops size
SPARK-48772 State Data Source Change Feed Reader Mode
SPARK-48742 Virtual Column Family for RocksDB
SPARK-48852 Fix string trim function in connect
SPARK-48343 Introduction of SQL Scripting interpreter
SPARK-48118 Support SPARK_SQL_LEGACY_CREATE_HIVE_TABLE env variable
SPARK-48804 Add classIsLoadable & OutputCommitter.isAssignableFrom check for output committer class configs
SPARK-47577 Correct misleading usage of log key TASK_ID
SPARK-48798 Introduce spark.profile.render for SparkSession-based profiling
SPARK-48686 Improve performance of ParserUtils.unescapeSQLString
SPARK-48611 Log TID for input split in HadoopRDD and NewHadoopRDD
SPARK-48720 Align the command ALTER TABLE ... UNSET TBLPROPERTIES ... in v1 and v2
SPARK-48710 Use NumPy 2.0 compatible types
SPARK-48810 Session stop() API should be idempotent
SPARK-48818 Simplify percentile functions
SPARK-48638 Add ExecutionInfo support for DataFrame
SPARK-48799 Refactor versioning for operator metadata read/write
SPARK-46122 Set spark.sql.legacy.createHiveTableByDefault to false by default
SPARK-48629 Migrate the residual code to structured logging framework
SPARK-48320 Sync the latest Logging trait and test cases from OSS Spark
SPARK-48573 Upgrade ICU version
SPARK-48687 Add state schema validation and update on driver in planning phase for stateful queries
SPARK-47579 Migrate logInfo with variables to structured logging framework (PART 1–4)
SPARK-48713 Add index range check for UnsafeRow.pointTo when baseObject is byte array
SPARK-48498 Always do char padding in predicates
SPARK-48598 Propagate cached schema in dataframe operations
SPARK-47599 MLLib: Migrate logWarn with variables to structured logging framework
SPARK-48576 Rename UTF8_BINARY_LCASE to UTF8_LCASE
SPARK-48650 Display correct call site from IPython Notebook
SPARK-48059 Structured log framework on the Java side
SPARK-48482 dropDuplicates and dropDuplicatesWithinWatermark should accept variable length args
SPARK-48620 Fix internal raw data leak in YearMonthIntervalType and CalendarIntervalType
SPARK-48555 Support using Columns as parameters for several functions
SPARK-48591 Add a helper function to simplify Column.py
SPARK-48459 Implement DataFrameQueryContext in Spark Connect
SPARK-48610 Refactor: use auxiliary idMap instead of OP_ID_TAG
SPARK-47923 Upgrade the minimum version of arrow R package to 10.0.0
SPARK-48593 Fix the string representation of lambda function
SPARK-46947 Delay memory manager initialization until Driver plugin is loaded
SPARK-48220 Allow passing PyArrow Table to createDataFrame()
SPARK-48564 Propagate cached schema in set operations
SPARK-48561 Throw PandasNotImplementedError for unsupported plotting functions
SPARK-48513 Add error class for state schema compatibility
SPARK-48553 Cache more properties
SPARK-48550 Directly use the parent Window class
SPARK-48504 Parent Window class for Spark Connect and Spark Classic
SPARK-48508 Cache user specified schema in DataFrame.{to, mapInPandas, mapInArrow}
SPARK-48496 Use static regex Pattern instances in JavaUtils
SPARK-47578 Manual backport: migrate logWarning with variables
SPARK-47737 Bump PyArrow to 10.0.0
SPARK-48159 Extend support for collated strings on datetime expressions
SPARK-48454 Directly use the parent DataFrame class
SPARK-48438 Directly use the parent Column class
SPARK-47597 Manual backport: migrate logInfo with variables
SPARK-48434 Make printSchema use the cached schema
SPARK-46998 Deprecate SQL config spark.sql.legacy.allowZeroIndexInFormatString
SPARK-46569 Remove ThreadLocal for SecureRandom since JDK9
SPARK-46455 Remove redundant type conversion
SPARK-46270 Use java16 instanceof expressions
SPARK-46479 Use utility method from commons-lang3 for Java version check
SPARK-45998 Cleanup redundant type cast
SPARK-45533 Use j.l.r.Cleaner instead of finalize for RocksDBIterator/LevelDBIterator
SPARK-45309 Remove all SystemUtils.isJavaVersionAtLeast with JDK 9/11/17
SPARK-48295 Turn on compute.ops_on_diff_frames by default
SPARK-47960 Allow chaining other stateful operators after transformWithState
SPARK-48367 Fix lint-scala for scalafmt file detection
SPARK-48247 Use all values in dict for MapType schema inference
SPARK-48370 Checkpoint and localCheckpoint in Scala Spark Connect client
SPARK-48258 Checkpoint and localCheckpoint in Spark Connect
SPARK-48293 Add test for ForeachBatchUserFuncException wrapping interrupt
SPARK-48031 Decompose viewSchemaMode config; add SHOW CREATE TABLE support
SPARK-48288 Add source data type for connector cast expression
SPARK-48310 Cached properties must return copies
SPARK-48287 Apply builtin timestamp_diff method
SPARK-44444 Use ANSI SQL mode by default
SPARK-48276 Add missing __repr__ for SQLExpression
SPARK-46991 Replace IllegalArgumentException with SparkIllegalArgumentException in catalyst
SPARK-48031 Support view schema evolution
SPARK-48113 Allow Plugins to integrate with Spark Connect
SPARK-47158 Assign name and sqlState to legacy error codes
SPARK-47545 Dataset observe support for Scala client
SPARK-47993 Drop Python 3.8
SPARK-48260 Disable output committer coordination in ParquetIOSuite
SPARK-47365 Add toArrow() DataFrame method to PySpark
SPARK-47963 Enable structured logging for external Spark ecosystem
SPARK-48045 Fix multi-agg-relabel ignoring as_index=False
SPARK-47719 Change timeParserPolicy default to CORRECTED
SPARK-48075 Type checking for PySpark avro functions
SPARK-48102 Track metrics duration in streaming query progress
SPARK-47858 Refactor DataFrame error context
SPARK-48052 Recover pyspark-connect CI by parent classes
SPARK-45284 Update SparkR minimum SystemRequirements to Java 17
SPARK-47933 Parent Column class for Spark Connect and Classic
SPARK-48053 SparkSession.createDataFrame should warn for unsupported options
SPARK-48044 Cache DataFrame.isStreaming
SPARK-47594 Structured log migrations
SPARK-47764 Cleanup shuffle dependencies by ShuffleCleanupMode
SPARK-45501 Use pattern matching for type checking and conversion
SPARK-45515 Use enhanced switch expressions to replace the regular switch statement
SPARK-47417 Collation support: Ascii, Chr, Base64, UnBase64, Decode, StringDecode, Encode, ToBinary, FormatNumber, Sentences
SPARK-47909 Parent DataFrame class for Spark Connect and Spark Classic
SPARK-47602 Core/MLLib/Resource managers: structured logging migration
SPARK-47390 PostgresDialect distinguishes TIMESTAMP from TIMESTAMP_TZ
SPARK-47868 Fix recursion limit error in SparkConnectPlanner and SparkSession
SPARK-45802 Remove no longer needed Java majorVersion checks in Platform
SPARK-47818 Introduce plan cache in SparkConnectPlanner to improve performance of Analyze requests
SPARK-46031 Replace !Optional.isPresent() with Optional.isEmpty()
SPARK-45659 Add since field to Java API marked as @Deprecated
SPARK-45596 Use java.lang.ref.Cleaner instead of org.apache.spark.sql.connect.client.util.Cleaner
SPARK-47807 Make pyspark.ml compatible with pyspark-connect
SPARK-45830 Refactor StorageUtils#bufferCleaner
SPARK-45578 Remove InaccessibleObjectException usage by using trySetAccessible
SPARK-44895 Add 'daemon', 'priority' for ThreadStackTrace
SPARK-45295 Remove Utils.isMemberClass workaround for JDK 8
SPARK-47081 Support Query Execution Progress
SPARK-45322 Use ProcessHandle to get pid directly
SPARK-46812 Make mapInPandas / mapInArrow support ResourceProfile
SPARK-47406 Handle TIMESTAMP and DATETIME in MYSQLDialect
SPARK-47712 Allow connect plugins to create and process Datasets
SPARK-47720 Update spark.speculation.multiplier to 3 and spark.speculation.quantile to 0.9
SPARK-47665 Use SMALLINT to Write ShortType to MYSQL
SPARK-47722 Wait until RocksDB background work finish before closing
SPARK-47610 Always set io.netty.tryReflectionSetAccessible=true
SPARK-47372 Add support for range scan based key state encoder for use with state store provider
SPARK-44708 Migrate test_reset_index assert_eq to use assertDataFrameEqual
SPARK-47346 Make daemon mode configurable when creating Python planner workers
SPARK-47419 Move log4j2-defaults.properties to common/utils
SPARK-47380 Ensure on the server side that the SparkSession is the same
SPARK-47055 Upgrade MyPy 1.8.0
SPARK-46795 Replace UnsupportedOperationException by SparkUnsupportedOperationException in sql/core
SPARK-46648 Use zstd as the default ORC compression
SPARK-47322 Make withColumnsRenamed column names duplication handling consistent with withColumnRenamed
SPARK-47011 Remove deprecated BinaryClassificationMetrics.scoreLabelsWeight
SPARK-46332 Migrate CatalogNotFoundException to the error class CATALOG_NOT_FOUND
SPARK-46975 Support dedicated fallback methods
SPARK-47069 Introduce spark.profile.show/dump for SparkSession-based profiling
SPARK-47062 Move Connect Plugins to Java for Compatibility
SPARK-46833 Collations - Introducing CollationFactory which provides comparison and hashing rules for supported collations
SPARK-46984 Remove pyspark.copy_func
SPARK-46849 Run optimizer on CREATE TABLE column defaults
SPARK-46976 Implement DataFrameGroupBy.corr
SPARK-46911 Adding deleteIfExists operator to StatefulProcessorHandleImpl
SPARK-46955 Implement Frame.to_stata
SPARK-46936 Implement Frame.to_feather
SPARK-46655 Skip query context catching in DataFrame methods
SPARK-46926 Add convert_dtypes, infer_objects and set_axis in fallback list
SPARK-46683 Write a subquery generator that generates subqueries permutations to increase testing coverage
SPARK-46777 Refactor StreamingDataSourceV2Relation catalyst structure to be more on-par with the batch version
SPARK-46620 Introduce a basic fallback mechanism for frame methods
SPARK-46808 Refine error classes in Python with automatic sorting function
SPARK-46686 Basic support of SparkSession based Python UDF profiler
SPARK-46258 Add RocksDBPersistenceEngine
SPARK-46665 Remove assertPandasOnSparkEqual
SPARK-46227 Move withSQLConf from SQLHelper to SQLConfHelper
SPARK-40876 Widening type promotions in Parquet readers
SPARK-46101 Reduce stack depth by replace (string|array).size with (string|array).length
SPARK-46170 Support inject adaptive query post planner strategy rules in SparkSessionExtensions
SPARK-46246 EXECUTE IMMEDIATE SQL support
SPARK-46466 Vectorized parquet reader should never do rebase for timestamp ntz
SPARK-46399 Add exit status to the Application End event for the use of Spark Listener
SPARK-45506 Add ivy URI support to SparkConnect addArtifact
SPARK-45597 Support creating table using a Python data source in SQL (DSv2 exec)
SPARK-46402 Add getMessageParameters and getQueryContext support
SPARK-46213 Introduce PySparkImportError for error framework
SPARK-46226 Migrate all remaining RuntimeError into PySpark error framework
SPARK-45886 Output full stack trace in callSite of DataFrame context
SPARK-46256 Parallel Compression Support for ZSTD
SPARK-46249 Require instance lock for acquiring RocksDB metrics to prevent race with background operations
SPARK-45667 Clean up the deprecated API usage related to IterableOnceExtensionMethods
SPARK-46254 Remove stale Python 3.8/3.7 version checking
SPARK-46213 Introduce PySparkImportError for error framework
SPARK-46188 Fix the CSS of Spark doc's generated tables
SPARK-45670 SparkSubmit does not support --total-executor-cores when deploying on K8s
SPARK-46169 Assign appropriate JIRA numbers for missing parameters from DataFrame API
SPARK-45022 Provide context for dataset API errors
SPARK-46062 Sync the isStreaming flag between CTE definition and reference
SPARK-45698 Clean up the deprecated API usage related to Buffer
SPARK-45136 Enhance ClosureCleaner with Ammonite support
SPARK-44442 Remove Mesos support
SPARK-45996 Show proper dependency requirement messages for Spark Connect
SPARK-45767 Delete TimeStampedHashMap and its UT
SPARK-45912 Enhancement of XSDToSchema API: Change to HDFS API for cloud storage accessibility
SPARK-45338 Replace scala.collection.JavaConverters to scala.jdk.CollectionConverters
SPARK-45828 Remove deprecated method in dsl
SPARK-45718 Remove remaining deprecated Pandas features from Spark 3.4.0
SPARK-45990 Upgrade protobuf to 4.25.1 to support Python 3.11
SPARK-45941 Upgrade pandas to version 2.1.3
SPARK-45555 Includes a debuggable object for failed assertion
SPARK-45710 Assign names to error _LEGACY_ERROR_TEMP_21[59,60,61,62]
SPARK-45733 Support multiple retry policies
SPARK-45503 Add Conf to Set RocksDB Compression
SPARK-45614 Assign names to error _LEGACY_ERROR_TEMP_215[6,7,8]
SPARK-45680 Release session
SPARK-45620 Fix user-facing APIs related to Python UDTF to use camelCase
SPARK-45634 Remove DataFrame.get_dtype_counts from Pandas API on Spark
SPARK-44752 XML: Update Spark Docs
SPARK-45523 Return useful error message if UDTF returns None for any non-nullable column
SPARK-45558 Introduce a metadata file for streaming stateful operator
SPARK-45390 Remove distutils usage
SPARK-45517 Expand more exception constructors to support error framework parameters
SPARK-45427 Add RPC SSL settings to SSLOptions and SparkTransportConf
SPARK-45581 Make SQLSTATE mandatory.
SPARK-44784 Make SBT testing hermetic.
SPARK-45550 Remove deprecated APIs from Pandas API on Spark
SPARK-45415 Allow selective disabling of "fallocate" in RocksDB statestore
SPARK-45487 Fix SQLSTATEs and temp errors
SPARK-45505 Refactor analyzeInPython to make it reusable
SPARK-45451 Make the default storage level of dataset cache configurable
SPARK-45065 Support Pandas 2.1.0
SPARK-45450 Fix imports according to PEP8: pyspark.pandas and pyspark (core)
SPARK-43299 Convert StreamingQueryException in Scala Client
SPARK-42617 Support isocalendar from the pandas 2.0.0
SPARK-45441 Introduce more util functions for PythonWorkerUtils
SPARK-43620 Fix Pandas APIs depends on unsupported features
SPARK-45330 Upgrade ammonite to 2.5.11
SPARK-45267 Change the default value for numeric_only.
SPARK-45303 Remove JDK 8/11 workaround in KryoSerializerBenchmark
SPARK-43433 Match GroupBy.nth behavior to the latest Pandas
SPARK-45166 Clean up unused code paths for pyarrow<4
SPARK-44823 Update black to 23.9.1 and fix erroneous check
SPARK-45165 Remove inplace parameter from CategoricalIndex APIs
SPARK-45180 Remove boolean inputs for inclusive parameter from Series.between
SPARK-45164 Remove deprecated Index APIs
SPARK-45179 Increase Numpy minimum version to 1.21
SPARK-45177 Remove col_space parameter from to_latex
SPARK-43241 MultiIndex.append not checking names for equality
SPARK-43123 Raise TypeError for DataFrame.interpolate when all columns are object-dtype.
SPARK-43295 Support string type columns for DataFrameGroupBy.sum
SPARK-42619 Add show_counts parameter for DataFrame.info
SPARK-44863 Add a button to download thread dump as a txt in Spark UI
SPARK-44713 Move shared classes to sql/api
SPARK-44692 Move Trigger(s) to sql/api
SPARK-43563 Remove squeeze from read_csv & enabling more tests.
SPARK-43476 Support StringMethods for pandas 2.0.0 and above
SPARK-43872 Support (DataFrame|Series).plot with pandas 2.0.0 and above.
SPARK-42620 Add inclusive parameter for (DataFrame|Series).between_time
SPARK-44289 Support indexer_between_time for pandas 2.0.0 & enabling more tests.
SPARK-42621 Add inclusive parameter for pd.date_range
SPARK-43709 Remove closed parameter from ps.date_range & enable test.
SPARK-43568 Support Categorical APIs for pandas 2
SPARK-44842 Support stat functions for pandas 2.0.0 and enabling tests.
SPARK-43606 Remove Int64Index & Float64Index
SPARK-43873 Enabling FrameDescribeTests
SPARK-44841 Support value_counts for pandas 2.0.0 and above.
SPARK-44686 Add the ability to create a RowEncoder in Encoders.scala.
SPARK-41400 Remove Connect Client Catalyst Dependency
SPARK-44538 Reinstate Row.jsonValue and friends
SPARK-44507 Move AnalysisException to sql/api
SPARK-44531 Move encoder inference to sql/api
SPARK-43744 Fix class loading problem caused by stub user classes not found on the server classpath
SPARK-36612 Support left outer join build left or right outer join build right in shuffled hash join
SPARK-44541 Remove useless function hasRangeExprAgainstEventTimeCol from UnsupportedOperationChecker
SPARK-44059 Add analyzer support of named arguments for built-in functions
SPARK-44216 Make assertSchemaEqual API public
SPARK-43755 Move execution out of SparkExecutePlanStreamHandler and to a different thread
SPARK-44201 Add support for Streaming Listener in Scala for Spark Connect
SPARK-43965 Support Python UDTF in Spark Connect
SPARK-44398 Scala foreachBatch API
SPARK-44044 Improve Error message for Window functions with streaming

Databricks ODBC/JDBC driver support

Databricks supports ODBC/JDBC drivers released in the past 2 years. Please download the recently released drivers and upgrade (download ODBC, download JDBC).

System environment

Operating System: Ubuntu 24.04.2 LTS
Java: Zulu17.54+21-CA
Scala: 2.13.16
Python: 3.12.3
R: 4.4.2
Delta Lake: 4.0.0

Installed Python libraries

Library	Version	Library	Version	Library	Version
annotated-types	0.7.0	anyio	4.6.2	argon2-cffi	21.3.0
argon2-cffi-bindings	21.2.0	arrow	1.3.0	asttokens	2.0.5
astunparse	1.6.3	async-lru	2.0.4	attrs	24.3.0
autocommand	2.2.2	azure-common	1.1.28	azure-core	1.34.0
azure-identity	1.20.0	azure-mgmt-core	1.5.0	azure-mgmt-web	8.0.0
azure-storage-blob	12.23.0	azure-storage-file-datalake	12.17.0	babel	2.16.0
backports.tarfile	1.2.0	beautifulsoup4	4.12.3	black	24.10.0
bleach	6.2.0	blinker	1.7.0	boto3	1.36.2
botocore	1.36.3	cachetools	5.5.1	certifi	2025.1.31
cffi	1.17.1	chardet	4.0.0	charset-normalizer	3.3.2
click	8.1.7	cloudpickle	3.0.0	comm	0.2.1
contourpy	1.3.1	cryptography	43.0.3	cycler	0.11.0
Cython	3.0.12	databricks-sdk	0.49.0	dbus-python	1.3.2
debugpy	1.8.11	decorator	5.1.1	defusedxml	0.7.1
Deprecated	1.2.13	distlib	0.3.9	docstring-to-markdown	0.11
executing	0.8.3	facets-overview	1.1.1	fastapi	0.115.12
fastjsonschema	2.21.1	filelock	3.18.0	fonttools	4.55.3
fqdn	1.5.1	fsspec	2023.5.0	gitdb	4.0.11
GitPython	3.1.43	google-api-core	2.20.0	google-auth	2.40.0
google-cloud-core	2.4.3	google-cloud-storage	3.1.0	google-crc32c	1.7.1
google-resumable-media	2.7.2	googleapis-common-protos	1.65.0	grpcio	1.67.0
grpcio-status	1.67.0	h11	0.14.0	httpcore	1.0.2
httplib2	0.20.4	httpx	0.27.0	idna	3.7
importlib-metadata	6.6.0	importlib_resources	6.4.0	inflect	7.3.1
iniconfig	1.1.1	ipyflow-core	0.0.209	ipykernel	6.29.5
ipython	8.30.0	ipython-genutils	0.2.0	ipywidgets	7.8.1
isodate	0.6.1	isoduration	20.11.0	jaraco.context	5.3.0
jaraco.functools	4.0.1	jaraco.text	3.12.1	jedi	0.19.2
Jinja2	3.1.5	jmespath	1.0.1	joblib	1.4.2
json5	0.9.25	jsonpointer	3.0.0	jsonschema	4.23.0
jsonschema-specifications	2023.7.1	jupyter-events	0.10.0	jupyter-lsp	2.2.0
jupyter_client	8.6.3	jupyter_core	5.7.2	jupyter_server	2.14.1
jupyter_server_terminals	0.4.4	jupyterlab	4.3.4	jupyterlab-pygments	0.1.2
jupyterlab-widgets	1.0.0	jupyterlab_server	2.27.3	kiwisolver	1.4.8
launchpadlib	1.11.0	lazr.restfulclient	0.14.6	lazr.uri	1.0.6
markdown-it-py	2.2.0	MarkupSafe	3.0.2	matplotlib	3.10.0
matplotlib-inline	0.1.7	mccabe	0.7.0	mdurl	0.1.0
mistune	2.0.4	mlflow-skinny	2.22.0	mmh3	5.1.0
more-itertools	10.3.0	msal	1.32.3	msal-extensions	1.3.1
mypy-extensions	1.0.0	nbclient	0.8.0	nbconvert	7.16.4
nbformat	5.10.4	nest-asyncio	1.6.0	nodeenv	1.9.1
notebook	7.3.2	notebook_shim	0.2.3	numpy	2.1.3
oauthlib	3.2.2	opentelemetry-api	1.32.1	opentelemetry-sdk	1.32.1
opentelemetry-semantic-conventions	0.53b1	overrides	7.4.0	packaging	24.1
pandas	2.2.3	pandocfilters	1.5.0	parso	0.8.4
pathspec	0.10.3	patsy	1.0.1	pexpect	4.8.0
pillow	11.1.0	pip	24.2	platformdirs	3.10.0
plotly	5.24.1	pluggy	1.5.0	prometheus_client	0.21.0
prompt-toolkit	3.0.43	proto-plus	1.26.1	protobuf	5.29.4
psutil	5.9.0	psycopg2	2.9.3	ptyprocess	0.7.0
pure-eval	0.2.2	pyarrow	19.0.1	pyasn1	0.4.8
pyasn1-modules	0.2.8	pyccolo	0.0.71	pycparser	2.21
pydantic	2.10.6	pydantic_core	2.27.2	pyflakes	3.2.0
Pygments	2.15.1	PyGObject	3.48.2	pyiceberg	0.9.0
PyJWT	2.10.1	pyodbc	5.2.0	pyparsing	3.2.0
pyright	1.1.394	pytest	8.3.5	python-dateutil	2.9.0.post0
python-json-logger	3.2.1	python-lsp-jsonrpc	1.1.2	python-lsp-server	1.12.0
pytoolconfig	1.2.6	pytz	2024.1	PyYAML	6.0.2
pyzmq	26.2.0	referencing	0.30.2	requests	2.32.3
rfc3339-validator	0.1.4	rfc3986-validator	0.1.1	rich	13.9.4
rope	1.12.0	rpds-py	0.22.3	rsa	4.9.1
s3transfer	0.11.3	scikit-learn	1.6.1	scipy	1.15.1
seaborn	0.13.2	Send2Trash	1.8.2	setuptools	74.0.0
six	1.16.0	smmap	5.0.0	sniffio	1.3.0
sortedcontainers	2.4.0	soupsieve	2.5	sqlparse	0.5.3
ssh-import-id	5.11	stack-data	0.2.0	starlette	0.46.2
statsmodels	0.14.4	strictyaml	1.7.3	tenacity	9.0.0
terminado	0.17.1	threadpoolctl	3.5.0	tinycss2	1.4.0
tokenize_rt	6.1.0	tomli	2.0.1	tornado	6.4.2
traitlets	5.14.3	typeguard	4.3.0	types-python-dateutil	2.9.0.20241206
typing_extensions	4.12.2	tzdata	2024.1	ujson	5.10.0
unattended-upgrades	0.1	uri-template	1.3.0	urllib3	2.3.0
uvicorn	0.34.2	virtualenv	20.29.3	wadllib	1.3.6
wcwidth	0.2.5	webcolors	24.11.1	webencodings	0.5.1
websocket-client	1.8.0	whatthepatch	1.0.2	wheel	0.45.1
widgetsnbextension	3.6.6	wrapt	1.17.0	yapf	0.40.2
zipp	3.21.0

Installed R libraries

R libraries are installed from the Posit Package Manager CRAN snapshot on 2025-03-20.

Library	Version	Library	Version	Library	Version
arrow	19.0.1	askpass	1.2.1	assertthat	0.2.1
backports	1.5.0	base	4.4.2	base64enc	0.1-3
bigD	0.3.0	bit	4.6.0	bit64	4.6.0-1
bitops	1.0-9	blob	1.2.4	boot	1.3-30
brew	1.0-10	brio	1.1.5	broom	1.0.7
bslib	0.9.0	cachem	1.1.0	callr	3.7.6
caret	7.0-1	cellranger	1.1.0	chron	2.3-62
class	7.3-22	cli	3.6.4	clipr	0.8.0
clock	0.7.2	cluster	2.1.6	codetools	0.2-20
colorspace	2.1-1	commonmark	1.9.5	compiler	4.4.2
config	0.3.2	conflicted	1.2.0	cpp11	0.5.2
crayon	1.5.3	credentials	2.0.2	curl	6.2.1
data.table	1.17.0	datasets	4.4.2	DBI	1.2.3
dbplyr	2.5.0	desc	1.4.3	devtools	2.4.5
diagram	1.6.5	diffobj	0.3.5	digest	0.6.37
downlit	0.4.4	dplyr	1.1.4	dtplyr	1.3.1
e1071	1.7-16	ellipsis	0.3.2	evaluate	1.0.3
fansi	1.0.6	farver	2.1.2	fastmap	1.2.0
fontawesome	0.5.3	forcats	1.0.0	foreach	1.5.2
foreign	0.8-86	forge	0.2.0	fs	1.6.5
future	1.34.0	future.apply	1.11.3	gargle	1.5.2
generics	0.1.3	gert	2.1.4	ggplot2	3.5.1
gh	1.4.1	git2r	0.35.0	gitcreds	0.1.2
glmnet	4.1-8	globals	0.16.3	glue	1.8.0
googledrive	2.1.1	googlesheets4	1.1.1	gower	1.0.2
graphics	4.4.2	grDevices	4.4.2	grid	4.4.2
gridExtra	2.3	gsubfn	0.7	gt	0.11.1
gtable	0.3.6	hardhat	1.4.1	haven	2.5.4
highr	0.11	hms	1.1.3	htmltools	0.5.8.1
htmlwidgets	1.6.4	httpuv	1.6.15	httr	1.4.7
httr2	1.1.1	ids	1.0.1	ini	0.3.1
ipred	0.9-15	isoband	0.2.7	iterators	1.0.14
jquerylib	0.1.4	jsonlite	1.9.1	juicyjuice	0.1.0
KernSmooth	2.23-22	knitr	1.50	labeling	0.4.3
later	1.4.1	lattice	0.22-5	lava	1.8.1
lifecycle	1.0.4	listenv	0.9.1	lubridate	1.9.4
magrittr	2.0.3	markdown	1.13	MASS	7.3-60.0.1
Matrix	1.6-5	memoise	2.0.1	methods	4.4.2
mgcv	1.9-1	mime	0.13	miniUI	0.1.1.1
mlflow	2.20.4	ModelMetrics	1.2.2.2	modelr	0.1.11
munsell	0.5.1	nlme	3.1-164	nnet	7.3-19
numDeriv	2016.8-1.1	openssl	2.3.2	parallel	4.4.2
parallelly	1.42.0	pillar	1.10.1	pkgbuild	1.4.6
pkgconfig	2.0.3	pkgdown	2.1.1	pkgload	1.4.0
plogr	0.2.0	plyr	1.8.9	praise	1.0.0
prettyunits	1.2.0	pROC	1.18.5	processx	3.8.6
prodlim	2024.06.25	profvis	0.4.0	progress	1.2.3
progressr	0.15.1	promises	1.3.2	proto	1.0.0
proxy	0.4-27	ps	1.9.0	purrr	1.0.4
R6	2.6.1	ragg	1.3.3	randomForest	4.7-1.2
rappdirs	0.3.3	rcmdcheck	1.4.0	RColorBrewer	1.1-3
Rcpp	1.0.14	RcppEigen	0.3.4.0.2	reactable	0.4.4
reactR	0.6.1	readr	2.1.5	readxl	1.4.5
recipes	1.2.0	rematch	2.0.0	rematch2	2.1.2
remotes	2.5.0	reprex	2.1.1	reshape2	1.4.4
rlang	1.1.5	rmarkdown	2.29	RODBC	1.3-26
roxygen2	7.3.2	rpart	4.1.23	rprojroot	2.0.4
Rserve	1.8-15	RSQLite	2.3.9	rstudioapi	0.17.1
rversions	2.1.2	rvest	1.0.4	sass	0.4.9
scales	1.3.0	selectr	0.4-2	sessioninfo	1.2.3
shape	1.4.6.1	shiny	1.10.0	sourcetools	0.1.7-1
sparklyr	1.9.0	SparkR	4.0.0	sparsevctrs	0.3.1
spatial	7.3-17	splines	4.4.2	sqldf	0.4-11
SQUAREM	2021.1	stats	4.4.2	stats4	4.4.2
stringi	1.8.4	stringr	1.5.1	survival	3.5-8
swagger	5.17.14.1	sys	3.4.3	systemfonts	1.2.1
tcltk	4.4.2	testthat	3.2.3	textshaping	1.0.0
tibble	3.2.1	tidyr	1.3.1	tidyselect	1.2.1
tidyverse	2.0.0	timechange	0.3.0	timeDate	4041.110
tinytex	0.56	tools	4.4.2	tzdb	0.5.0
urlchecker	1.0.1	usethis	3.1.0	utf8	1.2.4
utils	4.4.2	uuid	1.2-1	V8	6.0.2
vctrs	0.6.5	viridisLite	0.4.2	vroom	1.6.5
waldo	0.6.1	whisker	0.4.1	withr	3.0.2
xfun	0.51	xml2	1.3.8	xopen	1.0.1
xtable	1.8-4	yaml	2.3.10	zeallot	0.1.0
zip	2.3.2

Installed Java and Scala libraries (Scala 2.13 cluster version)

Group ID	Artifact ID	Version
antlr	antlr	2.7.7
com.amazonaws	amazon-kinesis-client	1.12.0
com.amazonaws	aws-java-sdk-autoscaling	1.12.638
com.amazonaws	aws-java-sdk-cloudformation	1.12.638
com.amazonaws	aws-java-sdk-cloudfront	1.12.638
com.amazonaws	aws-java-sdk-cloudhsm	1.12.638
com.amazonaws	aws-java-sdk-cloudsearch	1.12.638
com.amazonaws	aws-java-sdk-cloudtrail	1.12.638
com.amazonaws	aws-java-sdk-cloudwatch	1.12.638
com.amazonaws	aws-java-sdk-cloudwatchmetrics	1.12.638
com.amazonaws	aws-java-sdk-codedeploy	1.12.638
com.amazonaws	aws-java-sdk-cognitoidentity	1.12.638
com.amazonaws	aws-java-sdk-cognitosync	1.12.638
com.amazonaws	aws-java-sdk-config	1.12.638
com.amazonaws	aws-java-sdk-core	1.12.638
com.amazonaws	aws-java-sdk-datapipeline	1.12.638
com.amazonaws	aws-java-sdk-directconnect	1.12.638
com.amazonaws	aws-java-sdk-directory	1.12.638
com.amazonaws	aws-java-sdk-dynamodb	1.12.638
com.amazonaws	aws-java-sdk-ec2	1.12.638
com.amazonaws	aws-java-sdk-ecs	1.12.638
com.amazonaws	aws-java-sdk-efs	1.12.638
com.amazonaws	aws-java-sdk-elasticache	1.12.638
com.amazonaws	aws-java-sdk-elasticbeanstalk	1.12.638
com.amazonaws	aws-java-sdk-elasticloadbalancing	1.12.638
com.amazonaws	aws-java-sdk-elastictranscoder	1.12.638
com.amazonaws	aws-java-sdk-emr	1.12.638
com.amazonaws	aws-java-sdk-glacier	1.12.638
com.amazonaws	aws-java-sdk-glue	1.12.638
com.amazonaws	aws-java-sdk-iam	1.12.638
com.amazonaws	aws-java-sdk-importexport	1.12.638
com.amazonaws	aws-java-sdk-kinesis	1.12.638
com.amazonaws	aws-java-sdk-kms	1.12.638
com.amazonaws	aws-java-sdk-lambda	1.12.638
com.amazonaws	aws-java-sdk-logs	1.12.638
com.amazonaws	aws-java-sdk-machinelearning	1.12.638
com.amazonaws	aws-java-sdk-opsworks	1.12.638
com.amazonaws	aws-java-sdk-rds	1.12.638
com.amazonaws	aws-java-sdk-redshift	1.12.638
com.amazonaws	aws-java-sdk-route53	1.12.638
com.amazonaws	aws-java-sdk-s3	1.12.638
com.amazonaws	aws-java-sdk-ses	1.12.638
com.amazonaws	aws-java-sdk-simpledb	1.12.638
com.amazonaws	aws-java-sdk-simpleworkflow	1.12.638
com.amazonaws	aws-java-sdk-sns	1.12.638
com.amazonaws	aws-java-sdk-sqs	1.12.638
com.amazonaws	aws-java-sdk-ssm	1.12.638
com.amazonaws	aws-java-sdk-storagegateway	1.12.638
com.amazonaws	aws-java-sdk-sts	1.12.638
com.amazonaws	aws-java-sdk-support	1.12.638
com.amazonaws	aws-java-sdk-swf-libraries	1.11.22
com.amazonaws	aws-java-sdk-workspaces	1.12.638
com.amazonaws	jmespath-java	1.12.638
com.clearspring.analytics	stream	2.9.8
com.databricks	Rserve	1.8-3
com.databricks	databricks-sdk-java	0.27.0
com.databricks	jets3t	0.7.1-0
com.databricks.scalapb	scalapb-runtime_2.13	0.4.15-11
com.esotericsoftware	kryo-shaded	4.0.3
com.esotericsoftware	minlog	1.3.0
com.fasterxml	classmate	1.5.1
com.fasterxml.jackson.core	jackson-annotations	2.18.2
com.fasterxml.jackson.core	jackson-core	2.18.2
com.fasterxml.jackson.core	jackson-databind	2.18.2
com.fasterxml.jackson.dataformat	jackson-dataformat-cbor	2.18.2
com.fasterxml.jackson.dataformat	jackson-dataformat-yaml	2.15.2
com.fasterxml.jackson.datatype	jackson-datatype-joda	2.18.2
com.fasterxml.jackson.datatype	jackson-datatype-jsr310	2.18.2
com.fasterxml.jackson.module	jackson-module-paranamer	2.18.2
com.fasterxml.jackson.module	jackson-module-scala_2.13	2.18.2
com.github.ben-manes.caffeine	caffeine	2.9.3
com.github.blemale	scaffeine_2.13	4.1.0
com.github.fommil	jniloader	1.1
com.github.fommil.netlib	native_ref-java	1.1
com.github.fommil.netlib	native_ref-java	1.1-natives
com.github.fommil.netlib	native_system-java	1.1
com.github.fommil.netlib	native_system-java	1.1-natives
com.github.fommil.netlib	netlib-native_ref-linux-x86_64	1.1-natives
com.github.fommil.netlib	netlib-native_system-linux-x86_64	1.1-natives
com.github.luben	zstd-jni	1.5.6-10
com.github.virtuald	curvesapi	1.08
com.github.wendykierp	JTransforms	3.1
com.google.api.grpc	proto-google-common-protos	2.5.1
com.google.code.findbugs	jsr305	3.0.0
com.google.code.gson	gson	2.11.0
com.google.crypto.tink	tink	1.16.0
com.google.errorprone	error_prone_annotations	2.36.0
com.google.flatbuffers	flatbuffers-java	24.3.25
com.google.guava	failureaccess	1.0.2
com.google.guava	guava	33.4.0-jre
com.google.guava	listenablefuture	9999.0-empty-to-avoid-conflict-with-guava
com.google.j2objc	j2objc-annotations	3.0.0
com.google.protobuf	protobuf-java	3.25.5
com.google.protobuf	protobuf-java-util	3.25.5
com.helger	profiler	1.1.1
com.ibm.icu	icu4j	75.1
com.jcraft	jsch	0.1.55
com.lihaoyi	sourcecode_2.13	0.1.9
com.microsoft.azure	azure-data-lake-store-sdk	2.3.10
com.microsoft.sqlserver	mssql-jdbc	12.8.0.jre11
com.microsoft.sqlserver	mssql-jdbc	12.8.0.jre8
com.ning	compress-lzf	1.1.2
com.sun.mail	javax.mail	1.5.2
com.sun.xml.bind	jaxb-core	2.2.11
com.sun.xml.bind	jaxb-impl	2.2.11
com.tdunning	json	1.8
com.thoughtworks.paranamer	paranamer	2.8
com.trueaccord.lenses	lenses_2.13	0.4.13
com.twitter	chill-java	0.10.0
com.twitter	chill_2.13	0.10.0
com.twitter	util-app_2.13	19.8.1
com.twitter	util-core_2.13	19.8.1
com.twitter	util-function_2.13	19.8.1
com.twitter	util-jvm_2.13	19.8.1
com.twitter	util-lint_2.13	19.8.1
com.twitter	util-registry_2.13	19.8.1
com.twitter	util-stats_2.13	19.8.1
com.typesafe	config	1.4.3
com.typesafe.scala-logging	scala-logging_2.13	3.9.2
com.uber	h3	3.7.3
com.univocity	univocity-parsers	2.9.1
com.zaxxer	HikariCP	4.0.3
com.zaxxer	SparseBitSet	1.3
commons-cli	commons-cli	1.9.0
commons-codec	commons-codec	1.17.2
commons-collections	commons-collections	3.2.2
commons-dbcp	commons-dbcp	1.4
commons-fileupload	commons-fileupload	1.5
commons-httpclient	commons-httpclient	3.1
commons-io	commons-io	2.18.0
commons-lang	commons-lang	2.6
commons-logging	commons-logging	1.1.3
commons-pool	commons-pool	1.5.4
dev.ludovic.netlib	arpack	3.0.3
dev.ludovic.netlib	blas	3.0.3
dev.ludovic.netlib	lapack	3.0.3
info.ganglia.gmetric4j	gmetric4j	1.0.10
io.airlift	aircompressor	2.0.2
io.delta	delta-sharing-client_2.13	1.3.0
io.dropwizard.metrics	metrics-annotation	4.2.30
io.dropwizard.metrics	metrics-core	4.2.30
io.dropwizard.metrics	metrics-graphite	4.2.30
io.dropwizard.metrics	metrics-healthchecks	4.2.30
io.dropwizard.metrics	metrics-jetty9	4.2.30
io.dropwizard.metrics	metrics-jmx	4.2.30
io.dropwizard.metrics	metrics-json	4.2.30
io.dropwizard.metrics	metrics-jvm	4.2.30
io.dropwizard.metrics	metrics-servlets	4.2.30
io.github.java-diff-utils	java-diff-utils	4.15
io.netty	netty-all	4.1.118.Final
io.netty	netty-buffer	4.1.118.Final
io.netty	netty-codec	4.1.118.Final
io.netty	netty-codec-http	4.1.118.Final
io.netty	netty-codec-http2	4.1.118.Final
io.netty	netty-codec-socks	4.1.118.Final
io.netty	netty-common	4.1.118.Final
io.netty	netty-handler	4.1.118.Final
io.netty	netty-handler-proxy	4.1.118.Final
io.netty	netty-resolver	4.1.118.Final
io.netty	netty-tcnative-boringssl-static	2.0.70.Final-db-r0-linux-aarch_64
io.netty	netty-tcnative-boringssl-static	2.0.70.Final-db-r0-linux-x86_64
io.netty	netty-tcnative-boringssl-static	2.0.70.Final-db-r0-osx-aarch_64
io.netty	netty-tcnative-boringssl-static	2.0.70.Final-db-r0-osx-x86_64
io.netty	netty-tcnative-boringssl-static	2.0.70.Final-db-r0-windows-x86_64
io.netty	netty-tcnative-classes	2.0.70.Final
io.netty	netty-transport	4.1.118.Final
io.netty	netty-transport-classes-epoll	4.1.118.Final
io.netty	netty-transport-classes-kqueue	4.1.118.Final
io.netty	netty-transport-native-epoll	4.1.118.Final
io.netty	netty-transport-native-epoll	4.1.118.Final-linux-aarch_64
io.netty	netty-transport-native-epoll	4.1.118.Final-linux-riscv64
io.netty	netty-transport-native-epoll	4.1.118.Final-linux-x86_64
io.netty	netty-transport-native-kqueue	4.1.118.Final-osx-aarch_64
io.netty	netty-transport-native-kqueue	4.1.118.Final-osx-x86_64
io.netty	netty-transport-native-unix-common	4.1.118.Final
io.prometheus	simpleclient	0.16.1-databricks
io.prometheus	simpleclient_common	0.16.1-databricks
io.prometheus	simpleclient_dropwizard	0.16.1-databricks
io.prometheus	simpleclient_pushgateway	0.16.1-databricks
io.prometheus	simpleclient_servlet	0.16.1-databricks
io.prometheus	simpleclient_servlet_common	0.16.1-databricks
io.prometheus	simpleclient_tracer_common	0.16.1-databricks
io.prometheus	simpleclient_tracer_otel	0.16.1-databricks
io.prometheus	simpleclient_tracer_otel_agent	0.16.1-databricks
io.prometheus.jmx	collector	0.18.0
jakarta.annotation	jakarta.annotation-api	1.3.5
jakarta.servlet	jakarta.servlet-api	4.0.3
jakarta.validation	jakarta.validation-api	2.0.2
jakarta.ws.rs	jakarta.ws.rs-api	2.1.6
javax.activation	activation	1.1.1
javax.annotation	javax.annotation-api	1.3.2
javax.el	javax.el-api	2.2.4
javax.jdo	jdo-api	3.0.1
javax.transaction	jta	1.1
javax.transaction	transaction-api	1.1
javax.xml.bind	jaxb-api	2.2.11
javolution	javolution	5.5.1
jline	jline	2.14.6
joda-time	joda-time	2.13.0
net.java.dev.jna	jna	5.8.0
net.razorvine	pickle	1.5
net.sf.jpam	jpam	1.1
net.sf.opencsv	opencsv	2.3
net.sf.supercsv	super-csv	2.2.0
net.snowflake	snowflake-ingest-sdk	0.9.6
net.sourceforge.f2j	arpack_combined_all	0.1
org.acplt.remotetea	remotetea-oncrpc	1.1.2
org.antlr	ST4	4.0.4
org.antlr	antlr-runtime	3.5.2
org.antlr	antlr4-runtime	4.13.1
org.antlr	stringtemplate	3.2.1
org.apache.ant	ant	1.10.11
org.apache.ant	ant-jsch	1.10.11
org.apache.ant	ant-launcher	1.10.11
org.apache.arrow	arrow-format	18.2.0
org.apache.arrow	arrow-memory-core	18.2.0
org.apache.arrow	arrow-memory-netty	18.2.0
org.apache.arrow	arrow-memory-netty-buffer-patch	18.2.0
org.apache.arrow	arrow-vector	18.2.0
org.apache.avro	avro	1.12.0
org.apache.avro	avro-ipc	1.12.0
org.apache.avro	avro-mapred	1.12.0
org.apache.commons	commons-collections4	4.4
org.apache.commons	commons-compress	1.27.1
org.apache.commons	commons-crypto	1.1.0
org.apache.commons	commons-lang3	3.17.0
org.apache.commons	commons-math3	3.6.1
org.apache.commons	commons-text	1.13.0
org.apache.curator	curator-client	5.7.1
org.apache.curator	curator-framework	5.7.1
org.apache.curator	curator-recipes	5.7.1
org.apache.datasketches	datasketches-java	6.1.1
org.apache.datasketches	datasketches-memory	3.0.2
org.apache.derby	derby	10.14.2.0
org.apache.hadoop	hadoop-client-runtime	3.4.1
org.apache.hive	hive-beeline	2.3.10
org.apache.hive	hive-cli	2.3.10
org.apache.hive	hive-jdbc	2.3.10
org.apache.hive	hive-llap-client	2.3.10
org.apache.hive	hive-llap-common	2.3.10
org.apache.hive	hive-serde	2.3.10
org.apache.hive	hive-shims	2.3.10
org.apache.hive	hive-storage-api	2.8.1
org.apache.hive.shims	hive-shims-0.23	2.3.10
org.apache.hive.shims	hive-shims-common	2.3.10
org.apache.hive.shims	hive-shims-scheduler	2.3.10
org.apache.httpcomponents	httpclient	4.5.14
org.apache.httpcomponents	httpcore	4.4.16
org.apache.ivy	ivy	2.5.3
org.apache.logging.log4j	log4j-1.2-api	2.24.3
org.apache.logging.log4j	log4j-api	2.24.3
org.apache.logging.log4j	log4j-core	2.24.3
org.apache.logging.log4j	log4j-layout-template-json	2.24.3
org.apache.logging.log4j	log4j-slf4j2-impl	2.24.3
org.apache.orc	orc-core	2.1.1-shaded-protobuf
org.apache.orc	orc-format	1.1.0-shaded-protobuf
org.apache.orc	orc-mapreduce	2.1.1-shaded-protobuf
org.apache.orc	orc-shims	2.1.1
org.apache.poi	poi	5.4.1
org.apache.poi	poi-ooxml	5.4.1
org.apache.poi	poi-ooxml-full	5.4.1
org.apache.poi	poi-ooxml-lite	5.4.1
org.apache.thrift	libfb303	0.9.3
org.apache.thrift	libthrift	0.16.0
org.apache.ws.xmlschema	xmlschema-core	2.3.1
org.apache.xbean	xbean-asm9-shaded	4.26
org.apache.xmlbeans	xmlbeans	5.3.0
org.apache.yetus	audience-annotations	0.13.0
org.apache.zookeeper	zookeeper	3.9.3
org.apache.zookeeper	zookeeper-jute	3.9.3
org.checkerframework	checker-qual	3.43.0
org.codehaus.janino	commons-compiler	3.0.16
org.codehaus.janino	janino	3.0.16
org.datanucleus	datanucleus-api-jdo	4.2.4
org.datanucleus	datanucleus-core	4.1.17
org.datanucleus	datanucleus-rdbms	4.1.19
org.datanucleus	javax.jdo	3.2.0-m3
org.eclipse.jetty	jetty-client	9.4.53.v20231009
org.eclipse.jetty	jetty-continuation	9.4.53.v20231009
org.eclipse.jetty	jetty-http	9.4.53.v20231009
org.eclipse.jetty	jetty-io	9.4.53.v20231009
org.eclipse.jetty	jetty-jndi	9.4.53.v20231009
org.eclipse.jetty	jetty-plus	9.4.53.v20231009
org.eclipse.jetty	jetty-proxy	9.4.53.v20231009
org.eclipse.jetty	jetty-security	9.4.53.v20231009
org.eclipse.jetty	jetty-server	9.4.53.v20231009
org.eclipse.jetty	jetty-servlet	9.4.53.v20231009
org.eclipse.jetty	jetty-servlets	9.4.53.v20231009
org.eclipse.jetty	jetty-util	9.4.53.v20231009
org.eclipse.jetty	jetty-util-ajax	9.4.53.v20231009
org.eclipse.jetty	jetty-webapp	9.4.53.v20231009
org.eclipse.jetty	jetty-xml	9.4.53.v20231009
org.eclipse.jetty.websocket	websocket-api	9.4.53.v20231009
org.eclipse.jetty.websocket	websocket-client	9.4.53.v20231009
org.eclipse.jetty.websocket	websocket-common	9.4.53.v20231009
org.eclipse.jetty.websocket	websocket-server	9.4.53.v20231009
org.eclipse.jetty.websocket	websocket-servlet	9.4.53.v20231009
org.fusesource.leveldbjni	leveldbjni-all	1.8
org.glassfish.hk2	hk2-api	2.6.1
org.glassfish.hk2	hk2-locator	2.6.1
org.glassfish.hk2	hk2-utils	2.6.1
org.glassfish.hk2	osgi-resource-locator	1.0.3
org.glassfish.hk2.external	aopalliance-repackaged	2.6.1
org.glassfish.hk2.external	jakarta.inject	2.6.1
org.glassfish.jersey.containers	jersey-container-servlet	2.41
org.glassfish.jersey.containers	jersey-container-servlet-core	2.41
org.glassfish.jersey.core	jersey-client	2.41
org.glassfish.jersey.core	jersey-common	2.41
org.glassfish.jersey.core	jersey-server	2.41
org.glassfish.jersey.inject	jersey-hk2	2.41
org.hibernate.validator	hibernate-validator	6.2.5.Final
org.ini4j	ini4j	0.5.4
org.javassist	javassist	3.29.2-GA
org.jboss.logging	jboss-logging	3.4.1.Final
org.jdbi	jdbi	2.63.1
org.jetbrains	annotations	17.0.0
org.jline	jline	3.27.1-jdk8
org.joda	joda-convert	1.7
org.jodd	jodd-core	3.5.2
org.json4s	json4s-ast_2.13	4.0.7
org.json4s	json4s-core_2.13	4.0.7
org.json4s	json4s-jackson-core_2.13	4.0.7
org.json4s	json4s-jackson_2.13	4.0.7
org.json4s	json4s-scalap_2.13	4.0.7
org.lz4	lz4-java	1.8.0-databricks-1
org.mlflow	mlflow-spark_2.13	2.9.1
org.objenesis	objenesis	3.3
org.postgresql	postgresql	42.6.1
org.roaringbitmap	RoaringBitmap	1.2.1
org.rocksdb	rocksdbjni	9.8.4
org.rosuda.REngine	REngine	2.1.0
org.scala-lang	scala-compiler_2.13	2.13.16
org.scala-lang	scala-library_2.13	2.13.16
org.scala-lang	scala-reflect_2.13	2.13.16
org.scala-lang.modules	scala-collection-compat_2.13	2.11.0
org.scala-lang.modules	scala-java8-compat_2.13	0.9.1
org.scala-lang.modules	scala-parallel-collections_2.13	1.2.0
org.scala-lang.modules	scala-parser-combinators_2.13	2.4.0
org.scala-lang.modules	scala-xml_2.13	2.3.0
org.scala-sbt	test-interface	1.0
org.scalacheck	scalacheck_2.13	1.18.0
org.scalactic	scalactic_2.13	3.2.19
org.scalanlp	breeze-macros_2.13	2.1.0
org.scalanlp	breeze_2.13	2.1.0
org.scalatest	scalatest-compatible	3.2.19
org.scalatest	scalatest-core_2.13	3.2.19
org.scalatest	scalatest-diagrams_2.13	3.2.19
org.scalatest	scalatest-featurespec_2.13	3.2.19
org.scalatest	scalatest-flatspec_2.13	3.2.19
org.scalatest	scalatest-freespec_2.13	3.2.19
org.scalatest	scalatest-funspec_2.13	3.2.19
org.scalatest	scalatest-funsuite_2.13	3.2.19
org.scalatest	scalatest-matchers-core_2.13	3.2.19
org.scalatest	scalatest-mustmatchers_2.13	3.2.19
org.scalatest	scalatest-propspec_2.13	3.2.19
org.scalatest	scalatest-refspec_2.13	3.2.19
org.scalatest	scalatest-shouldmatchers_2.13	3.2.19
org.scalatest	scalatest-wordspec_2.13	3.2.19
org.scalatest	scalatest_2.13	3.2.19
org.slf4j	jcl-over-slf4j	2.0.16
org.slf4j	jul-to-slf4j	2.0.16
org.slf4j	slf4j-api	2.0.16
org.slf4j	slf4j-simple	1.7.25
org.threeten	threeten-extra	1.8.0
org.tukaani	xz	1.10
org.typelevel	algebra_2.13	2.8.0
org.typelevel	cats-kernel_2.13	2.8.0
org.typelevel	spire-macros_2.13	0.18.0
org.typelevel	spire-platform_2.13	0.18.0
org.typelevel	spire-util_2.13	0.18.0
org.typelevel	spire_2.13	0.18.0
org.wildfly.openssl	wildfly-openssl	1.1.3.Final
org.xerial	sqlite-jdbc	3.42.0.0
org.xerial.snappy	snappy-java	1.1.10.3
org.yaml	snakeyaml	2.0
oro	oro	2.0.8
pl.edu.icm	JLargeArrays	1.5
software.amazon.cryptools	AmazonCorrettoCryptoProvider	2.4.1-linux-x86_64
stax	stax-api	1.0.1

Tip

Share via