Κοινή χρήση μέσω

Databricks Runtime 13.3 LTS

  • Άρθρο

The following release notes provide information about Databricks Runtime 13.3 LTS, powered by Apache Spark 3.4.1.

Databricks released this version in August 2023.

Note

LTS means this version is under long-term support. See Databricks Runtime LTS version lifecycle.

Tip

To see release notes for Databricks Runtime versions that have reached end-of-support (EoS), see End-of-support Databricks Runtime release notes. The EoS Databricks Runtime versions have been retired and might not be updated.

New features and improvements

Scala support for shared clusters

Unity Catalog-enabled shared clusters now support Scala, when using Databricks Runtime 13.3 and above.

Allowlist for init scripts, JARs, and Maven coordinates on Unity Catalog shared clusters is in Public Preview

You can now use the Unity Catalog allowlist to control installation of init scripts, JARs, and Maven coordinates on compute with shared access mode running Databricks Runtime 13.3 and above. See Allowlist libraries and init scripts on shared compute.

Volumes support for init scripts and JARs is in Public Preview

You can now use Unity Catalog volumes to store init scripts and JARs on compute with assigned or shared access modes running Databricks Runtime 13.3 and above. See Where can init scripts be installed? and Cluster-scoped libraries.

Safe casts enabled by default for Delta Lake operations

Delta UPDATE and MERGE operations now result in an error when values cannot be safely cast to the type in the target table schema. All implicit casts in Delta now follow spark.sql.storeAssignmentPolicy instead of spark.sql.ansi.enabled. See ANSI compliance in Databricks Runtime.

Introducing tags with Unity Catalog

With Databricks Runtime 13.3 LTS and later, you can apply tags to your securable objects. Tags help you organize securable objects and simplify search and discovery of your data assets. See Apply tags to Unity Catalog securable objects.

Databricks Runtime returns an error if a file is modified between query planning and invocation

Databricks Runtime queries now return an error if a file is updated between query planning and invocation. Before this change, Databricks Runtime would read a file between these stages, which occasionally lead to unpredictable results.

Block schema overwrite when using dynamic partition overwrites

You cannot set overwriteSchema to true in combination with dynamic partition overwrites in Delta Lake. This prevents possible table corruption due to schema mismatch. See Dynamic partition overwrites.

Integrate copyFile API into dbutils.fs.cp

The dbutils copy command, dbutils.fs.cp is now optimized for faster copying. With this improvement, copy operations can be up to 100x faster, depending on the file size. The feature is available across all file systems accessible in Azure Databricks, including for Unity Catalog Volumes and Databricks Filesystem mounts.

Databricks ODBC/JDBC driver support

Azure Databricks supports ODBC and JDBC drivers released in the past 2 years. Download the latest driver at Download ODBC Driver and Download JDBC Driver.

Support for unnamed (positional) parameter markers

The SparkSession.sql() API now supports parameterized queries using the ? syntax in addition to parameterized queries using the :parm syntax. See parameter markers.

Enable time series column labeling

You can now label time series columns when creating primary key constraints.

New bitmap SQL functions

Azure Databricks now provided a set of functions which can be used to map integral numeric values onto a bitmap and aggregate them. This can be used to pre-compute number of distinct values.

Improved encryption functions

The aes_encrypt, aes_decrypt, and try_aes_decrypt functions have been improved with support for initialization vectors (IV) and authenticated additional data (AAD).

Unity Catalog support for REFRESH FOREIGN

You can now update foreign catalogs, schemas, and tables in Unity Catalog using REFRESH FOREIGN CATALOG, REFRESH FOREIGN SCHEMA, and REFRESH FOREIGN TABLE. See REFRESH FOREIGN (CATALOG, SCHEMA, and TABLE).

INSERT BY NAME

When inserting rows into a table you can now let Azure Databricks map columns and fields in the source query automatically to the table’s columns by their name. See INSERT INTO.

Share materialized views with Delta Sharing

The ALTER SHARE statement now supports materialized views.

Breaking changes

Changes to implicit casting when storing rows in a table with MERGE INTO and UPDATE

Azure Databricks now follows the configuration spark.sql.storeAssignmentPolicy for implicit casting when storing rows in a table. The default value ANSI throws an error when storing values that overflow. Previously, values would be stored as NULL by default.

As in the following example, you can address overflows by rewriting the table to use a broader type for the column causing the error.

-- Enable column mapping
ALTER TABLE MyTable SET TBLPROPERTIES (
    'delta.minReaderVersion' = '2',
    'delta.minWriterVersion' = '5',
    'delta.columnMapping.mode' = 'name'
  )

-- Rename the old column and create a new column with the new type using the old name
ALTER TABLE MyTable RENAME ID to ID_old
ALTER TABLE MyTable ADD COLUMN ID BIGINT

-- Rewrite the entire table
UPDATE MyTable SET ID = ID_old

-- Drop the old column
ALTER TABLE MyTable DROP COLUMN ID_old

Parquet Schema Inference Changes

When inferring schemas from Parquet files not written by Spark, int64 timestamp columns annotated with isAdjustedToUTC=false will now default to TIMESTAMP_NTZ type. Previously, these were inferred as TIMESTAMP type. This update enhances the accuracy of mapping Parquet types to Spark SQL types.

As a result, reading data from external Parquet files into a Delta table may lead to an error if the timestampNtz feature is not enabled. The error will display as follows:

Fatal exception of the update:
com.databricks.sql.transaction.tahoe.DeltaTableFeatureException
Your table schema requires manual enablement of the following table feature(s): timestampNtz.

To maintain the previous behavior and prevent this error, you can set the Spark configuration parameter spark.sql.parquet.inferTimestampNTZ.enabled to false.

Library upgrades

  • Upgraded Python libraries:
    • debugpy from 1.5.1 to 1.6.0
    • distlib from 0.3.6 to 0.3.7
    • fastjsonschema from 2.16.3 to 2.18.0
    • filelock from 3.12.0 to 3.12.2
  • Upgraded R libraries:
    • SparkR from 3.4.0 to 3.4.1
  • Upgraded Java libraries:
    • org.apache.orc.orc-core from 1.8.3-shaded-protobuf to 1.8.4-shaded-protobuf
    • org.apache.orc.orc-mapreduce from 1.8.3-shaded-protobuf to 1.8.4-shaded-protobuf
    • org.apache.orc.orc-shims from 1.8.3 to 1.8.4
    • org.eclipse.jetty.jetty-client from 9.4.50.v20221201 to 9.4.51.v20230217
    • org.eclipse.jetty.jetty-continuation from 9.4.50.v20221201 to 9.4.51.v20230217
    • org.eclipse.jetty.jetty-http from 9.4.50.v20221201 to 9.4.51.v20230217
    • org.eclipse.jetty.jetty-io from 9.4.50.v20221201 to 9.4.51.v20230217
    • org.eclipse.jetty.jetty-jndi from 9.4.50.v20221201 to 9.4.51.v20230217
    • org.eclipse.jetty.jetty-plus from 9.4.50.v20221201 to 9.4.51.v20230217
    • org.eclipse.jetty.jetty-proxy from 9.4.50.v20221201 to 9.4.51.v20230217
    • org.eclipse.jetty.jetty-security from 9.4.50.v20221201 to 9.4.51.v20230217
    • org.eclipse.jetty.jetty-server from 9.4.50.v20221201 to 9.4.51.v20230217
    • org.eclipse.jetty.jetty-servlet from 9.4.50.v20221201 to 9.4.51.v20230217
    • org.eclipse.jetty.jetty-servlets from 9.4.50.v20221201 to 9.4.51.v20230217
    • org.eclipse.jetty.jetty-util from 9.4.50.v20221201 to 9.4.51.v20230217
    • org.eclipse.jetty.jetty-util-ajax from 9.4.50.v20221201 to 9.4.51.v20230217
    • org.eclipse.jetty.jetty-webapp from 9.4.50.v20221201 to 9.4.51.v20230217
    • org.eclipse.jetty.jetty-xml from 9.4.50.v20221201 to 9.4.51.v20230217
    • org.eclipse.jetty.websocket.websocket-api from 9.4.50.v20221201 to 9.4.51.v20230217
    • org.eclipse.jetty.websocket.websocket-client from 9.4.50.v20221201 to 9.4.51.v20230217
    • org.eclipse.jetty.websocket.websocket-common from 9.4.50.v20221201 to 9.4.51.v20230217
    • org.eclipse.jetty.websocket.websocket-server from 9.4.50.v20221201 to 9.4.51.v20230217
    • org.eclipse.jetty.websocket.websocket-servlet from 9.4.50.v20221201 to 9.4.51.v20230217
    • org.xerial.snappy.snappy-java from 1.1.8.4 to 1.1.10.1

Apache Spark

Databricks Runtime 13.3 includes Apache Spark 3.4.1. This release includes all Spark fixes and improvements included in Databricks Runtime 13.2 (EoS), as well as the following additional bug fixes and improvements made to Spark:

  • You can now set cluster environment variable SNOWFLAKE_SPARK_CONNECTOR_VERSION=2.12 to use Spark-snowflake connector v2.12.0.
  • [SPARK-43380] [DBRRM-383] Revert “[SC-133100][SQL] Fix Avro data type conver…
  • [SPARK-44504] [Backport][13.2][13.x][13.3] Unload provider thereby forcing DB instance close and releasing resources on maintenance task error
  • [SPARK-44059] [SC-137023] Add analyzer support of named arguments for built-in functions
  • [SPARK-44485] [SC-137653][SQL] Optimize TreeNode.generateTreeString
  • [SPARK-42944] [SC-137164][SS][PYTHON] Streaming ForeachBatch in Python
  • [SPARK-44446] [SC-136994][PYTHON] Add checks for expected list type special cases
  • [SPARK-44439] [SC-136973][CONNECT][SS]Fixed listListeners to only send ids back to client
  • [SPARK-44216] [SC-136941] [PYTHON] Make assertSchemaEqual API public
  • [SPARK-44201] [SC-136778][CONNECT][SS]Add support for Streaming Listener in Scala for Spark Connect
  • [SPARK-43915] [SC-134766][SQL] Assign names to the error class LEGACY_ERROR_TEMP[2438-2445]
  • [SPARK-43798] Revert “[SC-133990][SQL][PYTHON] Support Python user-defined table functions”
  • [SPARK-44398] [SC-136720][CONNECT] Scala foreachBatch API
  • [SPARK-44476] [SC-137169][CORE][CONNECT] Fix population of artifacts for a JobArtifactState with no associated artifacts
  • [SPARK-44269] [SC-135817][SQL] Assign names to the error class LEGACY_ERROR_TEMP[2310-2314]
  • [SPARK-44395] [SC-136744][SQL] Update TVF arguments to require parentheses around identifier after TABLE keyword
  • [SPARK-43915] [SC-134766][SQL] Assign names to the error class LEGACY_ERROR_TEMP[2438-2445]
  • [SPARK-43995] [SC-136794][SPARK-43996][CONNECT] Add support for UDFRegistration to the Connect Scala Client
  • [SPARK-44109] [SC-134755][CORE] Remove duplicate preferred locations of each RDD partition
  • [SPARK-44169] [SC-135497][SQL] Assign names to the error class LEGACY_ERROR_TEMP[2300-2304]
  • [SPARK-41487] [SC-135636][SQL] Assign name to _LEGACY_ERROR_TEMP_1020
  • [SPARK-44398] [SC-136720][CONNECT] Scala foreachBatch API
  • [SPARK-44153] [SC-134977][CORE][UI] Support Heap Histogram column in Executors tab
  • [SPARK-44044] [SC-135695][SS] Improve Error message for Window functions with streaming
  • [SPARK-43914] [SC-135414][SQL] Assign names to the error class LEGACY_ERROR_TEMP[2433-2437]
  • [SPARK-44217] [SC-136805][PYTHON] Allow custom precision for fp approx equality
  • [SPARK-44370] [SC-136575][CONNECT] Migrate Buf remote generation alpha to remote plugins
  • [SPARK-44410] [SC-136930][PYTHON][CONNECT] Set active session in create, not just getOrCreate
  • [SPARK-44348] [SAS-1910][SC-136644][13.X][CORE][CONNECT][PYTHON] Reenable test_artifact with relevant changes
  • [SPARK-44145] [SC-136698][SQL] Callback when ready for execution
  • [SPARK-44264] [SC-136870][PYTHON][ML] FunctionPickler Class
  • [SPARK-43986] [SC-135623][SQL] Create error classes for HyperLogLog function call failures
  • [SPARK-44139] [SC-134967][SQL] Discard completely pushed down filters in group-based MERGE operations
  • [SPARK-44338] [SC-136718][SQL] Fix view schema mismatch error message
  • [SPARK-44208] [SC-135505][CORE][SQL] Assign clear error class names for some logic that directly uses exceptions
  • [SPARK-44364] [SC-136773] [PYTHON] Add support for List[Row] data type for expected
  • [SPARK-44180] [SC-136792][SQL] DistributionAndOrderingUtils should apply ResolveTimeZone
  • [SPARK-43389] [SC-136772][SQL] Added a null check for lineSep option
  • [SPARK-44388] [SC-136695][CONNECT] Fix protobuf cast issue when UDF instance is updated
  • [SPARK-44352] [SC-136373][CONNECT] Put back sameType and friends in DataType.
  • [SPARK-43975] [SC-134265][SQL] DataSource V2: Handle UPDATE commands for group-based sources
  • [SPARK-44360] [SC-136461][SQL] Support schema pruning in delta-based MERGE operations
  • [SPARK-44138] [SC-134907][SQL] Prohibit non-deterministic expressions, subqueries and aggregates in MERGE conditions
  • [SPARK-43321] [SC-136678][Connect][Followup] Better names for APIs used in Scala Client joinWith
  • [SPARK-44263] [SC-136447][CONNECT] Custom Interceptors Support
  • [SPARK-39851] [SC-123383][SQL] Improve join stats estimation if one side can keep uniqueness
  • [SPARK-44390] [SC-136677][CORE][SQL] Rename SparkSerDerseUtils to SparkSerDeUtils
  • [SPARK-44004] [SC-134751][SQL] Assign name & improve error message for frequent LEGACY errors.
  • [SPARK-44322] [SC-136204][CONNECT] Make parser use SqlApiConf instead of SQLConf.
  • [SPARK-44271] [SC-136347][SQL] Move default values functions from StructType to ResolveDefaultColumns
  • [SPARK-44332] [SC-136413][CORE][WEBUI] Fix the sorting error of Executor ID Column on Executors UI Page
  • [SPARK-44282] [SC-135948][CONNECT] Prepare DataType parsing for use in Spark Connect Scala Client
  • [SPARK-44349] [SC-136581][R] Add math functions to SparkR
  • [SPARK-44397] [SC-136647][PYTHON] Expose assertDataFrameEqual in pyspark.testing.utils
  • [SPARK-42981] [SC-136620][CONNECT] Add direct arrow serialization
  • [SPARK-44373] [SC-136577][SQL] Wrap withActive for Dataset API w/ parse logic to make parser related configuration work
  • [SPARK-44340] [SC-136574][SQL] Define the computing logic through PartitionEvaluator API and use it in WindowGroupLimitExec
  • [SPARK-43665] [SC-136573][CONNECT][PS] Enable PandasSQLStringFormatter.vformat to work with Spark Connect
  • [SPARK-44325] [SC-136572][SQL] Use PartitionEvaluator API in SortMergeJoinExec
  • [SPARK-44315] [SC-136171][SQL][CONNECT] Move DefinedByConstructorParams to sql/api
  • [SPARK-44275] [SC-136184][CONNECT] Add configurable retry mechanism to Scala Spark Connect
  • [SPARK-44326] [SC-136208][SQL][CONNECT] Move utils that are used from Scala client to the common modules
  • [SPARK-43321] [SC-136156][Connect] Dataset#Joinwith
  • [SPARK-44343] [SC-136372][CONNECT] Prepare ScalaReflection to the move to SQL/API
  • [SPARK-44313] [SC-136081][SQL] Fix generated column expression validation when there is a char/varchar column in the schema
  • [SPARK-43963] [SC-134145][SQL] DataSource V2: Handle MERGE commands for group-based sources
  • [SPARK-44363] [SC-136433] [PYTHON] Display percent of unequal rows in DataFrame comparison
  • [SPARK-44251] [SC-136432][SQL] Set nullable correctly on coalesced join key in full outer USING join
  • [SPARK-43948] [SC-133366][SQL] Assign names to the error class LEGACY_ERROR_TEMP[0050|0057|0058|0059]
  • [SPARK-44329] [SC-136326][CONNECT][PYTHON] Add hll_sketch_agg, hll_union_agg, to_varchar, try_aes_decrypt to Scala and Python
  • [SPARK-44351] [SC-136353][SQL] Make some syntactic simplification
  • [SPARK-44281] [SC-135963][SQL] Move QueryCompilation error that used by DataType to sql/api as DataTypeErrors
  • [SPARK-44283] [SC-136109][CONNECT] Move Origin to SQL/API
  • [SPARK-43926] [SC-135590][CONNECT][PYTHON] Add array_agg, array_size, cardinality, count_min_sketch,mask,named_struct,json_* to Scala and Python
  • [SPARK-44327] [SC-136187][SQL][CONNECT] Add functions any and len to Scala
  • [SPARK-44290] [SC-136300][CONNECT] Session-based files and archives in Spark Connect
  • [SPARK-44061] [SC-136299][PYTHON] Add assertDataFrameEqual util function
  • [SPARK-44331] [SC-136324][CONNECT][PYTHON] Add bitmap functions to Scala and Python
  • [SPARK-44342] [SC-136334][SQL] Replace SQLContext with SparkSession for GenTPCDSData
  • [SPARK-42583] [SC-124190][SQL] Remove the outer join if they are all distinct aggregate functions
  • [SPARK-44333] [SC-136206][CONNECT][SQL] Move EnhancedLogicalPlan out of ParserUtils
  • [SPARK-43824] [SC-132655][SPARK-43825] [SQL] Assign names to the error class _LEGACY_ERROR_TEMP_128[1-2]
  • [SPARK-43939] [SC-134602][CONNECT][PYTHON] Add try_* functions to Scala and Python
  • [SPARK-44193] [SC-135886][CONNECT] Implement GRPC exceptions interception for conversion
  • [SPARK-44284] [SC-136107][CONNECT] Create simple conf system for sql/api
  • [SPARK-43913] [SC-133368][SQL] Assign names to the error class LEGACY_ERROR_TEMP[2426-2432]
  • [SPARK-44291] Revert “[SC-135924][SPARK-43416][CONNECT] Fix incorrect schema for range query”
  • [SPARK-44312] [SC-136185][CONNECT][PYTHON] Allow to set a user agent with an environment variable
  • [SPARK-44154] [SC-136060] Implement bitmap functions
  • [SPARK-43885] [SC-133996][SQL] DataSource V2: Handle MERGE commands for delta-based sources
  • [SPARK-43924] [SC-135161][CONNECT][PYTHON] Add misc functions to Scala and Python
  • [SPARK-43969] [SC-134556][SQL] Refactor & Assign names to the error class _LEGACY_ERROR_TEMP_1170
  • [SPARK-44185] [SC-135730][SQL] Fix inconsistent path qualifying between catalog and data operations
  • [SPARK-44073] [SC-134562][SQL][PYTHON][CONNECT] Add date time functions to Scala, Python and Connect - part 2
  • [SPARK-43942] [SC-134527][CONNECT][PYTHON] Add string functions to Scala and Python - part 1
  • [SPARK-44266] [SC-135813][SQL] Move Util.truncatedString to sql/api
  • [SPARK-44195] [SC-135722][R] Add JobTag APIs to SparkR SparkContext
  • [SPARK-44200] [SC-135736][SQL] Support TABLE argument parser rule for TableValuedFunction
  • [SPARK-44254] [SC-135709][SQL] Move QueryExecutionErrors that used by DataType to sql/api as DataTypeErrors
  • [SPARK-44291] [SC-135924][SPARK-43416][CONNECT] Fix incorrect schema for range query
  • [SPARK-43922] [SC-135612][SQL] Add named parameter support in parser for function calls
  • [SPARK-44300] [SC-135882][CONNECT] Fix artifact cleanup to limit deletion scope to session specific artifacts
  • [SPARK-44293] [SC-135849][CONNECT] Fix invalid URI for custom JARs in Spark Connect
  • [SPARK-44215] [SC-135850][SHUFFLE] If num chunks are 0, then server should throw a RuntimeException
  • [SPARK-44245] [SC-135851][PYTHON] pyspark.sql.dataframe sample() doctests should be illustrative-only
  • [SPARK-44274] [SC-135814][CONNECT] Move out util functions used by ArtifactManager to common/utils
  • [SPARK-44194] [SC-135815][PYTHON][CORE] Add JobTag APIs to PySpark SparkContext
  • [SPARK-42828] [SC-135733][PYTHON][SQL] More explicit Python type annotations for GroupedData
  • [SPARK-44079] [SC-135486][SQL] Fix ArrayIndexOutOfBoundsException when parse array as struct using PERMISSIVE mode with corrupt record
  • [SPARK-44255] [SC-135809][SQL] Relocate StorageLevel to common/utils
  • [SPARK-42169] [SC-135795] [SQL] Implement code generation for to_csv function (StructsToCsv)
  • [SPARK-44249] [SC-135719][SQL][PYTHON] Refactor PythonUDTFRunner to send its return type separately
  • [SPARK-43353] [SC-132734][PYTHON] Migrate remaining session errors into error class
  • [SPARK-44211] [SC-135718][PYTHON][CONNECT] Implement SparkSession.is_stopped
  • [SPARK-42784] [SC-135691] should still create subDir when the number of subDir in merge dir is less than conf
  • [SPARK-41599] [SC-135616] Memory leak in FileSystem.CACHE when submitting apps to secure cluster using InProcessLauncher
  • [SPARK-44241] [SC-135613][Core] Mistakenly set io.connectionTimeout/connectionCreationTimeout to zero or negative will cause incessant executor cons/destructions
  • [SPARK-44133] [SC-134795][13.X][PYTHON] Upgrade MyPy from 0.920 to 0.982
  • [SPARK-42941] [SC-134707][SS][CONNECT][1/2] StreamingQueryListener - Event Serde in JSON format
  • [SPARK-44188] [SC-135173][CORE] Remove useless resetAllPartitions method in ActiveJob
  • [SPARK-43757] [SC-135418][CONNECT] Change client compatibility from allow list to deny list
  • [SPARK-43474] [SC-135521] [SS] [CONNECT] Add a spark connect access to runtime Dataframes by ID.
  • [SPARK-44183] [SC-135209][PYTHON] Increate PyArrow minimum version to 4.0.0
  • [SPARK-44248] [SC-135554][SS][SQL][Kafka] Add preferred location in kafka source v2
  • [SPARK-43929] [SC-134510][SQL][PYTHON][CONNECT] Add date time functions to Scala, Python and Connect API - part 1
  • [SPARK-44161] [SC-135326][Connect] Handle Row input for UDFs
  • [SPARK-44227] [SC-135520][SQL] Extract SchemaUtils from StructField
  • [SPARK-44064] [SC-135116][CORE][SQL] Add a new apply function to NonFateSharingCache
  • [SPARK-44171] [SC-135269][SQL] Assign names to the error class LEGACY_ERROR_TEMP[2279-2282] & delete some unused error classes
  • [SPARK-44056] [SC-134758][SQL] Include UDF name in UDF execution failure error message when available
  • [SPARK-44205] [SC-135422][SQL] Extract Catalyst Code from DecimalType
  • [SPARK-44220] [SC-135484][SQL] Move StringConcat to sql/api
  • [SPARK-44146] [SC-135230][CONNECT] Isolate Spark Connect Session jars and classfiles
  • [SPARK-44206] [SC-135411][SQL] DataSet.selectExpr scope Session.active
  • [SPARK-40850] [SC-135417][SQL] Fix test case interpreted queries may execute Codegen
  • [SPARK-44204] [SC-135253][SQL][HIVE] Add missing recordHiveCall for getPartitionNames
  • [SPARK-44237] [SC-135485][CORE] Simplify DirectByteBuffer constructor lookup logic
  • [SPARK-44189] [SC-135170][CONNECT][PYTHON] Support positional parameters by sql()
  • [SPARK-44140] [SC-134893][SQL][PYTHON] Support positional parameters in Python sql()
  • [SPARK-44164] [SC-135158][SQL] Extract toAttribute method from StructField to Util class
  • [SPARK-44178] [SC-135113][CONNECT] Support positional parameters in sql()
  • [SPARK-43208] [SC-128955][SQL][HIVE] IsolatedClassLoader should close barrier class InputStream after reading
  • [SPARK-43063] [SC-128382][SQL] df.show handle null should print NULL instead of null
  • [SPARK-42539] [SC-124464][SQL][HIVE] Eliminate separate classloader when using ‘builtin’ Hive version for metadata client
  • [SPARK-43992] [SC-133645][SQL][PYTHON][CONNECT] Add optional pattern for Catalog.listFunctions
  • [SPARK-44066] [SC-134878][SQL] Support positional parameters in Scala/Java sql()
  • [SPARK-44078] [SC-134869][CONNECT][CORE] Add support for classloader/resource isolation
  • [SPARK-43470] [SC-135187][CORE] Add OS, Java, Python version information to application log
  • [SPARK-43136] [SC-135238][CONNECT][Followup] Adding tests for KeyAs
  • [SPARK-39740] [SC-135093][UI]: Upgrade vis timeline to 7.7.2 to fix CVE-2020-28487
  • [SPARK-42298] [SC-133952][SQL] Assign name to _LEGACY_ERROR_TEMP_2132
  • [SPARK-43961] [SC-133459][SQL][PYTHON][CONNECT] Add optional pattern for Catalog.listTables
  • [SPARK-44026] [SC-135181] Allow providing initial value for SQLMetrics
  • [SPARK-44202] [SC-135244][CORE] Add JobTag APIs to JavaSparkContext
  • [SPARK-44030] [SC-135237][SQL] Implement DataTypeExpression to offer Unapply for expression
  • [SPARK-43876] Revert “[SC-134603][SQL] Enable fast hashmap for distinct queries”
  • [SPARK-40082] [SC-126437] Schedule mergeFinalize when push merge shuffleMapStage retry but no running tasks
  • [SPARK-43888] [SC-132893][CORE] Relocate Logging to common/utils
  • [SPARK-43474] [SC-134400] [SS] [CONNECT] Add SessionHolder to SparkConnectPlanner
  • [SPARK-43944] [SC-134490][CONNECT][PYTHON] Add string functions to Scala and Python - part 2
  • [SPARK-44134] [SC-134954][CORE] Fix setting resources (GPU/FPGA) to 0 when they are set in spark-defaults.conf
  • [SPARK-44142] [SC-134947][PYTHON] Replace type with tpe in utility to convert python types to spark types
  • [SPARK-43952] [SC-134695][CORE][CONNECT][SQL] Add SparkContext APIs for query cancellation by tag
  • [SPARK-44000] [SC-133634][SQL] Add hint to disable broadcasting and replicating one side of join
  • [SPARK-43937] [SC-134362][CONNECT][PYTHON] Add ifnull,isnotnull,equal_null,nullif,nvl,nvl2 to Scala and Python
  • [SPARK-43624] [13.x][SC-134557][PS][CONNECT] Add EWM to SparkConnectPlanner.
  • [SPARK-44040] [SC-134366][SQL] Fix compute stats when AggregateExec node above QueryStageExec
  • [SPARK-43485] [SC-131264][SQL] Fix the error message for the unit argument of the datetime add/diff functions
  • [SPARK-43794] [SC-132561][SQL] Assign a name to the error class _LEGACY_ERROR_TEMP_1335
  • [SPARK-43511] [SC-134749][CONNECT][SS]Implemented MapGroupsWithState and FlatMapGroupsWithState APIs for Spark Connect
  • [SPARK-43529] [SQL] Support CREATE/REPLACE TABLE OPTIONS expressions + code cleanup
  • [SPARK-44106] [SC-134750][PYTHON][CONNECT] Add __repr__ for GroupedData
  • [SPARK-42299] [SC-133190] Assign name to _LEGACY_ERROR_TEMP_2206
  • [SPARK-43290] [SC-134388][SQL] Adds support for aes_encrypt IVs and AAD
  • [SPARK-43932] [SC-134138][SQL][PYTHON][CONNECT] Add current like functions to Scala and Python
  • [SPARK-43934] [SC-134108][SQL][PYTHON][CONNECT] Add regexp_* functions to Scala and Python
  • [SPARK-44107] [SC-134746][CONNECT][PYTHON] Hide unsupported Column methods from auto-completion
  • [SPARK-42941] [SC-134707][SS][CONNECT][1/2] StreamingQueryListener - Event Serde in JSON format
  • [SPARK-43773] [SC-132659][CONNECT][PYTHON] Implement ‘levenshtein(str1, str2[, threshold])’ functions in python client
  • [SPARK-44125] [SC-134745][R] Support Java 21 in SparkR
  • [SPARK-43919] [SC-133374][SQL] Extract JSON functionality out of Row
  • [SPARK-43769] [SC-132520][CONNECT] Implement ‘levenshtein(str1, str2[, threshold])’ functions
  • [SPARK-44012] [SC-134662][SS] KafkaDataConsumer to print some read status
  • [SPARK-43876] [SC-134603][SQL] Enable fast hashmap for distinct queries
  • [SPARK-44024] [SC-134497][SQL] Change to use map when unzip only used to extract a single element
  • [SPARK-43928] [SC-134467][SQL][PYTHON][CONNECT] Add bit operations to Scala, Python and Connect API
  • [SPARK-44092] [SC-134506][CORE] Add Utils.isJavaVersionAtLeast21 and make core module pass with Java 21
  • [SPARK-44018] [SC-134512][SQL] Improve the hashCode and toString for some DS V2 Expression
  • [SPARK-44105] [SC-134582][SQL] LastNonNull should be lazily resolved
  • [SPARK-44081] [SC-134434] Simplify PartitionedFileUtil API a little
  • [SPARK-43791] [SC-132531][SQL] Assign a name to the error class _LEGACY_ERROR_TEMP_1336
  • [SPARK-44071] [SC-134435] Define and use Unresolved[Leaf|Unary]Node traits.
  • [SPARK-43852] [SC-132890][SPARK-43853][SPARK-43854][SPARK-43855][SPARK-43856] Assign names to the error class _LEGACY_ERROR_TEMP_2418-2425
  • [SPARK-43742] [SC-132684][SQL] Refactor default column value resolution
  • [SPARK-43493] [SC-132063][SQL] Add a max distance argument to the levenshtein() function
  • [SPARK-44075] [SC-134382][CONNECT] Make transformStatCorr lazy
  • [SPARK-43925] [SC-134381][SQL][PYTHON][CONNECT] Add some, bool_or,bool_and,every to Scala, Python and Connect
  • [SPARK-43931] [SC-134158][SQL][PYTHON][CONNECT] Add make_* functions to Scala and Python
  • [SPARK-43762] [SC-132602][SPARK-43763][SPARK-43764][SPARK-43765][SPARK-43766][SQL] Assign names to the error class _LEGACY_ERROR_TEMP_24[06-10]
  • [SPARK-43962] [SC-133367][SQL] Improve error messages: CANNOT_DECODE_URL, CANNOT_MERGE_INCOMPATIBLE_DATA_TYPE, CANNOT_PARSE_DECIMAL, CANNOT_READ_FILE_FOOTER, CANNOT_RECOGNIZE_HIVE_TYPE.
  • [SPARK-43938] [SC-133887][CONNECT][PYTHON] Add to_* functions to Scala and Python
  • [SPARK-44055] [SC-134321][CORE] Remove redundant override functions from CheckpointRDD
  • [SPARK-43802] [SC-132597][SQL] Fix codegen for unhex and unbase64 with failOnError=true
  • [SPARK-43798] [SC-133990][SQL][PYTHON] Support Python user-defined table functions
  • [SPARK-43941] [SC-134286][SQL][PYTHON][CONNECT] Add any_value, approx_percentile, count_if, first_value, histogram_numeric, last_value, reduce to Scala, Python and Connect API
  • [SPARK-43380] [SC-133100][SQL] Fix Avro data type conversion issues to avoid producing incorrect results
  • [SPARK-43803] [SC-133729] [SS] [CONNECT] Improve awaitTermination() to handle client disconnects
  • [SPARK-43179] [SC-129070][SHUFFLE] Allowing apps to control whether their metadata gets saved in the db by the External Shuffle Service

Databricks ODBC/JDBC driver support

Databricks supports ODBC/JDBC drivers released in the past 2 years. Please download the recently released drivers and upgrade (download ODBC, download JDBC).

See Databricks Runtime 13.3 maintenance updates.

System environment

  • Operating System: Ubuntu 22.04.2 LTS
  • Java: Zulu 8.70.0.23-CA-linux64
  • Scala: 2.12.15
  • Python: 3.10.12
  • R: 4.2.2
  • Delta Lake: 2.4.0

Installed Python libraries

Library Version Library Version Library Version
appdirs 1.4.4 argon2-cffi 21.3.0 argon2-cffi-bindings 21.2.0
asttokens 2.2.1 attrs 21.4.0 backcall 0.2.0
beautifulsoup4 4.11.1 black 22.6.0 bleach 4.1.0
blinker 1.4 boto3 1.24.28 botocore 1.27.28
certifi 2022.9.14 cffi 1.15.1 chardet 4.0.0
charset-normalizer 2.0.4 click 8.0.4 cryptography 37.0.1
cycler 0.11.0 Cython 0.29.32 databricks-sdk 0.1.6
dbus-python 1.2.18 debugpy 1.6.0 decorator 5.1.1
defusedxml 0.7.1 distlib 0.3.7 docstring-to-markdown 0.12
entrypoints 0.4 executing 1.2.0 facets-overview 1.0.3
fastjsonschema 2.18.0 filelock 3.12.2 fonttools 4.25.0
googleapis-common-protos 1.56.4 grpcio 1.48.1 grpcio-status 1.48.1
httplib2 0.20.2 idna 3.3 importlib-metadata 4.6.4
ipykernel 6.17.1 ipython 8.10.0 ipython-genutils 0.2.0
ipywidgets 7.7.2 jedi 0.18.1 jeepney 0.7.1
Jinja2 2.11.3 jmespath 0.10.0 joblib 1.2.0
jsonschema 4.16.0 jupyter-client 7.3.4 jupyter_core 4.11.2
jupyterlab-pygments 0.1.2 jupyterlab-widgets 1.0.0 keyring 23.5.0
kiwisolver 1.4.2 launchpadlib 1.10.16 lazr.restfulclient 0.14.4
lazr.uri 1.0.6 MarkupSafe 2.0.1 matplotlib 3.5.2
matplotlib-inline 0.1.6 mccabe 0.7.0 mistune 0.8.4
more-itertools 8.10.0 mypy-extensions 0.4.3 nbclient 0.5.13
nbconvert 6.4.4 nbformat 5.5.0 nest-asyncio 1.5.5
nodeenv 1.8.0 notebook 6.4.12 numpy 1.21.5
oauthlib 3.2.0 packaging 21.3 pandas 1.4.4
pandocfilters 1.5.0 parso 0.8.3 pathspec 0.9.0
patsy 0.5.2 pexpect 4.8.0 pickleshare 0.7.5
Pillow 9.2.0 pip 22.2.2 platformdirs 2.5.2
plotly 5.9.0 pluggy 1.0.0 prometheus-client 0.14.1
prompt-toolkit 3.0.36 protobuf 3.19.4 psutil 5.9.0
psycopg2 2.9.3 ptyprocess 0.7.0 pure-eval 0.2.2
pyarrow 8.0.0 pycparser 2.21 pydantic 1.10.6
pyflakes 3.0.1 Pygments 2.11.2 PyGObject 3.42.1
PyJWT 2.3.0 pyodbc 4.0.32 pyparsing 3.0.9
pyright 1.1.294 pyrsistent 0.18.0 python-dateutil 2.8.2
python-lsp-jsonrpc 1.0.0 python-lsp-server 1.7.1 pytoolconfig 1.2.2
pytz 2022.1 pyzmq 23.2.0 requests 2.28.1
rope 1.7.0 s3transfer 0.6.0 scikit-learn 1.1.1
scipy 1.9.1 seaborn 0.11.2 SecretStorage 3.3.1
Send2Trash 1.8.0 setuptools 63.4.1 six 1.16.0
soupsieve 2.3.1 ssh-import-id 5.11 stack-data 0.6.2
statsmodels 0.13.2 tenacity 8.1.0 terminado 0.13.1
testpath 0.6.0 threadpoolctl 2.2.0 tokenize-rt 4.2.1
tomli 2.0.1 tornado 6.1 traitlets 5.1.1
typing_extensions 4.3.0 ujson 5.4.0 unattended-upgrades 0.1
urllib3 1.26.11 virtualenv 20.16.3 wadllib 1.3.6
wcwidth 0.2.5 webencodings 0.5.1 whatthepatch 1.0.2
wheel 0.37.1 widgetsnbextension 3.6.1 yapf 0.31.0
zipp 1.0.0

Installed R libraries

R libraries are installed from the Microsoft CRAN snapshot on 2023-02-10.

Library Version Library Version Library Version
arrow 10.0.1 askpass 1.1 assertthat 0.2.1
backports 1.4.1 base 4.2.2 base64enc 0.1-3
bit 4.0.5 bit64 4.0.5 blob 1.2.3
boot 1.3-28 brew 1.0-8 brio 1.1.3
broom 1.0.3 bslib 0.4.2 cachem 1.0.6
callr 3.7.3 caret 6.0-93 cellranger 1.1.0
chron 2.3-59 class 7.3-21 cli 3.6.0
clipr 0.8.0 clock 0.6.1 cluster 2.1.4
codetools 0.2-19 colorspace 2.1-0 commonmark 1.8.1
compiler 4.2.2 config 0.3.1 cpp11 0.4.3
crayon 1.5.2 credentials 1.3.2 curl 5.0.0
data.table 1.14.6 datasets 4.2.2 DBI 1.1.3
dbplyr 2.3.0 desc 1.4.2 devtools 2.4.5
diffobj 0.3.5 digest 0.6.31 downlit 0.4.2
dplyr 1.1.0 dtplyr 1.2.2 e1071 1.7-13
ellipsis 0.3.2 evaluate 0.20 fansi 1.0.4
farver 2.1.1 fastmap 1.1.0 fontawesome 0.5.0
forcats 1.0.0 foreach 1.5.2 foreign 0.8-82
forge 0.2.0 fs 1.6.1 future 1.31.0
future.apply 1.10.0 gargle 1.3.0 generics 0.1.3
gert 1.9.2 ggplot2 3.4.0 gh 1.3.1
gitcreds 0.1.2 glmnet 4.1-6 globals 0.16.2
glue 1.6.2 googledrive 2.0.0 googlesheets4 1.0.1
gower 1.0.1 graphics 4.2.2 grDevices 4.2.2
grid 4.2.2 gridExtra 2.3 gsubfn 0.7
gtable 0.3.1 hardhat 1.2.0 haven 2.5.1
highr 0.10 hms 1.1.2 htmltools 0.5.4
htmlwidgets 1.6.1 httpuv 1.6.8 httr 1.4.4
ids 1.0.1 ini 0.3.1 ipred 0.9-13
isoband 0.2.7 iterators 1.0.14 jquerylib 0.1.4
jsonlite 1.8.4 KernSmooth 2.23-20 knitr 1.42
labeling 0.4.2 later 1.3.0 lattice 0.20-45
lava 1.7.1 lifecycle 1.0.3 listenv 0.9.0
lubridate 1.9.1 magrittr 2.0.3 markdown 1.5
MASS 7.3-58.2 Matrix 1.5-1 memoise 2.0.1
methods 4.2.2 mgcv 1.8-41 mime 0.12
miniUI 0.1.1.1 ModelMetrics 1.2.2.2 modelr 0.1.10
munsell 0.5.0 nlme 3.1-162 nnet 7.3-18
numDeriv 2016.8-1.1 openssl 2.0.5 parallel 4.2.2
parallelly 1.34.0 pillar 1.8.1 pkgbuild 1.4.0
pkgconfig 2.0.3 pkgdown 2.0.7 pkgload 1.3.2
plogr 0.2.0 plyr 1.8.8 praise 1.0.0
prettyunits 1.1.1 pROC 1.18.0 processx 3.8.0
prodlim 2019.11.13 profvis 0.3.7 progress 1.2.2
progressr 0.13.0 promises 1.2.0.1 proto 1.0.0
proxy 0.4-27 ps 1.7.2 purrr 1.0.1
r2d3 0.2.6 R6 2.5.1 ragg 1.2.5
randomForest 4.7-1.1 rappdirs 0.3.3 rcmdcheck 1.4.0
RColorBrewer 1.1-3 Rcpp 1.0.10 RcppEigen 0.3.3.9.3
readr 2.1.3 readxl 1.4.2 recipes 1.0.4
rematch 1.0.1 rematch2 2.1.2 remotes 2.4.2
reprex 2.0.2 reshape2 1.4.4 rlang 1.0.6
rmarkdown 2.20 RODBC 1.3-20 roxygen2 7.2.3
rpart 4.1.19 rprojroot 2.0.3 Rserve 1.8-12
RSQLite 2.2.20 rstudioapi 0.14 rversions 2.1.2
rvest 1.0.3 sass 0.4.5 scales 1.2.1
selectr 0.4-2 sessioninfo 1.2.2 shape 1.4.6
shiny 1.7.4 sourcetools 0.1.7-1 sparklyr 1.7.9
SparkR 3.4.1 spatial 7.3-15 splines 4.2.2
sqldf 0.4-11 SQUAREM 2021.1 stats 4.2.2
stats4 4.2.2 stringi 1.7.12 stringr 1.5.0
survival 3.5-3 sys 3.4.1 systemfonts 1.0.4
tcltk 4.2.2 testthat 3.1.6 textshaping 0.3.6
tibble 3.1.8 tidyr 1.3.0 tidyselect 1.2.0
tidyverse 1.3.2 timechange 0.2.0 timeDate 4022.108
tinytex 0.44 tools 4.2.2 tzdb 0.3.0
urlchecker 1.0.1 usethis 2.1.6 utf8 1.2.3
utils 4.2.2 uuid 1.1-0 vctrs 0.5.2
viridisLite 0.4.1 vroom 1.6.1 waldo 0.4.0
whisker 0.4.1 withr 2.5.0 xfun 0.37
xml2 1.3.3 xopen 1.0.0 xtable 1.8-4
yaml 2.3.7 zip 2.2.2

Installed Java and Scala libraries (Scala 2.12 cluster version)

Group ID Artifact ID Version
antlr antlr 2.7.7
com.amazonaws amazon-kinesis-client 1.12.0
com.amazonaws aws-java-sdk-autoscaling 1.12.390
com.amazonaws aws-java-sdk-cloudformation 1.12.390
com.amazonaws aws-java-sdk-cloudfront 1.12.390
com.amazonaws aws-java-sdk-cloudhsm 1.12.390
com.amazonaws aws-java-sdk-cloudsearch 1.12.390
com.amazonaws aws-java-sdk-cloudtrail 1.12.390
com.amazonaws aws-java-sdk-cloudwatch 1.12.390
com.amazonaws aws-java-sdk-cloudwatchmetrics 1.12.390
com.amazonaws aws-java-sdk-codedeploy 1.12.390
com.amazonaws aws-java-sdk-cognitoidentity 1.12.390
com.amazonaws aws-java-sdk-cognitosync 1.12.390
com.amazonaws aws-java-sdk-config 1.12.390
com.amazonaws aws-java-sdk-core 1.12.390
com.amazonaws aws-java-sdk-datapipeline 1.12.390
com.amazonaws aws-java-sdk-directconnect 1.12.390
com.amazonaws aws-java-sdk-directory 1.12.390
com.amazonaws aws-java-sdk-dynamodb 1.12.390
com.amazonaws aws-java-sdk-ec2 1.12.390
com.amazonaws aws-java-sdk-ecs 1.12.390
com.amazonaws aws-java-sdk-efs 1.12.390
com.amazonaws aws-java-sdk-elasticache 1.12.390
com.amazonaws aws-java-sdk-elasticbeanstalk 1.12.390
com.amazonaws aws-java-sdk-elasticloadbalancing 1.12.390
com.amazonaws aws-java-sdk-elastictranscoder 1.12.390
com.amazonaws aws-java-sdk-emr 1.12.390
com.amazonaws aws-java-sdk-glacier 1.12.390
com.amazonaws aws-java-sdk-glue 1.12.390
com.amazonaws aws-java-sdk-iam 1.12.390
com.amazonaws aws-java-sdk-importexport 1.12.390
com.amazonaws aws-java-sdk-kinesis 1.12.390
com.amazonaws aws-java-sdk-kms 1.12.390
com.amazonaws aws-java-sdk-lambda 1.12.390
com.amazonaws aws-java-sdk-logs 1.12.390
com.amazonaws aws-java-sdk-machinelearning 1.12.390
com.amazonaws aws-java-sdk-opsworks 1.12.390
com.amazonaws aws-java-sdk-rds 1.12.390
com.amazonaws aws-java-sdk-redshift 1.12.390
com.amazonaws aws-java-sdk-route53 1.12.390
com.amazonaws aws-java-sdk-s3 1.12.390
com.amazonaws aws-java-sdk-ses 1.12.390
com.amazonaws aws-java-sdk-simpledb 1.12.390
com.amazonaws aws-java-sdk-simpleworkflow 1.12.390
com.amazonaws aws-java-sdk-sns 1.12.390
com.amazonaws aws-java-sdk-sqs 1.12.390
com.amazonaws aws-java-sdk-ssm 1.12.390
com.amazonaws aws-java-sdk-storagegateway 1.12.390
com.amazonaws aws-java-sdk-sts 1.12.390
com.amazonaws aws-java-sdk-support 1.12.390
com.amazonaws aws-java-sdk-swf-libraries 1.11.22
com.amazonaws aws-java-sdk-workspaces 1.12.390
com.amazonaws jmespath-java 1.12.390
com.clearspring.analytics stream 2.9.6
com.databricks Rserve 1.8-3
com.databricks databricks-sdk-java 0.2.0
com.databricks jets3t 0.7.1-0
com.databricks.scalapb compilerplugin_2.12 0.4.15-10
com.databricks.scalapb scalapb-runtime_2.12 0.4.15-10
com.esotericsoftware kryo-shaded 4.0.2
com.esotericsoftware minlog 1.3.0
com.fasterxml classmate 1.3.4
com.fasterxml.jackson.core jackson-annotations 2.14.2
com.fasterxml.jackson.core jackson-core 2.14.2
com.fasterxml.jackson.core jackson-databind 2.14.2
com.fasterxml.jackson.dataformat jackson-dataformat-cbor 2.14.2
com.fasterxml.jackson.datatype jackson-datatype-joda 2.14.2
com.fasterxml.jackson.datatype jackson-datatype-jsr310 2.13.4
com.fasterxml.jackson.module jackson-module-paranamer 2.14.2
com.fasterxml.jackson.module jackson-module-scala_2.12 2.14.2
com.github.ben-manes.caffeine caffeine 2.9.3
com.github.fommil jniloader 1.1
com.github.fommil.netlib native_ref-java 1.1
com.github.fommil.netlib native_ref-java 1.1-natives
com.github.fommil.netlib native_system-java 1.1
com.github.fommil.netlib native_system-java 1.1-natives
com.github.fommil.netlib netlib-native_ref-linux-x86_64 1.1-natives
com.github.fommil.netlib netlib-native_system-linux-x86_64 1.1-natives
com.github.luben zstd-jni 1.5.2-5
com.github.wendykierp JTransforms 3.1
com.google.code.findbugs jsr305 3.0.0
com.google.code.gson gson 2.8.9
com.google.crypto.tink tink 1.7.0
com.google.errorprone error_prone_annotations 2.10.0
com.google.flatbuffers flatbuffers-java 1.12.0
com.google.guava guava 15.0
com.google.protobuf protobuf-java 2.6.1
com.h2database h2 2.1.214
com.helger profiler 1.1.1
com.jcraft jsch 0.1.55
com.jolbox bonecp 0.8.0.RELEASE
com.lihaoyi sourcecode_2.12 0.1.9
com.microsoft.azure azure-data-lake-store-sdk 2.3.9
com.microsoft.sqlserver mssql-jdbc 11.2.2.jre8
com.ning compress-lzf 1.1.2
com.sun.mail javax.mail 1.5.2
com.sun.xml.bind jaxb-core 2.2.11
com.sun.xml.bind jaxb-impl 2.2.11
com.tdunning json 1.8
com.thoughtworks.paranamer paranamer 2.8
com.trueaccord.lenses lenses_2.12 0.4.12
com.twitter chill-java 0.10.0
com.twitter chill_2.12 0.10.0
com.twitter util-app_2.12 7.1.0
com.twitter util-core_2.12 7.1.0
com.twitter util-function_2.12 7.1.0
com.twitter util-jvm_2.12 7.1.0
com.twitter util-lint_2.12 7.1.0
com.twitter util-registry_2.12 7.1.0
com.twitter util-stats_2.12 7.1.0
com.typesafe config 1.2.1
com.typesafe.scala-logging scala-logging_2.12 3.7.2
com.uber h3 3.7.0
com.univocity univocity-parsers 2.9.1
com.zaxxer HikariCP 4.0.3
commons-cli commons-cli 1.5.0
commons-codec commons-codec 1.15
commons-collections commons-collections 3.2.2
commons-dbcp commons-dbcp 1.4
commons-fileupload commons-fileupload 1.5
commons-httpclient commons-httpclient 3.1
commons-io commons-io 2.11.0
commons-lang commons-lang 2.6
commons-logging commons-logging 1.1.3
commons-pool commons-pool 1.5.4
dev.ludovic.netlib arpack 3.0.3
dev.ludovic.netlib blas 3.0.3
dev.ludovic.netlib lapack 3.0.3
info.ganglia.gmetric4j gmetric4j 1.0.10
io.airlift aircompressor 0.21
io.delta delta-sharing-spark_2.12 0.7.1
io.dropwizard.metrics metrics-core 4.2.10
io.dropwizard.metrics metrics-graphite 4.2.10
io.dropwizard.metrics metrics-healthchecks 4.2.10
io.dropwizard.metrics metrics-jetty9 4.2.10
io.dropwizard.metrics metrics-jmx 4.2.10
io.dropwizard.metrics metrics-json 4.2.10
io.dropwizard.metrics metrics-jvm 4.2.10
io.dropwizard.metrics metrics-servlets 4.2.10
io.netty netty-all 4.1.87.Final
io.netty netty-buffer 4.1.87.Final
io.netty netty-codec 4.1.87.Final
io.netty netty-codec-http 4.1.87.Final
io.netty netty-codec-http2 4.1.87.Final
io.netty netty-codec-socks 4.1.87.Final
io.netty netty-common 4.1.87.Final
io.netty netty-handler 4.1.87.Final
io.netty netty-handler-proxy 4.1.87.Final
io.netty netty-resolver 4.1.87.Final
io.netty netty-transport 4.1.87.Final
io.netty netty-transport-classes-epoll 4.1.87.Final
io.netty netty-transport-classes-kqueue 4.1.87.Final
io.netty netty-transport-native-epoll 4.1.87.Final
io.netty netty-transport-native-epoll 4.1.87.Final-linux-aarch_64
io.netty netty-transport-native-epoll 4.1.87.Final-linux-x86_64
io.netty netty-transport-native-kqueue 4.1.87.Final-osx-aarch_64
io.netty netty-transport-native-kqueue 4.1.87.Final-osx-x86_64
io.netty netty-transport-native-unix-common 4.1.87.Final
io.prometheus simpleclient 0.7.0
io.prometheus simpleclient_common 0.7.0
io.prometheus simpleclient_dropwizard 0.7.0
io.prometheus simpleclient_pushgateway 0.7.0
io.prometheus simpleclient_servlet 0.7.0
io.prometheus.jmx collector 0.12.0
jakarta.annotation jakarta.annotation-api 1.3.5
jakarta.servlet jakarta.servlet-api 4.0.3
jakarta.validation jakarta.validation-api 2.0.2
jakarta.ws.rs jakarta.ws.rs-api 2.1.6
javax.activation activation 1.1.1
javax.el javax.el-api 2.2.4
javax.jdo jdo-api 3.0.1
javax.transaction jta 1.1
javax.transaction transaction-api 1.1
javax.xml.bind jaxb-api 2.2.11
javolution javolution 5.5.1
jline jline 2.14.6
joda-time joda-time 2.12.1
ml.combust.mleap mleap-databricks-runtime_2.12 v0.20.0-db2
net.java.dev.jna jna 5.8.0
net.razorvine pickle 1.3
net.sf.jpam jpam 1.1
net.sf.opencsv opencsv 2.3
net.sf.supercsv super-csv 2.2.0
net.snowflake snowflake-ingest-sdk 0.9.6
net.snowflake snowflake-jdbc 3.13.33
net.sourceforge.f2j arpack_combined_all 0.1
org.acplt.remotetea remotetea-oncrpc 1.1.2
org.antlr ST4 4.0.4
org.antlr antlr-runtime 3.5.2
org.antlr antlr4-runtime 4.9.3
org.antlr stringtemplate 3.2.1
org.apache.ant ant 1.9.16
org.apache.ant ant-jsch 1.9.16
org.apache.ant ant-launcher 1.9.16
org.apache.arrow arrow-format 11.0.0
org.apache.arrow arrow-memory-core 11.0.0
org.apache.arrow arrow-memory-netty 11.0.0
org.apache.arrow arrow-vector 11.0.0
org.apache.avro avro 1.11.1
org.apache.avro avro-ipc 1.11.1
org.apache.avro avro-mapred 1.11.1
org.apache.commons commons-collections4 4.4
org.apache.commons commons-compress 1.21
org.apache.commons commons-crypto 1.1.0
org.apache.commons commons-lang3 3.12.0
org.apache.commons commons-math3 3.6.1
org.apache.commons commons-text 1.10.0
org.apache.curator curator-client 2.13.0
org.apache.curator curator-framework 2.13.0
org.apache.curator curator-recipes 2.13.0
org.apache.datasketches datasketches-java 3.1.0
org.apache.datasketches datasketches-memory 2.0.0
org.apache.derby derby 10.14.2.0
org.apache.hadoop hadoop-client-runtime 3.3.4
org.apache.hive hive-beeline 2.3.9
org.apache.hive hive-cli 2.3.9
org.apache.hive hive-jdbc 2.3.9
org.apache.hive hive-llap-client 2.3.9
org.apache.hive hive-llap-common 2.3.9
org.apache.hive hive-serde 2.3.9
org.apache.hive hive-shims 2.3.9
org.apache.hive hive-storage-api 2.8.1
org.apache.hive.shims hive-shims-0.23 2.3.9
org.apache.hive.shims hive-shims-common 2.3.9
org.apache.hive.shims hive-shims-scheduler 2.3.9
org.apache.httpcomponents httpclient 4.5.14
org.apache.httpcomponents httpcore 4.4.16
org.apache.ivy ivy 2.5.1
org.apache.logging.log4j log4j-1.2-api 2.19.0
org.apache.logging.log4j log4j-api 2.19.0
org.apache.logging.log4j log4j-core 2.19.0
org.apache.logging.log4j log4j-slf4j2-impl 2.19.0
org.apache.mesos mesos 1.11.0-shaded-protobuf
org.apache.orc orc-core 1.8.4-shaded-protobuf
org.apache.orc orc-mapreduce 1.8.4-shaded-protobuf
org.apache.orc orc-shims 1.8.4
org.apache.thrift libfb303 0.9.3
org.apache.thrift libthrift 0.12.0
org.apache.xbean xbean-asm9-shaded 4.22
org.apache.yetus audience-annotations 0.13.0
org.apache.zookeeper zookeeper 3.6.3
org.apache.zookeeper zookeeper-jute 3.6.3
org.checkerframework checker-qual 3.19.0
org.codehaus.jackson jackson-core-asl 1.9.13
org.codehaus.jackson jackson-mapper-asl 1.9.13
org.codehaus.janino commons-compiler 3.0.16
org.codehaus.janino janino 3.0.16
org.datanucleus datanucleus-api-jdo 4.2.4
org.datanucleus datanucleus-core 4.1.17
org.datanucleus datanucleus-rdbms 4.1.19
org.datanucleus javax.jdo 3.2.0-m3
org.eclipse.jetty jetty-client 9.4.51.v20230217
org.eclipse.jetty jetty-continuation 9.4.51.v20230217
org.eclipse.jetty jetty-http 9.4.51.v20230217
org.eclipse.jetty jetty-io 9.4.51.v20230217
org.eclipse.jetty jetty-jndi 9.4.51.v20230217
org.eclipse.jetty jetty-plus 9.4.51.v20230217
org.eclipse.jetty jetty-proxy 9.4.51.v20230217
org.eclipse.jetty jetty-security 9.4.51.v20230217
org.eclipse.jetty jetty-server 9.4.51.v20230217
org.eclipse.jetty jetty-servlet 9.4.51.v20230217
org.eclipse.jetty jetty-servlets 9.4.51.v20230217
org.eclipse.jetty jetty-util 9.4.51.v20230217
org.eclipse.jetty jetty-util-ajax 9.4.51.v20230217
org.eclipse.jetty jetty-webapp 9.4.51.v20230217
org.eclipse.jetty jetty-xml 9.4.51.v20230217
org.eclipse.jetty.websocket websocket-api 9.4.51.v20230217
org.eclipse.jetty.websocket websocket-client 9.4.51.v20230217
org.eclipse.jetty.websocket websocket-common 9.4.51.v20230217
org.eclipse.jetty.websocket websocket-server 9.4.51.v20230217
org.eclipse.jetty.websocket websocket-servlet 9.4.51.v20230217
org.fusesource.leveldbjni leveldbjni-all 1.8
org.glassfish.hk2 hk2-api 2.6.1
org.glassfish.hk2 hk2-locator 2.6.1
org.glassfish.hk2 hk2-utils 2.6.1
org.glassfish.hk2 osgi-resource-locator 1.0.3
org.glassfish.hk2.external aopalliance-repackaged 2.6.1
org.glassfish.hk2.external jakarta.inject 2.6.1
org.glassfish.jersey.containers jersey-container-servlet 2.36
org.glassfish.jersey.containers jersey-container-servlet-core 2.36
org.glassfish.jersey.core jersey-client 2.36
org.glassfish.jersey.core jersey-common 2.36
org.glassfish.jersey.core jersey-server 2.36
org.glassfish.jersey.inject jersey-hk2 2.36
org.hibernate.validator hibernate-validator 6.1.7.Final
org.ini4j ini4j 0.5.4
org.javassist javassist 3.25.0-GA
org.jboss.logging jboss-logging 3.3.2.Final
org.jdbi jdbi 2.63.1
org.jetbrains annotations 17.0.0
org.joda joda-convert 1.7
org.jodd jodd-core 3.5.2
org.json4s json4s-ast_2.12 3.7.0-M11
org.json4s json4s-core_2.12 3.7.0-M11
org.json4s json4s-jackson_2.12 3.7.0-M11
org.json4s json4s-scalap_2.12 3.7.0-M11
org.lz4 lz4-java 1.8.0
org.mariadb.jdbc mariadb-java-client 2.7.4
org.mlflow mlflow-spark 2.2.0
org.objenesis objenesis 2.5.1
org.postgresql postgresql 42.3.8
org.roaringbitmap RoaringBitmap 0.9.39
org.roaringbitmap shims 0.9.39
org.rocksdb rocksdbjni 7.8.3
org.rosuda.REngine REngine 2.1.0
org.scala-lang scala-compiler_2.12 2.12.15
org.scala-lang scala-library_2.12 2.12.15
org.scala-lang scala-reflect_2.12 2.12.15
org.scala-lang.modules scala-collection-compat_2.12 2.4.3
org.scala-lang.modules scala-parser-combinators_2.12 1.1.2
org.scala-lang.modules scala-xml_2.12 1.2.0
org.scala-sbt test-interface 1.0
org.scalacheck scalacheck_2.12 1.14.2
org.scalactic scalactic_2.12 3.2.15
org.scalanlp breeze-macros_2.12 2.1.0
org.scalanlp breeze_2.12 2.1.0
org.scalatest scalatest-compatible 3.2.15
org.scalatest scalatest-core_2.12 3.2.15
org.scalatest scalatest-diagrams_2.12 3.2.15
org.scalatest scalatest-featurespec_2.12 3.2.15
org.scalatest scalatest-flatspec_2.12 3.2.15
org.scalatest scalatest-freespec_2.12 3.2.15
org.scalatest scalatest-funspec_2.12 3.2.15
org.scalatest scalatest-funsuite_2.12 3.2.15
org.scalatest scalatest-matchers-core_2.12 3.2.15
org.scalatest scalatest-mustmatchers_2.12 3.2.15
org.scalatest scalatest-propspec_2.12 3.2.15
org.scalatest scalatest-refspec_2.12 3.2.15
org.scalatest scalatest-shouldmatchers_2.12 3.2.15
org.scalatest scalatest-wordspec_2.12 3.2.15
org.scalatest scalatest_2.12 3.2.15
org.slf4j jcl-over-slf4j 2.0.6
org.slf4j jul-to-slf4j 2.0.6
org.slf4j slf4j-api 2.0.6
org.threeten threeten-extra 1.7.1
org.tukaani xz 1.9
org.typelevel algebra_2.12 2.0.1
org.typelevel cats-kernel_2.12 2.1.1
org.typelevel spire-macros_2.12 0.17.0
org.typelevel spire-platform_2.12 0.17.0
org.typelevel spire-util_2.12 0.17.0
org.typelevel spire_2.12 0.17.0
org.wildfly.openssl wildfly-openssl 1.1.3.Final
org.xerial sqlite-jdbc 3.42.0.0
org.xerial.snappy snappy-java 1.1.10.1
org.yaml snakeyaml 1.33
oro oro 2.0.8
pl.edu.icm JLargeArrays 1.5
software.amazon.cryptools AmazonCorrettoCryptoProvider 1.6.1-linux-x86_64
software.amazon.ion ion-java 1.0.2
stax stax-api 1.0.1