Κοινή χρήση μέσω

Databricks Runtime 15.4 LTS

  • Άρθρο

The following release notes provide information about Databricks Runtime 15.4 LTS, powered by Apache Spark 3.5.0.

Databricks released this version in August 2024.

Note

LTS means this version is under long-term support. See Databricks Runtime LTS version lifecycle.

Tip

To see release notes for Databricks Runtime versions that have reached end-of-support (EoS), see End-of-support Databricks Runtime release notes. The EoS Databricks Runtime versions have been retired and might not be updated.

Behavioral changes

Using a VARIANT as the input or output type with a Python UDF, UDAF, or UDTF throws an exception

[Breaking change] In Databricks Runtime 15.3 and above, calling any Python user-defined function (UDF), user-defined aggregate function (UDAF), or user-defined table function (UDTF) that uses a VARIANT type as an argument or return value throws an exception. This change is made to prevent issues that might occur because of an invalid value returned by one of these functions. To learn more about the VARIANT type, see use VARIANTs to store semi-structured data.

Change to the default schema binding mode for views

Views now adapt to schema changes in the underlying query by using schema compensation with regular casting rules. This is a change from the previous default of BINDING mode, which raised errors when a safe cast could not be performed when referencing the view.

See CREATE VIEW and cast function.

Disallow using the undocumented ! syntax instead of NOT outside boolean expressions

With this release, the use of ! as a synonym for NOT outside of boolean expressions is no longer allowed. For example, statements such as the following: CREATE ... IF ! EXISTS, IS ! NULL, a ! NULL column or field property, ! IN and ! BETWEEN, must be replaced with: CREATE ... IF NOT EXISTS, IS NOT NULL, a NOT NULL column or field property, NOT IN and NOT BETWEEN.

This change ensures consistency, aligns with the SQL standard, and makes your SQL more portable.

The boolean prefix operator ! (for example, !is_mgr or !(true AND false)) is unaffected by this change.

Disallow undocumented column definition syntax in views

Databricks supports CREATE VIEW with named columns and column comments. Previously, the specification of column types, NOT NULL constraints, or DEFAULT has been allowed. With this release, you can no longer use this syntax.

This change ensures consistency, aligns with the SQL standard, and supports future enhancements.

Consistent error handling for Base64 decoding in Spark and Photon

This release changes how Photon handles Base64 decoding errors to match the Spark handling of these errors. Before these changes, the Photon and Spark code generation path sometimes failed to raise parsing exceptions, while the Spark interpreted execution correctly raised IllegalArgumentException or ConversionInvalidInputError. This update ensures that Photon consistently raises the same exceptions as Spark during Base64 decoding errors, providing more predictable and reliable error handling.

Adding a CHECK constraint on an invalid column now returns the UNRESOLVED_COLUMN.WITH_SUGGESTION error class

To provide more useful error messaging, in Databricks Runtime 15.3 and above, an ALTER TABLE ADD CONSTRAINT statement that includes a CHECK constraint referencing an invalid column name returns the UNRESOLVED_COLUMN.WITH_SUGGESTION error class. Previously, an INTERNAL_ERROR was returned.

New features and improvements

UTF-8 validation functions

This release introduces the following functions for validating UTF-8 strings:

  • is_valid_utf8 verified whether a string is a valid UTF-8 string.
  • make_valid_utf8 converts a potentially invalid UTF-8 string to a valid UTF-8 string using substitution characters
  • validate_utf8 raises an error if the input is not a valid UTF-8 string.
  • try_validate_utf8 returns NULL if the input is not a valid UTF-8 string.

Typed Dataset APIs with Scala UDFs

This release includes added support for typed Dataset APIs with Scala user-defined functions (excluding user-defined aggregate functions) on Unity Catalog-enabled compute with shared access mode. See Typed Dataset APIs.

Enable UniForm Iceberg using ALTER TABLE

You can now enable UniForm Iceberg on existing tables without rewriting data files. See Enable by altering an existing table.

try_url_decode function

This release introduces the try_url_decode function, which decodes a URL-encoded string. If the string is not in the correct format, the function returns NULL instead of raising an error.

Optionally allow the optimizer to rely on unenforced foreign key constraints

To improve query performance, you can now specify the RELY keyword on FOREIGN KEY constraints when you CREATE or ALTER a table.

Parallelized job runs for selective overwrites

Selective overwrites using replaceWhere now run jobs that delete data and insert new data in parallel, improving query performance and cluster utilization.

Improved performance for change data feed with selective overwrites

Selective overwrites using replaceWhere on tables with change data feed no longer write separate change data files for inserted data. These operations use a hidden _change_type column present in the underlying Parquet data files to record changes without write amplification.

Improved query latency for the COPY INTO command

This release includes a change that improves the query latency for the COPY INTO command. This improvement is implemented by making the loading of state by the RocksDB state store asynchronous. With this change, you should see an improvement in start times for queries with large states, such as queries with a large number of already ingested files.

Support for dropping the check constraints table feature

You can now drop the checkConstraints table feature from a Delta table using ALTER TABLE table_name DROP FEATURE checkConstraints. See Disable check constraints.

Single user compute supports fine-grained access control, materialized views, and streaming tables (Public Preview)

When a workspace is enabled for serverless compute, Databricks Runtime 15.4 LTS adds support for fine-grained access control on single user compute. When a query accesses any of the following objects, the single user compute resource on Databricks Runtime 15.4 LTS passes the query to the serverless compute to run data filtering:

  • Views defined over tables on which the user does not have the SELECT privilege.
  • Dynamic views.
  • Tables with row filters or column masks applied.
  • Materialized views and streaming tables.

These queries are unsupported on single user compute running Databricks Runtime 15.3 and below.

For more information, see Fine-grained access control on single user compute.

Expanded support for Java and Scala libraries

Starting in Databricks Runtime 15.4 LTS, all Java and Scala libraries bundled with Databricks Runtime are available in all compute access modes when you use Unity Catalog. To learn more about language support on Unity Catalog-enabled compute, see Compute access mode limitations for Unity Catalog.

Expanded support for Scala Dataset operations

With this release, Unity Catalog-enabled compute resources that use shared access mode support the following Scala Dataset operations: map, mapPartitions, foreachPartition, flatMap, reduce and filter.

Scala is GA on Unity Catalog shared compute

With this release, Scala is generally available on shared access mode Unity Catalog-enabled compute, including support for scalar user-defined functions (UDFs). Structured Streaming, Hive UDFs, and Hive user-defined aggregate functions are not supported. For a complete list of limitations, see Compute access mode limitations for Unity Catalog.

Bug fixes

Library upgrades

  • Upgraded Python libraries:
    • azure-core from 1.30.1 to 1.30.2
    • google-auth from 2.29.0 to 2.31.0
    • google-cloud-storage from 2.16.0 to 2.17.0
    • google-resumable-media from 2.7.0 to 2.7.1
    • googleapis-common-protos from 1.63.0 to 1.63.2
    • mlflow-skinny from 2.11.3 to 2.11.4
    • proto-plus from 1.23.0 to 1.24.0
    • s3transfer from 0.10.1 to 0.10.2
  • Upgraded R libraries:
  • Upgraded Java libraries:
    • com.databricks.databricks-sdk-java from 0.17.1 to 0.27.0
    • com.ibm.icu.icu4j from 72.1 to 75.1
    • io.delta.delta-sharing-client_2.12 from 1.0.5 to 1.1.1
    • software.amazon.cryptools.AmazonCorrettoCryptoProvider from 1.6.1-linux-x86_64 to 1.6.2-linux-x86_64

Apache Spark

Databricks Runtime 15.4 LTS includes Apache Spark 3.5.0. This release includes all Spark fixes and improvements included in Databricks Runtime 15.3, and the following additional bug fixes and improvements made to Spark:

  • [SPARK-49054] [DBRRM-1153][SQL] Column default value should support current_* functions
  • [SPARK-48503] [DBRRM-1150][SC-172196][SQL] Allow grouping on expressions in scalar subqueries, if they are bound to outer rows
  • [SPARK-48873] [SC-171061][SQL] Use UnsafeRow in JSON parser.
  • [SPARK-48834] [BEHAVE-79][SC-170972][SQL] Disable variant input/output to python scalar UDFs, UDTFs, UDAFs during query compilation
  • [SPARK-48441] [SC-170980][SQL][WARMFIX] Fix StringTrim behaviour for non-UTF8_BINARY collations
  • [SPARK-48440] [SC-170895][SQL][WARMFIX] Fix StringTranslate behaviour for non-UTF8_BINARY collations
  • [SPARK-48872] [SC-170866][PYTHON] Reduce the overhead of _capture_call_site
  • [SPARK-48862] [SC-170845][PYTHON][CONNECT] Avoid calling _proto_to_string when INFO level is not enabled
  • [SPARK-48852] [SC-170837][CONNECT] Fix string trim function in connect
  • [SPARK-48791] [SC-170658][CORE] Fix perf regression caused by the accumulators registration overhead using CopyOnWriteArrayList
  • [SPARK-48118] [SQL] Support SPARK_SQL_LEGACY_CREATE_HIVE_TABLE env variable
  • [SPARK-48241] [SC-165811][SQL] CSV parsing failure with char/varchar type columns
  • [SPARK-48168] [SC-166900][SQL] Add bitwise shifting operators support
  • [SPARK-48148] [SC-165630][CORE] JSON objects should not be modified when read as STRING
  • [SPARK-46625] [SC-170561] CTE with Identifier clause as reference
  • [SPARK-48771] [SC-170546][SQL] Speed up LogicalPlanIntegrity.validateExprIdUniqueness for large query plans
  • [SPARK-48831] [BEHAVE-76][SC-170554][CONNECT] Make default column name of cast compatible with Spark Classic
  • [SPARK-48623] [SC-170544][CORE] Structured logging migrations [Part 2]
  • [SPARK-48296] [SC-166138][SQL] Codegen Support for to_xml
  • [SPARK-48027] [SC-165154][SQL] InjectRuntimeFilter for multi-level join should check child join type
  • [SPARK-46957] [SC-169807][CORE][15.x][15.3][15.2] Decommission migrated shuffle files should be able to cleanup from executor
  • [SPARK-48686] [SC-170365][SQL] Improve performance of ParserUtils.unescapeSQLString
  • [SPARK-48798] [SC-170588][PYTHON] Introduce spark.profile.render for SparkSession-based profiling
  • [SPARK-48048] [SC-169099] Revert “[SC-164846][CONNECT][SS] Added client side listener support for Scala”
  • [SPARK-47910] [SC-168929][CORE] close stream when DiskBlockObjectWriter closeResources to avoid memory leak
  • [SPARK-48816] [SC-170547][SQL] Shorthand for interval converters in UnivocityParser
  • [SPARK-48589] [SC-170132][SQL][SS] Add option snapshotStartBatchId and snapshotPartitionId to state data source
  • [SPARK-48280] [SC-170293][SQL] Improve collation testing surface area using expression walking
  • [SPARK-48837] [SC-170540][ML] In CountVectorizer, only read binary parameter once per transform, not once per row
  • [SPARK-48803] [SC-170541][SQL] Throw internal error in Orc(De)serializer to align with ParquetWriteSupport
  • [SPARK-48764] [SC-170129][PYTHON] Filtering out IPython-related frames from user stack
  • [SPARK-48818] [SC-170414][PYTHON] Simplify percentile functions
  • [SPARK-48479] [SC-169079][SQL] Support creating scalar and table SQL UDFs in parser
  • [SPARK-48697] [SC-170122][LC-4703][SQL] Add collation aware string filters
  • [SPARK-48800] [SC-170409][CONNECT][SS] Deflake ClientStreamingQuerySuite
  • [SPARK-48738] [SC-169814][SQL] Correct since version for built-in func alias random, position, mod, cardinality, current_schema, user, session_user,char_length,character_length
  • [SPARK-48638] [SC-169575][CONNECT] Add ExecutionInfo support for DataFrame
  • [SPARK-48064] [SC-164697][SQL] Update error messages for routine related error classes
  • [SPARK-48810] [CONNECT] Session stop() API should be idempotent and not fail if the session is already closed by the server
  • [SPARK-48650] [15.x][PYTHON] Display correct call site from IPython Notebook
  • [SPARK-48271] [SC-166076][SQL] Turn match error in RowEncoder into UNSUPPORTED_DATA_TYPE_FOR_ENCODER
  • [SPARK-48709] [SC-169603][SQL] Fix varchar type resolution mismatch for DataSourceV2 CTAS
  • [SPARK-48792] [SC-170335][SQL] Fix regression for INSERT with partial column list to a table with char/varchar
  • [SPARK-48767] [SC-170330][SQL] Fix some error prompts when variant type data is invalid
  • [SPARK-48719] [SC-170339][SQL] Fix the calculation bug of RegrSlope & RegrIntercept when the first parameter is null
  • [SPARK-48815] [SC-170334][CONNECT] Update environment when stoping connect session
  • [SPARK-48646] [SC-169020][PYTHON] Refine Python data source API docstring and type hints
  • [SPARK-48806] [SC-170310][SQL] Pass actual exception when url_decode fails
  • [SPARK-48653] [SC-169267][PYTHON] Fix invalid Python data source error class references
  • [SPARK-47777] [SC-168818] fix python streaming data source connect test
  • [SPARK-48732] [SC-169793][SQL] Cleanup deprecated api usage related to JdbcDialect.compileAggregate
  • [SPARK-48675] [SC-169538][SQL] Fix cache table with collated column
  • [SPARK-48623] [SC-169034][CORE] Structured Logging Migrations
  • [SPARK-48655] [SC-169542][SQL] SPJ: Add tests for shuffle skipping for aggregate queries
  • [SPARK-48586] [SC-169808][SS] Remove lock acquisition in doMaintenance() by making a deep copy of file mappings in RocksDBFileManager in load()
  • [SPARK-48799] [Backport][15.x][SC-170283][SS] Refactor versioning for operator metadata read/write and callers
  • [SPARK-48808] [SC-170309][SQL] Fix NPE when connecting thriftserver through Hive 1.2.1 and the result schema is empty
  • [SPARK-48715] [SC-170291][SQL] Integrate UTF8String validation into collation-aware string function implementations
  • [SPARK-48747] [SC-170120][SQL] Add code point iterator to UTF8String
  • [SPARK-48748] [SC-170115][SQL] Cache numChars in UTF8String
  • [SPARK-48744] [SC-169817][Core] Log entry should be constructed only once
  • [SPARK-46122] [SC-164313][SQL] Set spark.sql.legacy.createHiveTableByDefault to false by default
  • [SPARK-48765] [SC-170119][DEPLOY] Enhance default value evaluation for SPARK_IDENT_STRING
  • [SPARK-48759] [SC-170128][SQL] Add migration doc for CREATE TABLE AS SELECT behavior change behavior change since Spark 3.4
  • [SPARK-48598] [SC-169484][PYTHON][CONNECT] Propagate cached schema in dataframe operations
  • [SPARK-48766] [SC-170126][PYTHON] Document the behavior difference of extraction between element_at and try_element_at
  • [SPARK-48768] [SC-170124][PYTHON][CONNECT] Should not cache explain
  • [SPARK-48770] [Backport][15.x][SC-170133][SS] Change to read operator metadata once on driver to check if we can find info for numColsPrefixKey used for session window agg queries
  • [SPARK-48656] [SC-169529][CORE] Do a length check and throw COLLECTION_SIZE_LIMIT_EXCEEDED error in CartesianRDD.getPartitions
  • [SPARK-48597] [SC-168817][SQL] Introduce a marker for isStreaming property in text representation of logical plan
  • [SPARK-48472] [SC-169044][SQL] Enable reflect expressions with collated strings
  • [SPARK-48699] [SC-169597][SQL] Refine collation API
  • [SPARK-48682] [SC-169812][SQL][BEHAVE-58] Use ICU in InitCap expression for UTF8_BINARY strings
  • [SPARK-48282] [SC-169813][SQL] Alter string search logic for UTF8_BINARY_LCASE collation (StringReplace, FindInSet)
  • [SPARK-47353] [SC-169599][SQL] Enable collation support for the Mode expression
  • [SPARK-48320] [SPARK-48490] Sync the latest Logging trait and test cases from OSS Spark
  • [SPARK-48629] [SC-169479] Migrate the residual code to structured logging framework
  • [SPARK-48681] [SC-169469][SQL][BEHAVE-58] Use ICU in Lower/Upper expressions for UTF8_BINARY strings
  • [SPARK-48573] [15.x][SC-169582][SQL] Upgrade ICU version
  • [SPARK-48687] [Backport][15.x][SS] Add change to perform state schema validation and update on driver for stateful queries
  • [SPARK-47579] [15.x][SC-167310][CORE][PART4] Migrate logInfo with variables to structured logging framework
  • [SPARK-48008] [SC-167363][1/2] Support UDAFs in Spark Connect
  • [SPARK-48713] [SQL] Add index range check for UnsafeRow.pointTo when baseObject is byte array
  • [SPARK-48718] [ES-1159458][SC-169581][SQL] Handle and fix the case when deserializer in cogroup is resolved during application of DeduplicateRelation rule
  • [SPARK-48578] [SC-169505][SQL] add UTF8 string validation related functions
  • [SPARK-48670] [SC-169598][SQL] Providing suggestion as part of error message when invalid collation name is given
  • [SPARK-48059] [SPARK-48145][SPARK-48134][SPARK-48182][SPARK-48209][SPA… …RK-48291] Structured log framework on the java side
  • [SPARK-47599] [15.x][SC-166000][MLLIB] MLLib: Migrate logWarn with variables to structured logging framework
  • [SPARK-48705] [SC-169590][PYTHON] Explicitly use worker_main when it starts with pyspark
  • [SPARK-48706] [SC-169589][PYTHON] Python UDF in higher order functions should not throw internal error
  • [SPARK-48498] [BEHAVE-38][SC-168060][SQL] Always do char padding in predicates
  • [SPARK-48662] [SC-169533][SQL] Fix StructsToXml expression with collations
  • [SPARK-48482] [SC-167702][PYTHON][15.x] dropDuplicates and dropDuplicatesWIthinWatermark should accept variable length args
  • [SPARK-48678] [SC-169463][CORE] Performance optimizations for SparkConf.get(ConfigEntry)
  • [SPARK-48576] [SQL] Rename UTF8_BINARY_LCASE to UTF8_LCASE
  • [SPARK-47927] [SC-164123][SQL]: Fix nullability attribute in UDF decoder
  • [SPARK-47579] [SC-165297][CORE][PART1] Migrate logInfo with variables to structured logging framework (new)
  • [SPARK-48695] [SC-169473][PYTHON] TimestampNTZType.fromInternal not use the deprecated methods
  • [SPARK-48431] [SC-167290][LC-4066][SQL] Do not forward predicates on collated columns to file readers
  • [SPARK-47579] Revert “[SC-165297][CORE][PART1] Migrate logInfo with variables to structured logging framework”
  • [SPARK-47585] [SC-164306][SQL] SQL core: Migrate logInfo with variables to structured logging framework
  • [SPARK-48466] [SC-169042][SQL] Create dedicated node for EmptyRelation in AQE
  • [SPARK-47579] [SC-165297][CORE][PART1] Migrate logInfo with variables to structured logging framework
  • [SPARK-48410] [SC-168320][SQL] Fix InitCap expression for UTF8_BINARY_LCASE & ICU collations
  • [SPARK-48318] [SC-167709][SQL] Enable hash join support for all collations (complex types)
  • [SPARK-48435] [SC-168128][SQL] UNICODE collation should not support binary equality
  • [SPARK-48555] [SC-169041][SQL][PYTHON][CONNECT] Support using Columns as parameters for several functions in pyspark/scala
  • [SPARK-48591] [SC-169081][PYTHON] Add a helper function to simplify Column.py
  • [SPARK-48574] [SC-169043][SQL] Fix support for StructTypes with collations
  • [SPARK-48305] [SC-166390][SQL] Add collation support for CurrentLike expressions
  • [SPARK-48342] [SC-168941][SQL] Introduction of SQL Scripting Parser
  • [SPARK-48649] [SC-169024][SQL] Add “ignoreInvalidPartitionPaths” and “spark.sql.files.ignoreInvalidPartitionPaths” configs to allow ignoring invalid partition paths
  • [SPARK-48000] [SC-167194][SQL] Enable hash join support for all collations (StringType)
  • [SPARK-48459] [SC-168947][CONNECT][PYTHON] Implement DataFrameQueryContext in Spark Connect
  • [SPARK-48602] [SC-168692][SQL] Make csv generator support different output style with spark.sql.binaryOutputStyle
  • [SPARK-48283] [SC-168129][SQL] Modify string comparison for UTF8_BINARY_LCASE
  • [SPARK-48610] [SC-168830][SQL] refactor: use auxiliary idMap instead of OP_ID_TAG
  • [SPARK-48634] [SC-169021][PYTHON][CONNECT] Avoid statically initialize threadpool at ExecutePlanResponseReattachableIterator
  • [SPARK-48648] [SC-168932][PYTHON][CONNECT] Make SparkConnectClient.tags properly threadlocal
  • [SPARK-47911] [SC-164658][SQL] Introduces a universal BinaryFormatter to make binary output consistent
  • [SPARK-48642] [SC-168889][CORE] False SparkOutOfMemoryError caused by killing task on spilling
  • [SPARK-48572] [SC-168844][SQL] Fix DateSub, DateAdd, WindowTime, TimeWindow and SessionWindow expressions
  • [SPARK-48600] [SC-168841][SQL] Fix FrameLessOffsetWindowFunction expressions implicit casting
  • [SPARK-48644] [SC-168933][SQL] Do a length check and throw COLLECTION_SIZE_LIMIT_EXCEEDED error in Hex.hex
  • [SPARK-48587] [SC-168824][VARIANT] Avoid storage amplification when accessing a sub-Variant
  • [SPARK-48647] [SC-168936][PYTHON][CONNECT] Refine the error message for YearMonthIntervalType in df.collect
  • [SPARK-48307] [SC-167802][SQL] InlineCTE should keep not-inlined relations in the original WithCTE node
  • [SPARK-48596] [SC-168581][SQL] Perf improvement for calculating hex string for long
  • [SPARK-48621] [SC-168726][SQL] Fix Like simplification in Optimizer for collated strings
  • [SPARK-47148] [SC-164179][SQL] Avoid to materialize AQE ExchangeQueryStageExec on the cancellation
  • [SPARK-48584] [SC-168579][SQL] Perf improvement for unescapePathName
  • [SPARK-48281] [SC-167260][SQL] Alter string search logic for UTF8_BINARY_LCASE collation (StringInStr, SubstringIndex)
  • [SPARK-48577] [SC-168826][SQL] Invalid UTF-8 byte sequence replacement
  • [SPARK-48595] [SC-168580][CORE] Cleanup deprecated api usage related to commons-compress
  • [SPARK-48030] [SC-164303][SQL] SPJ: cache rowOrdering and structType for InternalRowComparableWrapper
  • [SPARK-48004] [SC-164005][SQL] Add WriteFilesExecBase trait for v1 write
  • [SPARK-48551] [SC-168438][SQL] Perf improvement for escapePathName
  • [SPARK-48565] [SC-168437][UI] Fix thread dump display in UI
  • [SPARK-48364] [SC-166782][SQL] Add AbstractMapType type casting and fix RaiseError parameter map to work with collated strings
  • [SPARK-48421] [SC-168689][SQL] SPJ: Add documentation
  • [SPARK-48604] [SC-168698][SQL] Replace deprecated new ArrowType.Decimal(precision, scale) method call
  • [SPARK-46947] [SC-157561][CORE] Delay memory manager initialization until Driver plugin is loaded
  • [SPARK-48411] [SC-168576][SS][PYTHON] Add E2E test for DropDuplicateWithinWatermark
  • [SPARK-48543] [SC-168697][SS] Track state row validation failures using explicit error class
  • [SPARK-48221] [SC-167143][SQL] Alter string search logic for UTF8_BINARY_LCASE collation (Contains, StartsWith, EndsWith, StringLocate)
  • [SPARK-47415] [SC-168441][SQL] Add collation support for Levenshtein expression
  • [SPARK-48593] [SC-168719][PYTHON][CONNECT] Fix the string representation of lambda function
  • [SPARK-48622] [SC-168710][SQL] get SQLConf once when resolving column names
  • [SPARK-48594] [SC-168685][PYTHON][CONNECT] Rename parent field to child in ColumnAlias
  • [SPARK-48403] [SC-168319][SQL] Fix Lower & Upper expressions for UTF8_BINARY_LCASE & ICU collations
  • [SPARK-48162] [SC-166062][SQL] Add collation support for MISC expressions
  • [SPARK-48518] [SC-167718][CORE] Make LZF compression be able to run in parallel
  • [SPARK-48474] [SC-167447][CORE] Fix the class name of the log in SparkSubmitArguments & SparkSubmit
  • [SPARK-48012] [SC-168267][SQL] SPJ: Support Transfrom Expressions for One Side Shuffle
  • [SPARK-48552] [SC-168212][SQL] multi-line CSV schema inference should also throw FAILED_READ_FILE
  • [SPARK-48560] [SC-168268][SS][PYTHON] Make StreamingQueryListener.spark settable
  • [SPARK-48569] [SC-168321][SS][CONNECT] Handle edge cases in query.name
  • [SPARK-48544] [SC-168323][SQL] Reduce memory pressure of empty TreeNode BitSets
  • [SPARK-47260] [SC-167323][SQL] Assign name to error class _LEGACY_ERROR_TEMP_3250
  • [SPARK-48564] [SC-168327][PYTHON][CONNECT] Propagate cached schema in set operations
  • [SPARK-48155] [SC-165910][SQL] AQEPropagateEmptyRelation for join should check if remain child is just BroadcastQueryStageExec
  • [SPARK-48506] [SC-167720][CORE] Compression codec short names are case insensitive except for event logging
  • [SPARK-48447] [SC-167607][SS] Check state store provider class before invoking the constructor
  • [SPARK-47977] [SC-167650] DateTimeUtils.timestampDiff and DateTimeUtils.timestampAdd should not throw INTERNAL_ERROR exception
  • [SPARK-48513] [Backport][15.x][SC-168085][SS] Add error class for state schema compatibility and minor refactoring
  • [SPARK-48413] [SC-167669][SQL] ALTER COLUMN with collation
  • [SPARK-48561] [SC-168250][PS][CONNECT] Throw PandasNotImplementedError for unsupported plotting functions
  • [SPARK-48465] [SC-167531][SQL] Avoid no-op empty relation propagation
  • [SPARK-48553] [SC-168166][PYTHON][CONNECT] Cache more properties
  • [SPARK-48540] [SC-168069][CORE] Avoid ivy output loading settings to stdout
  • [SPARK-48535] [SC-168057][SS] Update config docs to indicate possibility of data loss/corruption issue if skip nulls for stream-stream joins config is enabled
  • [SPARK-48536] [SC-168059][PYTHON][CONNECT] Cache user specified schema in applyInPandas and applyInArrow
  • [SPARK-47873] [SC-163473][SQL] Write collated strings to Hive metastore using the regular string type
  • [SPARK-48461] [SC-167442][SQL] Replace NullPointerExceptions with error class in AssertNotNull expression
  • [SPARK-47833] [SC-163191][SQL][CORE] Supply caller stackstrace for checkAndGlobPathIfNecessary AnalysisException
  • [SPARK-47898] [SC-163146][SQL] Port HIVE-12270: Add DBTokenStore support to HS2 delegation token
  • [SPARK-47578] [SC-167497][R] Migrate RPackageUtils with variables to structured logging framework
  • [SPARK-47875] [SC-162935][CORE] Remove spark.deploy.recoverySerializer
  • [SPARK-47552] [SC-160880][CORE] Set spark.hadoop.fs.s3a.connection.establish.timeout to 30s if missing
  • [SPARK-47972] [SC-167692][SQL] Restrict CAST expression for collations
  • [SPARK-48430] [SC-167489][SQL] Fix map value extraction when map contains collated strings
  • [SPARK-47318] [SC-162712][CORE][3.5] Adds HKDF round to AuthEngine key derivation to follow standard KEX practices
  • [SPARK-48503] [BEHAVE-29][ES-1135236][SQL] Fix invalid scalar subqueries with group-by on non-equivalent columns that were incorrectly allowed
  • [SPARK-48508] [SC-167695][CONNECT][PYTHON] Cache user specified schema in DataFrame.{to, mapInPandas, mapInArrow}
  • [SPARK-23015] [SC-167188][WINDOWS] Fix bug in Windows where starting multiple Spark instances within the same second causes a failure
  • [SPARK-45891] [SC-167608]Revert ” Describe shredding scheme for Variant”
  • [SPARK-48391] [SC-167554][CORE]Using addAll instead of add function in fromAccumulatorInfos method of TaskMetrics Class
  • [SPARK-48496] [SC-167600][CORE] Use static regex Pattern instances in JavaUtils.timeStringAs and JavaUtils.byteStringAs
  • [SPARK-48476] [SC-167488][SQL] fix NPE error message for null delmiter csv
  • [SPARK-48489] [SC-167598][SQL] Throw a better, user-facing error when reading illegal schema from text Data Source
  • [SPARK-48471] [SC-167324][CORE] Improve documentation and usage guide for history server
  • [SPARK-45891] [SC-167597] Describe shredding scheme for Variant
  • [SPARK-47333] [SC-159043][SQL] Use checkInputDataTypes to check the parameter types of the function to_xml
  • [SPARK-47387] [SC-159310][SQL] Remove some unused error classes
  • [SPARK-48265] [ES-1131440][SQL] Infer window group limit batch should do constant folding
  • [SPARK-47716] [SC-167444][SQL] Avoid view name conflict in SQLQueryTestSuite semantic sort test case
  • [SPARK-48159] [SC-167141][SQL] Extending support for collated strings on datetime expressions
  • [SPARK-48462] [SC-167271][SQL][Tests] Use withSQLConf in tests: Refactor HiveQuerySuite and HiveTableScanSuite
  • [SPARK-48392] [SC-167195][CORE] Also load spark-defaults.conf when provided --properties-file
  • [SPARK-48468] [SC-167417] Add LogicalQueryStage interface in catalyst
  • [SPARK-47578] [SC-164988][CORE] Manual backport for Spark PR #46309: Migrate logWarning with variables to structured logging framework
  • [SPARK-48415] [SC-167321]Revert “[PYTHON] Refactor TypeName to support parameterized datatypes”
  • [SPARK-46544] [SC-151361][SQL] Support v2 DESCRIBE TABLE EXTENDED with table stats
  • [SPARK-48325] [SC-166963][CORE] Always specify messages in ExecutorRunner.killProcess
  • [SPARK-46841] [SC-167139][SQL] Add collation support for ICU locales and collation specifiers
  • [SPARK-47221] [SC-157870][SQL] Uses signatures from CsvParser to AbstractParser
  • [SPARK-47246] [SC-158138][SQL] Replace InternalRow.fromSeq with new GenericInternalRow to save a collection conversion
  • [SPARK-47597] [SC-163932][STREAMING] Manual backport for Spark PR #46192: Streaming: Migrate logInfo with variables to structured logging framework
  • [SPARK-48415] [SC-167130][PYTHON] Refactor TypeName to support parameterized datatypes
  • [SPARK-48434] [SC-167132][PYTHON][CONNECT] Make printSchema use the cached schema
  • [SPARK-48432] [ES-1097114][SQL] Avoid unboxing integers in UnivocityParser
  • [SPARK-47463] [SC-162840][SQL] Use V2Predicate to wrap expression with return type of boolean
  • [SPARK-47781] [SC-162293][SQL] Handle negative scale decimals for JDBC data sources
  • [SPARK-48394] [SC-166966][CORE] Cleanup mapIdToMapIndex on mapoutput unregister
  • [SPARK-47072] [SC-156933][SQL] Fix supported interval formats in error messages
  • [SPARK-47001] [SC-162487][SQL] Pushdown verification in optimizer
  • [SPARK-48335] [SC-166387][PYTHON][CONNECT] Make _parse_datatype_string compatible with Spark Connect
  • [SPARK-48329] [SC-166518][SQL] Enable spark.sql.sources.v2.bucketing.pushPartValues.enabled by default
  • [SPARK-48412] [SC-166898][PYTHON] Refactor data type json parse
  • [SPARK-48215] [SC-166781][SQL] Extending support for collated strings on date_format expression
  • [SPARK-45009] [SC-166873][SQL][FOLLOW UP] Add error class and tests for decorrelation of predicate subqueries in join condition which reference both join child
  • [SPARK-47960] [SC-165295][SS][15.x] Allow chaining other stateful operators after transformWithState operator.
  • [SPARK-48340] [SC-166468][PYTHON] Support TimestampNTZ infer schema miss prefer_timestamp_ntz
  • [SPARK-48157] [SC-165902][SQL] Add collation support for CSV expressions
  • [SPARK-48158] [SC-165652][SQL] Add collation support for XML expressions
  • [SPARK-48160] [SC-166064][SQL] Add collation support for XPATH expressions
  • [SPARK-48229] [SC-165901][SQL] Add collation support for inputFile expressions
  • [SPARK-48367] [SC-166487][CONNECT] Fix lint-scala for scalafmt to detect files to format properly
  • [SPARK-47858] [SC-163095][SPARK-47852][PYTHON][SQL] Refactoring the structure for DataFrame error context
  • [SPARK-48370] [SC-166787][CONNECT] Checkpoint and localCheckpoint in Scala Spark Connect client
  • [SPARK-48247] [SC-166028][PYTHON] Use all values in a dict when inferring MapType schema
  • [SPARK-48395] [SC-166794][PYTHON] Fix StructType.treeString for parameterized types
  • [SPARK-48393] [SC-166784][PYTHON] Move a group of constants to pyspark.util
  • [SPARK-48372] [SC-166776][SPARK-45716][PYTHON] Implement StructType.treeString
  • [SPARK-48258] [SC-166467][PYTHON][CONNECT] Checkpoint and localCheckpoint in Spark Connect

See Databricks Runtime 15.4 LTS maintenance updates.

System environment

  • Operating System: Ubuntu 22.04.4 LTS
  • Java: Zulu 8.78.0.19-CA-linux64
  • Scala: 2.12.18
  • Python: 3.11.0
  • R: 4.3.2
  • Delta Lake: 3.2.0

Installed Python libraries

Library Version Library Version Library Version
asttokens 2.0.5 astunparse 1.6.3 azure-core 1.30.2
azure-storage-blob 12.19.1 azure-storage-file-datalake 12.14.0 backcall 0.2.0
black 23.3.0 blinker 1.4 boto3 1.34.39
botocore 1.34.39 cachetools 5.3.3 certifi 2023.7.22
cffi 1.15.1 chardet 4.0.0 charset-normalizer 2.0.4
click 8.0.4 cloudpickle 2.2.1 comm 0.1.2
contourpy 1.0.5 cryptography 41.0.3 cycler 0.11.0
Cython 0.29.32 databricks-sdk 0.20.0 dbus-python 1.2.18
debugpy 1.6.7 decorator 5.1.1 distlib 0.3.8
entrypoints 0.4 executing 0.8.3 facets-overview 1.1.1
filelock 3.13.4 fonttools 4.25.0 gitdb 4.0.11
GitPython 3.1.43 google-api-core 2.18.0 google-auth 2.31.0
google-cloud-core 2.4.1 google-cloud-storage 2.17.0 google-crc32c 1.5.0
google-resumable-media 2.7.1 googleapis-common-protos 1.63.2 grpcio 1.60.0
grpcio-status 1.60.0 httplib2 0.20.2 idna 3.4
importlib-metadata 6.0.0 ipyflow-core 0.0.198 ipykernel 6.25.1
ipython 8.15.0 ipython-genutils 0.2.0 ipywidgets 7.7.2
isodate 0.6.1 jedi 0.18.1 jeepney 0.7.1
jmespath 0.10.0 joblib 1.2.0 jupyter_client 7.4.9
jupyter_core 5.3.0 keyring 23.5.0 kiwisolver 1.4.4
launchpadlib 1.10.16 lazr.restfulclient 0.14.4 lazr.uri 1.0.6
matplotlib 3.7.2 matplotlib-inline 0.1.6 mlflow-skinny 2.11.4
more-itertools 8.10.0 mypy-extensions 0.4.3 nest-asyncio 1.5.6
numpy 1.23.5 oauthlib 3.2.0 packaging 23.2
pandas 1.5.3 parso 0.8.3 pathspec 0.10.3
patsy 0.5.3 pexpect 4.8.0 pickleshare 0.7.5
Pillow 9.4.0 pip 23.2.1 platformdirs 3.10.0
plotly 5.9.0 prompt-toolkit 3.0.36 proto-plus 1.24.0
protobuf 4.24.1 psutil 5.9.0 psycopg2 2.9.3
ptyprocess 0.7.0 pure-eval 0.2.2 pyarrow 14.0.1
pyasn1 0.4.8 pyasn1-modules 0.2.8 pyccolo 0.0.52
pycparser 2.21 pydantic 1.10.6 Pygments 2.15.1
PyGObject 3.42.1 PyJWT 2.3.0 pyodbc 4.0.38
pyparsing 3.0.9 python-dateutil 2.8.2 python-lsp-jsonrpc 1.1.1
pytz 2022.7 PyYAML 6.0 pyzmq 23.2.0
requests 2.31.0 rsa 4.9 s3transfer 0.10.2
scikit-learn 1.3.0 scipy 1.11.1 seaborn 0.12.2
SecretStorage 3.3.1 setuptools 68.0.0 six 1.16.0
smmap 5.0.1 sqlparse 0.5.0 ssh-import-id 5.11
stack-data 0.2.0 statsmodels 0.14.0 tenacity 8.2.2
threadpoolctl 2.2.0 tokenize-rt 4.2.1 tornado 6.3.2
traitlets 5.7.1 typing_extensions 4.10.0 tzdata 2022.1
ujson 5.4.0 unattended-upgrades 0.1 urllib3 1.26.16
virtualenv 20.24.2 wadllib 1.3.6 wcwidth 0.2.5
wheel 0.38.4 zipp 3.11.0

Installed R libraries

R libraries are installed from the Posit Package Manager CRAN snapshot.

Library Version Library Version Library Version
arrow 14.0.0.2 askpass 1.2.0 assertthat 0.2.1
backports 1.4.1 base 4.3.2 base64enc 0.1-3
bigD 0.2.0 bit 4.0.5 bit64 4.0.5
bitops 1.0-7 blob 1.2.4 boot 1.3-28
brew 1.0-10 brio 1.1.4 broom 1.0.5
bslib 0.6.1 cachem 1.0.8 callr 3.7.3
caret 6.0-94 cellranger 1.1.0 chron 2.3-61
class 7.3-22 cli 3.6.2 clipr 0.8.0
clock 0.7.0 cluster 2.1.4 codetools 0.2-19
colorspace 2.1-0 commonmark 1.9.1 compiler 4.3.2
config 0.3.2 conflicted 1.2.0 cpp11 0.4.7
crayon 1.5.2 credentials 2.0.1 curl 5.2.0
data.table 1.15.0 datasets 4.3.2 DBI 1.2.1
dbplyr 2.4.0 desc 1.4.3 devtools 2.4.5
diagram 1.6.5 diffobj 0.3.5 digest 0.6.34
downlit 0.4.3 dplyr 1.1.4 dtplyr 1.3.1
e1071 1.7-14 ellipsis 0.3.2 evaluate 0.23
fansi 1.0.6 farver 2.1.1 fastmap 1.1.1
fontawesome 0.5.2 forcats 1.0.0 foreach 1.5.2
foreign 0.8-85 forge 0.2.0 fs 1.6.3
future 1.33.1 future.apply 1.11.1 gargle 1.5.2
generics 0.1.3 gert 2.0.1 ggplot2 3.4.4
gh 1.4.0 git2r 0.33.0 gitcreds 0.1.2
glmnet 4.1-8 globals 0.16.2 glue 1.7.0
googledrive 2.1.1 googlesheets4 1.1.1 gower 1.0.1
graphics 4.3.2 grDevices 4.3.2 grid 4.3.2
gridExtra 2.3 gsubfn 0.7 gt 0.10.1
gtable 0.3.4 hardhat 1.3.1 haven 2.5.4
highr 0.10 hms 1.1.3 htmltools 0.5.7
htmlwidgets 1.6.4 httpuv 1.6.14 httr 1.4.7
httr2 1.0.0 ids 1.0.1 ini 0.3.1
ipred 0.9-14 isoband 0.2.7 iterators 1.0.14
jquerylib 0.1.4 jsonlite 1.8.8 juicyjuice 0.1.0
KernSmooth 2.23-21 knitr 1.45 labeling 0.4.3
later 1.3.2 lattice 0.21-8 lava 1.7.3
lifecycle 1.0.4 listenv 0.9.1 lubridate 1.9.3
magrittr 2.0.3 markdown 1.12 MASS 7.3-60
Matrix 1.5-4.1 memoise 2.0.1 methods 4.3.2
mgcv 1.8-42 mime 0.12 miniUI 0.1.1.1
mlflow 2.10.0 ModelMetrics 1.2.2.2 modelr 0.1.11
munsell 0.5.0 nlme 3.1-163 nnet 7.3-19
numDeriv 2016.8-1.1 openssl 2.1.1 parallel 4.3.2
parallelly 1.36.0 pillar 1.9.0 pkgbuild 1.4.3
pkgconfig 2.0.3 pkgdown 2.0.7 pkgload 1.3.4
plogr 0.2.0 plyr 1.8.9 praise 1.0.0
prettyunits 1.2.0 pROC 1.18.5 processx 3.8.3
prodlim 2023.08.28 profvis 0.3.8 progress 1.2.3
progressr 0.14.0 promises 1.2.1 proto 1.0.0
proxy 0.4-27 ps 1.7.6 purrr 1.0.2
R6 2.5.1 ragg 1.2.7 randomForest 4.7-1.1
rappdirs 0.3.3 rcmdcheck 1.4.0 RColorBrewer 1.1-3
Rcpp 1.0.12 RcppEigen 0.3.3.9.4 reactable 0.4.4
reactR 0.5.0 readr 2.1.5 readxl 1.4.3
recipes 1.0.9 rematch 2.0.0 rematch2 2.1.2
remotes 2.4.2.1 reprex 2.1.0 reshape2 1.4.4
rlang 1.1.3 rmarkdown 2.25 RODBC 1.3-23
roxygen2 7.3.1 rpart 4.1.21 rprojroot 2.0.4
Rserve 1.8-13 RSQLite 2.3.5 rstudioapi 0.15.0
rversions 2.1.2 rvest 1.0.3 sass 0.4.8
scales 1.3.0 selectr 0.4-2 sessioninfo 1.2.2
shape 1.4.6 shiny 1.8.0 sourcetools 0.1.7-1
sparklyr 1.8.4 spatial 7.3-15 splines 4.3.2
sqldf 0.4-11 SQUAREM 2021.1 stats 4.3.2
stats4 4.3.2 stringi 1.8.3 stringr 1.5.1
survival 3.5-5 swagger 3.33.1 sys 3.4.2
systemfonts 1.0.5 tcltk 4.3.2 testthat 3.2.1
textshaping 0.3.7 tibble 3.2.1 tidyr 1.3.1
tidyselect 1.2.0 tidyverse 2.0.0 timechange 0.3.0
timeDate 4032.109 tinytex 0.49 tools 4.3.2
tzdb 0.4.0 urlchecker 1.0.1 usethis 2.2.2
utf8 1.2.4 utils 4.3.2 uuid 1.2-0
V8 4.4.1 vctrs 0.6.5 viridisLite 0.4.2
vroom 1.6.5 waldo 0.5.2 whisker 0.4.1
withr 3.0.0 xfun 0.41 xml2 1.3.6
xopen 1.0.0 xtable 1.8-4 yaml 2.3.8
zeallot 0.1.0 zip 2.3.1

Installed Java and Scala libraries (Scala 2.12 cluster version)

Group ID Artifact ID Version
antlr antlr 2.7.7
com.amazonaws amazon-kinesis-client 1.12.0
com.amazonaws aws-java-sdk-autoscaling 1.12.610
com.amazonaws aws-java-sdk-cloudformation 1.12.610
com.amazonaws aws-java-sdk-cloudfront 1.12.610
com.amazonaws aws-java-sdk-cloudhsm 1.12.610
com.amazonaws aws-java-sdk-cloudsearch 1.12.610
com.amazonaws aws-java-sdk-cloudtrail 1.12.610
com.amazonaws aws-java-sdk-cloudwatch 1.12.610
com.amazonaws aws-java-sdk-cloudwatchmetrics 1.12.610
com.amazonaws aws-java-sdk-codedeploy 1.12.610
com.amazonaws aws-java-sdk-cognitoidentity 1.12.610
com.amazonaws aws-java-sdk-cognitosync 1.12.610
com.amazonaws aws-java-sdk-config 1.12.610
com.amazonaws aws-java-sdk-core 1.12.610
com.amazonaws aws-java-sdk-datapipeline 1.12.610
com.amazonaws aws-java-sdk-directconnect 1.12.610
com.amazonaws aws-java-sdk-directory 1.12.610
com.amazonaws aws-java-sdk-dynamodb 1.12.610
com.amazonaws aws-java-sdk-ec2 1.12.610
com.amazonaws aws-java-sdk-ecs 1.12.610
com.amazonaws aws-java-sdk-efs 1.12.610
com.amazonaws aws-java-sdk-elasticache 1.12.610
com.amazonaws aws-java-sdk-elasticbeanstalk 1.12.610
com.amazonaws aws-java-sdk-elasticloadbalancing 1.12.610
com.amazonaws aws-java-sdk-elastictranscoder 1.12.610
com.amazonaws aws-java-sdk-emr 1.12.610
com.amazonaws aws-java-sdk-glacier 1.12.610
com.amazonaws aws-java-sdk-glue 1.12.610
com.amazonaws aws-java-sdk-iam 1.12.610
com.amazonaws aws-java-sdk-importexport 1.12.610
com.amazonaws aws-java-sdk-kinesis 1.12.610
com.amazonaws aws-java-sdk-kms 1.12.610
com.amazonaws aws-java-sdk-lambda 1.12.610
com.amazonaws aws-java-sdk-logs 1.12.610
com.amazonaws aws-java-sdk-machinelearning 1.12.610
com.amazonaws aws-java-sdk-opsworks 1.12.610
com.amazonaws aws-java-sdk-rds 1.12.610
com.amazonaws aws-java-sdk-redshift 1.12.610
com.amazonaws aws-java-sdk-route53 1.12.610
com.amazonaws aws-java-sdk-s3 1.12.610
com.amazonaws aws-java-sdk-ses 1.12.610
com.amazonaws aws-java-sdk-simpledb 1.12.610
com.amazonaws aws-java-sdk-simpleworkflow 1.12.610
com.amazonaws aws-java-sdk-sns 1.12.610
com.amazonaws aws-java-sdk-sqs 1.12.610
com.amazonaws aws-java-sdk-ssm 1.12.610
com.amazonaws aws-java-sdk-storagegateway 1.12.610
com.amazonaws aws-java-sdk-sts 1.12.610
com.amazonaws aws-java-sdk-support 1.12.610
com.amazonaws aws-java-sdk-swf-libraries 1.11.22
com.amazonaws aws-java-sdk-workspaces 1.12.610
com.amazonaws jmespath-java 1.12.610
com.clearspring.analytics stream 2.9.6
com.databricks Rserve 1.8-3
com.databricks databricks-sdk-java 0.27.0
com.databricks jets3t 0.7.1-0
com.databricks.scalapb compilerplugin_2.12 0.4.15-10
com.databricks.scalapb scalapb-runtime_2.12 0.4.15-10
com.esotericsoftware kryo-shaded 4.0.2
com.esotericsoftware minlog 1.3.0
com.fasterxml classmate 1.3.4
com.fasterxml.jackson.core jackson-annotations 2.15.2
com.fasterxml.jackson.core jackson-core 2.15.2
com.fasterxml.jackson.core jackson-databind 2.15.2
com.fasterxml.jackson.dataformat jackson-dataformat-cbor 2.15.2
com.fasterxml.jackson.dataformat jackson-dataformat-yaml 2.15.2
com.fasterxml.jackson.datatype jackson-datatype-joda 2.15.2
com.fasterxml.jackson.datatype jackson-datatype-jsr310 2.16.0
com.fasterxml.jackson.module jackson-module-paranamer 2.15.2
com.fasterxml.jackson.module jackson-module-scala_2.12 2.15.2
com.github.ben-manes.caffeine caffeine 2.9.3
com.github.fommil jniloader 1.1
com.github.fommil.netlib native_ref-java 1.1
com.github.fommil.netlib native_ref-java 1.1-natives
com.github.fommil.netlib native_system-java 1.1
com.github.fommil.netlib native_system-java 1.1-natives
com.github.fommil.netlib netlib-native_ref-linux-x86_64 1.1-natives
com.github.fommil.netlib netlib-native_system-linux-x86_64 1.1-natives
com.github.luben zstd-jni 1.5.5-4
com.github.wendykierp JTransforms 3.1
com.google.code.findbugs jsr305 3.0.0
com.google.code.gson gson 2.10.1
com.google.crypto.tink tink 1.9.0
com.google.errorprone error_prone_annotations 2.10.0
com.google.flatbuffers flatbuffers-java 23.5.26
com.google.guava guava 15.0
com.google.protobuf protobuf-java 2.6.1
com.helger profiler 1.1.1
com.ibm.icu icu4j 75.1
com.jcraft jsch 0.1.55
com.jolbox bonecp 0.8.0.RELEASE
com.lihaoyi sourcecode_2.12 0.1.9
com.microsoft.azure azure-data-lake-store-sdk 2.3.9
com.microsoft.sqlserver mssql-jdbc 11.2.2.jre8
com.ning compress-lzf 1.1.2
com.sun.mail javax.mail 1.5.2
com.sun.xml.bind jaxb-core 2.2.11
com.sun.xml.bind jaxb-impl 2.2.11
com.tdunning json 1.8
com.thoughtworks.paranamer paranamer 2.8
com.trueaccord.lenses lenses_2.12 0.4.12
com.twitter chill-java 0.10.0
com.twitter chill_2.12 0.10.0
com.twitter util-app_2.12 7.1.0
com.twitter util-core_2.12 7.1.0
com.twitter util-function_2.12 7.1.0
com.twitter util-jvm_2.12 7.1.0
com.twitter util-lint_2.12 7.1.0
com.twitter util-registry_2.12 7.1.0
com.twitter util-stats_2.12 7.1.0
com.typesafe config 1.4.3
com.typesafe.scala-logging scala-logging_2.12 3.7.2
com.uber h3 3.7.3
com.univocity univocity-parsers 2.9.1
com.zaxxer HikariCP 4.0.3
commons-cli commons-cli 1.5.0
commons-codec commons-codec 1.16.0
commons-collections commons-collections 3.2.2
commons-dbcp commons-dbcp 1.4
commons-fileupload commons-fileupload 1.5
commons-httpclient commons-httpclient 3.1
commons-io commons-io 2.13.0
commons-lang commons-lang 2.6
commons-logging commons-logging 1.1.3
commons-pool commons-pool 1.5.4
dev.ludovic.netlib arpack 3.0.3
dev.ludovic.netlib blas 3.0.3
dev.ludovic.netlib lapack 3.0.3
info.ganglia.gmetric4j gmetric4j 1.0.10
io.airlift aircompressor 0.25
io.delta delta-sharing-client_2.12 1.1.1
io.dropwizard.metrics metrics-annotation 4.2.19
io.dropwizard.metrics metrics-core 4.2.19
io.dropwizard.metrics metrics-graphite 4.2.19
io.dropwizard.metrics metrics-healthchecks 4.2.19
io.dropwizard.metrics metrics-jetty9 4.2.19
io.dropwizard.metrics metrics-jmx 4.2.19
io.dropwizard.metrics metrics-json 4.2.19
io.dropwizard.metrics metrics-jvm 4.2.19
io.dropwizard.metrics metrics-servlets 4.2.19
io.netty netty-all 4.1.96.Final
io.netty netty-buffer 4.1.96.Final
io.netty netty-codec 4.1.96.Final
io.netty netty-codec-http 4.1.96.Final
io.netty netty-codec-http2 4.1.96.Final
io.netty netty-codec-socks 4.1.96.Final
io.netty netty-common 4.1.96.Final
io.netty netty-handler 4.1.96.Final
io.netty netty-handler-proxy 4.1.96.Final
io.netty netty-resolver 4.1.96.Final
io.netty netty-tcnative-boringssl-static 2.0.61.Final-linux-aarch_64
io.netty netty-tcnative-boringssl-static 2.0.61.Final-linux-x86_64
io.netty netty-tcnative-boringssl-static 2.0.61.Final-osx-aarch_64
io.netty netty-tcnative-boringssl-static 2.0.61.Final-osx-x86_64
io.netty netty-tcnative-boringssl-static 2.0.61.Final-windows-x86_64
io.netty netty-tcnative-classes 2.0.61.Final
io.netty netty-transport 4.1.96.Final
io.netty netty-transport-classes-epoll 4.1.96.Final
io.netty netty-transport-classes-kqueue 4.1.96.Final
io.netty netty-transport-native-epoll 4.1.96.Final
io.netty netty-transport-native-epoll 4.1.96.Final-linux-aarch_64
io.netty netty-transport-native-epoll 4.1.96.Final-linux-x86_64
io.netty netty-transport-native-kqueue 4.1.96.Final-osx-aarch_64
io.netty netty-transport-native-kqueue 4.1.96.Final-osx-x86_64
io.netty netty-transport-native-unix-common 4.1.96.Final
io.prometheus simpleclient 0.7.0
io.prometheus simpleclient_common 0.7.0
io.prometheus simpleclient_dropwizard 0.7.0
io.prometheus simpleclient_pushgateway 0.7.0
io.prometheus simpleclient_servlet 0.7.0
io.prometheus.jmx collector 0.12.0
jakarta.annotation jakarta.annotation-api 1.3.5
jakarta.servlet jakarta.servlet-api 4.0.3
jakarta.validation jakarta.validation-api 2.0.2
jakarta.ws.rs jakarta.ws.rs-api 2.1.6
javax.activation activation 1.1.1
javax.el javax.el-api 2.2.4
javax.jdo jdo-api 3.0.1
javax.transaction jta 1.1
javax.transaction transaction-api 1.1
javax.xml.bind jaxb-api 2.2.11
javolution javolution 5.5.1
jline jline 2.14.6
joda-time joda-time 2.12.1
net.java.dev.jna jna 5.8.0
net.razorvine pickle 1.3
net.sf.jpam jpam 1.1
net.sf.opencsv opencsv 2.3
net.sf.supercsv super-csv 2.2.0
net.snowflake snowflake-ingest-sdk 0.9.6
net.sourceforge.f2j arpack_combined_all 0.1
org.acplt.remotetea remotetea-oncrpc 1.1.2
org.antlr ST4 4.0.4
org.antlr antlr-runtime 3.5.2
org.antlr antlr4-runtime 4.9.3
org.antlr stringtemplate 3.2.1
org.apache.ant ant 1.10.11
org.apache.ant ant-jsch 1.10.11
org.apache.ant ant-launcher 1.10.11
org.apache.arrow arrow-format 15.0.0
org.apache.arrow arrow-memory-core 15.0.0
org.apache.arrow arrow-memory-netty 15.0.0
org.apache.arrow arrow-vector 15.0.0
org.apache.avro avro 1.11.3
org.apache.avro avro-ipc 1.11.3
org.apache.avro avro-mapred 1.11.3
org.apache.commons commons-collections4 4.4
org.apache.commons commons-compress 1.23.0
org.apache.commons commons-crypto 1.1.0
org.apache.commons commons-lang3 3.12.0
org.apache.commons commons-math3 3.6.1
org.apache.commons commons-text 1.10.0
org.apache.curator curator-client 2.13.0
org.apache.curator curator-framework 2.13.0
org.apache.curator curator-recipes 2.13.0
org.apache.datasketches datasketches-java 3.1.0
org.apache.datasketches datasketches-memory 2.0.0
org.apache.derby derby 10.14.2.0
org.apache.hadoop hadoop-client-runtime 3.3.6
org.apache.hive hive-beeline 2.3.9
org.apache.hive hive-cli 2.3.9
org.apache.hive hive-jdbc 2.3.9
org.apache.hive hive-llap-client 2.3.9
org.apache.hive hive-llap-common 2.3.9
org.apache.hive hive-serde 2.3.9
org.apache.hive hive-shims 2.3.9
org.apache.hive hive-storage-api 2.8.1
org.apache.hive.shims hive-shims-0.23 2.3.9
org.apache.hive.shims hive-shims-common 2.3.9
org.apache.hive.shims hive-shims-scheduler 2.3.9
org.apache.httpcomponents httpclient 4.5.14
org.apache.httpcomponents httpcore 4.4.16
org.apache.ivy ivy 2.5.1
org.apache.logging.log4j log4j-1.2-api 2.22.1
org.apache.logging.log4j log4j-api 2.22.1
org.apache.logging.log4j log4j-core 2.22.1
org.apache.logging.log4j log4j-layout-template-json 2.22.1
org.apache.logging.log4j log4j-slf4j2-impl 2.22.1
org.apache.orc orc-core 1.9.2-shaded-protobuf
org.apache.orc orc-mapreduce 1.9.2-shaded-protobuf
org.apache.orc orc-shims 1.9.2
org.apache.thrift libfb303 0.9.3
org.apache.thrift libthrift 0.12.0
org.apache.ws.xmlschema xmlschema-core 2.3.0
org.apache.xbean xbean-asm9-shaded 4.23
org.apache.yetus audience-annotations 0.13.0
org.apache.zookeeper zookeeper 3.6.3
org.apache.zookeeper zookeeper-jute 3.6.3
org.checkerframework checker-qual 3.31.0
org.codehaus.jackson jackson-core-asl 1.9.13
org.codehaus.jackson jackson-mapper-asl 1.9.13
org.codehaus.janino commons-compiler 3.0.16
org.codehaus.janino janino 3.0.16
org.datanucleus datanucleus-api-jdo 4.2.4
org.datanucleus datanucleus-core 4.1.17
org.datanucleus datanucleus-rdbms 4.1.19
org.datanucleus javax.jdo 3.2.0-m3
org.eclipse.collections eclipse-collections 11.1.0
org.eclipse.collections eclipse-collections-api 11.1.0
org.eclipse.jetty jetty-client 9.4.52.v20230823
org.eclipse.jetty jetty-continuation 9.4.52.v20230823
org.eclipse.jetty jetty-http 9.4.52.v20230823
org.eclipse.jetty jetty-io 9.4.52.v20230823
org.eclipse.jetty jetty-jndi 9.4.52.v20230823
org.eclipse.jetty jetty-plus 9.4.52.v20230823
org.eclipse.jetty jetty-proxy 9.4.52.v20230823
org.eclipse.jetty jetty-security 9.4.52.v20230823
org.eclipse.jetty jetty-server 9.4.52.v20230823
org.eclipse.jetty jetty-servlet 9.4.52.v20230823
org.eclipse.jetty jetty-servlets 9.4.52.v20230823
org.eclipse.jetty jetty-util 9.4.52.v20230823
org.eclipse.jetty jetty-util-ajax 9.4.52.v20230823
org.eclipse.jetty jetty-webapp 9.4.52.v20230823
org.eclipse.jetty jetty-xml 9.4.52.v20230823
org.eclipse.jetty.websocket websocket-api 9.4.52.v20230823
org.eclipse.jetty.websocket websocket-client 9.4.52.v20230823
org.eclipse.jetty.websocket websocket-common 9.4.52.v20230823
org.eclipse.jetty.websocket websocket-server 9.4.52.v20230823
org.eclipse.jetty.websocket websocket-servlet 9.4.52.v20230823
org.fusesource.leveldbjni leveldbjni-all 1.8
org.glassfish.hk2 hk2-api 2.6.1
org.glassfish.hk2 hk2-locator 2.6.1
org.glassfish.hk2 hk2-utils 2.6.1
org.glassfish.hk2 osgi-resource-locator 1.0.3
org.glassfish.hk2.external aopalliance-repackaged 2.6.1
org.glassfish.hk2.external jakarta.inject 2.6.1
org.glassfish.jersey.containers jersey-container-servlet 2.40
org.glassfish.jersey.containers jersey-container-servlet-core 2.40
org.glassfish.jersey.core jersey-client 2.40
org.glassfish.jersey.core jersey-common 2.40
org.glassfish.jersey.core jersey-server 2.40
org.glassfish.jersey.inject jersey-hk2 2.40
org.hibernate.validator hibernate-validator 6.1.7.Final
org.ini4j ini4j 0.5.4
org.javassist javassist 3.29.2-GA
org.jboss.logging jboss-logging 3.3.2.Final
org.jdbi jdbi 2.63.1
org.jetbrains annotations 17.0.0
org.joda joda-convert 1.7
org.jodd jodd-core 3.5.2
org.json4s json4s-ast_2.12 3.7.0-M11
org.json4s json4s-core_2.12 3.7.0-M11
org.json4s json4s-jackson_2.12 3.7.0-M11
org.json4s json4s-scalap_2.12 3.7.0-M11
org.lz4 lz4-java 1.8.0
org.mlflow mlflow-spark_2.12 2.9.1
org.objenesis objenesis 2.5.1
org.postgresql postgresql 42.6.1
org.roaringbitmap RoaringBitmap 0.9.45-databricks
org.roaringbitmap shims 0.9.45-databricks
org.rocksdb rocksdbjni 8.11.4
org.rosuda.REngine REngine 2.1.0
org.scala-lang scala-compiler_2.12 2.12.15
org.scala-lang scala-library_2.12 2.12.15
org.scala-lang scala-reflect_2.12 2.12.15
org.scala-lang.modules scala-collection-compat_2.12 2.11.0
org.scala-lang.modules scala-parser-combinators_2.12 1.1.2
org.scala-lang.modules scala-xml_2.12 1.2.0
org.scala-sbt test-interface 1.0
org.scalacheck scalacheck_2.12 1.14.2
org.scalactic scalactic_2.12 3.2.15
org.scalanlp breeze-macros_2.12 2.1.0
org.scalanlp breeze_2.12 2.1.0
org.scalatest scalatest-compatible 3.2.15
org.scalatest scalatest-core_2.12 3.2.15
org.scalatest scalatest-diagrams_2.12 3.2.15
org.scalatest scalatest-featurespec_2.12 3.2.15
org.scalatest scalatest-flatspec_2.12 3.2.15
org.scalatest scalatest-freespec_2.12 3.2.15
org.scalatest scalatest-funspec_2.12 3.2.15
org.scalatest scalatest-funsuite_2.12 3.2.15
org.scalatest scalatest-matchers-core_2.12 3.2.15
org.scalatest scalatest-mustmatchers_2.12 3.2.15
org.scalatest scalatest-propspec_2.12 3.2.15
org.scalatest scalatest-refspec_2.12 3.2.15
org.scalatest scalatest-shouldmatchers_2.12 3.2.15
org.scalatest scalatest-wordspec_2.12 3.2.15
org.scalatest scalatest_2.12 3.2.15
org.slf4j jcl-over-slf4j 2.0.7
org.slf4j jul-to-slf4j 2.0.7
org.slf4j slf4j-api 2.0.7
org.slf4j slf4j-simple 1.7.25
org.threeten threeten-extra 1.7.1
org.tukaani xz 1.9
org.typelevel algebra_2.12 2.0.1
org.typelevel cats-kernel_2.12 2.1.1
org.typelevel spire-macros_2.12 0.17.0
org.typelevel spire-platform_2.12 0.17.0
org.typelevel spire-util_2.12 0.17.0
org.typelevel spire_2.12 0.17.0
org.wildfly.openssl wildfly-openssl 1.1.3.Final
org.xerial sqlite-jdbc 3.42.0.0
org.xerial.snappy snappy-java 1.1.10.3
org.yaml snakeyaml 2.0
oro oro 2.0.8
pl.edu.icm JLargeArrays 1.5
software.amazon.cryptools AmazonCorrettoCryptoProvider 1.6.2-linux-x86_64
software.amazon.ion ion-java 1.0.2
stax stax-api 1.0.1