Databricks Runtime 13.1 (unsupported)

  • Article

Azure Databricks released these images in May 2023.

The following release notes provide information about Databricks Runtime 13.1, powered by Apache Spark 3.4.0.

New features and improvements

Cluster support for JDK 17 (Public Preview)

Databricks now provides cluster support for Java Development Kit (JDK) 17. See Databricks SDK for Java.

Add, change, or delete data in streaming tables

You can now use DML statements to modify streaming tables published to Unity Catalog by Delta Live Tables pipelines. See Add, change, or delete data in a streaming table and Add, change, or delete data in a target streaming table. You can also use DML statements to modify streaming tables created in Databricks SQL.

Read Kafka with SQL

You can now use the read_kafka SQL function for reading Kafka data. Streaming with SQL is supported only in Delta Live Tables or with streaming tables in Databricks SQL. See read_kafka table-valued function.

New SQL built-in functions

The following functions have been added:

Unity Catalog support for cluster-scoped Python libraries

Unity Catalog has some limitations on library usage. On Databricks Runtime 13.1 and above, cluster-scoped Python libraries are supported, including Python wheel files that are uploaded as workspace files. Libraries that are referenced using DBFS filepaths are not supported, whether in the DBFS root or an external location mounted to DBFS. Non-Python libraries are not supported. See Cluster libraries.

On Databricks Runtime 13.0 and below, cluster-scoped libraries are not supported on clusters that use shared access mode in a Unity Catalog-enabled workspace.

Expanded default enablement for optimized writes in Unity Catalog

Default optimized write support for Delta tables registered in Unity Catalog has expanded to include CTAS statements and INSERT operations for partitioned tables. This behavior aligns to defaults on SQL warehouses. See Optimized writes for Delta Lake on Azure Databricks.

Advanced support for stateful operators in Structured Streaming workloads

You can now chain multiple stateful operators together, meaning that you can feed the output of an operation such as a windowed aggregation to another stateful operation such as a join. See Optimize stateful Structured Streaming queries.

Delta clone for Unity Catalog is in Public Preview

You can now use shallow clone to create new Unity Catalog managed tables from existing Unity Catalog managed tables. See Shallow clone for Unity Catalog tables.

Pub/Sub support for Structured Streaming

You can now use a built-in connector to subscribe to Google Pub/Sub with Structured Streaming. See Subscribe to Google Pub/Sub.

Drop duplicates within watermarks in Structured Streaming

You can now use dropDuplicatesWithinWatermark in combination with a specified watermark threshold to deduplicate records in Structured Streaming. See Drop duplicates within watermark.

Expanded support for Delta conversions from Iceberg tables with truncated partition columns

You can now use CLONE and CONVERT TO DELTA with Iceberg tables that have partitions defined on truncated columns of types int, long, and string. Truncated columns of type decimal are not supported.

Stream schema changes with column mapping in Delta Lake

You now can provide a schema tracking location to enable streaming from Delta tables with column mapping enabled. See Streaming with column mapping and schema changes.

Remove START VERSION

START VERSION is now deprecated for ALTER SHARE.

New H3 expressions available with Python

The h3_coverash3 and h3_coverash3string expressions are available with Python.

Bug fixes

Parquet failOnUnknownFields no longer silently drop data on type mismatch

If a Parquet file was read with just the failOnUnknownFields option or with Auto Loader in the failOnNewColumns schema evolution mode, columns that have different data types now fail and recommend using rescuedDataColumn. Auto Loader now correctly reads and no longer rescues Integer, Short, or Byte types if one of these data types is provided. The Parquet file suggests one of the other two types.

Breaking changes

Upgrade sqlite-jdbc version to 3.42.0.0 to address CVE-2023-32697

Upgrade sqlite-jdbc version from 3.8.11.2 to 3.42.0.0. The APIs of version 3.42.0.0 are not fully compatible with 3.8.11.2. If using sqlite-jdbc in your code, check the sqlite-jdbc compatibility report for details. If you migrate to 13.1 and use sqlite, confirm your methods and return type in version 3.42.0.0.

Library upgrades

  • Upgraded Python libraries:
    • facets-overview from 1.0.2 to 1.0.3
    • filelock from 3.10.7 to 3.12.0
    • pyarrow from 7.0.0 to 8.0.0
    • tenacity from 8.0.1 to 8.1.0
  • Upgraded R libraries:
  • Upgraded Java libraries:
    • com.github.ben-manes.caffeine.caffeine from 2.3.4 to 2.9.3
    • org.apache.orc.orc-core from 1.8.2-shaded-protobuf to 1.8.3-shaded-protobuf
    • org.apache.orc.orc-mapreduce from 1.8.2-shaded-protobuf to 1.8.3-shaded-protobuf
    • org.apache.orc.orc-shims from 1.8.2 to 1.8.3
    • org.checkerframework.checker-qual from 3.5.0 to 3.19.0
    • org.scalactic.scalactic_2.12 from 3.0.8 to 3.2.15
    • org.scalatest.scalatest_2.12 from 3.0.8 to 3.2.15
    • org.xerial.sqlite-jdbc from 3.8.11.2 to 3.42.0.0

Apache Spark

Databricks Runtime 13.1 includes Apache Spark 3.4.0. This release includes all Spark fixes and improvements included in Databricks Runtime 13.0 (unsupported), as well as the following additional bug fixes and improvements made to Spark:

  • [SPARK-42719] [DBRRM-199][SC-131578] Revert “[SC-125225] `MapOutputTracker#getMap…
  • [SPARK-39696] [DBRRM-166][SC-130056][CORE] Revert [SC-127830]/
  • [SPARK-43331] [SC-130064][CONNECT] Add Spark Connect SparkSession.interruptAll
  • [SPARK-16484] [13.1][CHERRY_PICK][SC-130076][SQL] Use 8-bit registers for representing DataSketches
  • [SPARK-43332] [SC-130051][CONNECT][PYTHON] Make it possible to extend ChannelBuilder for SparkConnectClient
  • [SPARK-43323] [SC-129966][SQL][PYTHON] Fix DataFrame.toPandas with Arrow enabled to handle exceptions properly
  • [SPARK-42940] [SC-129896][SS][CONNECT] Improve session management for streaming queries
  • [SPARK-43336] [SQL] Casting between Timestamp and TimestampNTZ requires timezone
  • [SPARK-43032] [SC-125756] [CONNECT][SS] Add Streaming query manager
  • [SPARK-16484] [SC-129975][SQL] Add support for Datasketches HllSketch
  • [SPARK-43260] [SC-129281][PYTHON] Migrate the Spark SQL pandas arrow type errors into error class.
  • [SPARK-41766] [SC-129964][CORE] Handle decommission request sent before executor registration
  • [SPARK-43307] [SC-129971][PYTHON] Migrate PandasUDF value errors into error class
  • [SPARK-43206] [SC-129903] [SS] [CONNECT] StreamingQuery exception() include stack trace
  • [SPARK-43311] [SC-129905][SS] Add RocksDB state store provider memory management enhancements
  • [SPARK-43237] [SC-129898][CORE] Handle null exception message in event log
  • [SPARK-43320] [SC-129899][SQL][HIVE] Directly call Hive 2.3.9 API
  • [SPARK-43270] [SC-129897][PYTHON] Implement __dir__() in pyspark.sql.dataframe.DataFrame to include columns
  • [SPARK-43183] Revert “[SC-128938][SS] Introduce a new callback “…
  • [SPARK-43143] [SC-129902] [SS] [CONNECT] Scala StreamingQuery awaitTermination()
  • [SPARK-43257] [SC-129675][SQL] Replace the error class _LEGACY_ERROR_TEMP_2022 by an internal error
  • [SPARK-43198] [SC-129470][CONNECT] Fix “Could not initialise class ammonite…” error when using filter
  • [SPARK-43165] [SC-129777][SQL] Move canWrite to DataTypeUtils
  • [SPARK-43298] [SC-129729][PYTHON][ML] predict_batch_udf with scalar input fails with batch size of one
  • [SPARK-43298] [SC-129700]Revert “[PYTHON][ML] predict_batch_udf with scalar input fails with batch size of one”
  • [SPARK-43052] [SC-129663][CORE] Handle stacktrace with null file name in event log
  • [SPARK-43183] [SC-128938][SS] Introduce a new callback “onQueryIdle” to StreamingQueryListener
  • [SPARK-43209] [SC-129190][CONNECT][PYTHON] Migrate Expression errors into error class
  • [SPARK-42151] [SC-128754][SQL] Align UPDATE assignments with table attributes
  • [SPARK-43134] [SC-129468] [CONNECT] [SS] JVM client StreamingQuery exception() API
  • [SPARK-43298] [SC-129699][PYTHON][ML] predict_batch_udf with scalar input fails with batch size of one
  • [SPARK-43248] [SC-129660][SQL] Unnecessary serialize/deserialize of Path on parallel gather partition stats
  • [SPARK-43274] [SC-129464][SPARK-43275][PYTHON][CONNECT] Introduce PySparkNotImplementedError
  • [SPARK-43146] [SC-128804][CONNECT][PYTHON] Implement eager evaluation for repr and repr_html
  • [SPARK-42953] [SC-129469][Connect][Followup] Fix maven test build for Scala client UDF tests
  • [SPARK-43144] [SC-129280] Scala Client DataStreamReader table() API
  • [SPARK-43136] [SC-129358][CONNECT] Adding groupByKey + mapGroup + coGroup functions
  • [SPARK-43156] [SC-129672][SC-128532][SQL] Fix COUNT(*) is null bug in correlated scalar subquery
  • [SPARK-43046] [SC-129110] [SS] [Connect] Implemented Python API dropDuplicatesWithinWatermark for Spark Connect
  • [SPARK-43199] [SC-129467][SQL] Make InlineCTE idempotent
  • [SPARK-43293] [SC-129657][SQL] __qualified_access_only should be ignored in normal columns
  • [SPARK-43276] [SC-129461][CONNECT][PYTHON] Migrate Spark Connect Window errors into error class
  • [SPARK-43174] [SC-129109][SQL] Fix SparkSQLCLIDriver completer
  • [SPARK-43084] [SC-128654] [SS] Add applyInPandasWithState support for spark connect
  • [SPARK-43119] [SC-129040][SQL] Support Get SQL Keywords Dynamically Thru JDBC API and TVF
  • [SPARK-43082] [SC-129112][CONNECT][PYTHON] Arrow-optimized Python UDFs in Spark Connect
  • [SPARK-43085] [SC-128432][SQL] Support column DEFAULT assignment for multi-part table names
  • [SPARK-43156] [SC-129463][SPARK-43098][SQL] Extend scalar subquery count bug test with decorrelateInnerQuery disabled
  • [SPARK-43226] [LC-671] Define extractors for file-constant metadata
  • [SPARK-43210] [SC-129189][CONNECT][PYTHON] Introduce PySparkAssertionError
  • [SPARK-43214] [SC-129199][SQL] Post driver-side metrics for LocalTableScanExec/CommandResultExec
  • [SPARK-43285] [SC-129347] Fix ReplE2ESuite consistently failing with JDK 17
  • [SPARK-43268] [SC-129249][SQL] Use proper error classes when exceptions are constructed with a message
  • [SPARK-43142] [SC-129299] Fix DSL expressions on attributes with special characters
  • [SPARK-43129] [SC-128896] Scala core API for streaming Spark Connect
  • [SPARK-43233] [SC-129250] [SS] Add logging for Kafka Batch Reading for topic partition, offset range and task ID
  • [SPARK-43249] [SC-129195][CONNECT] Fix missing stats for SQL Command
  • [SPARK-42945] [SC-129188][CONNECT] Support PYSPARK_JVM_STACKTRACE_ENABLED in Spark Connect
  • [SPARK-43178] [SC-129197][CONNECT][PYTHON] Migrate UDF errors into PySpark error framework
  • [SPARK-43123] [SC-128494][SQL] Internal field metadata should not be leaked to catalogs
  • [SPARK-43217] [SC-129205] Correctly recurse in nested maps/arrays in findNestedField
  • [SPARK-43243] [SC-129294][PYTHON][CONNECT] Add level param to printSchema for Python
  • [SPARK-43230] [SC-129191][CONNECT] Simplify DataFrameNaFunctions.fillna
  • [SPARK-43088] [SC-128403][SQL] Respect RequiresDistributionAndOrdering in CTAS/RTAS
  • [SPARK-43234] [SC-129192][CONNECT][PYTHON] Migrate ValueError from Conect DataFrame into error class
  • [SPARK-43212] [SC-129187][SS][PYTHON] Migrate Structured Streaming errors into error class
  • [SPARK-43239] [SC-129186][PS] Remove null_counts from info()
  • [SPARK-43190] [SC-128930][SQL] ListQuery.childOutput should be consistent with child output
  • [SPARK-43191] [SC-128924][CORE] Replace reflection w/ direct calling for Hadoop CallerContext
  • [SPARK-43193] [SC-129042][SS] Remove workaround for HADOOP-12074
  • [SPARK-42657] [SC-128621][CONNECT] Support to find and transfer client-side REPL classfiles to server as artifacts
  • [SPARK-43098] [SC-77059][SQL] Fix correctness COUNT bug when scalar subquery has group by clause
  • [SPARK-43213] [SC-129062][PYTHON] Add DataFrame.offset to vanilla PySpark
  • [SPARK-42982] [SC-128400][CONNECT][PYTHON] Fix createDataFrame to respect the given schema ddl
  • [SPARK-43124] [SC-129011][SQL] Dataset.show projects CommandResults locally
  • [SPARK-42998] [SC-127422][CONNECT][PYTHON] Fix DataFrame.collect with null struct
  • [SPARK-41498] [SC-125343]Revert ” Propagate metadata through Union”
  • [SPARK-42960] [SC-129010] [CONNECT] [SS] Add await_termination() and exception() API for Streaming Query in Python
  • [SPARK-42552] [SC-128824][SQL] Correct the two-stage parsing strategy of antlr parser
  • [SPARK-43207] [SC-128937][CONNECT] Add helper functions to extract value from literal expression
  • [SPARK-43186] [SC-128841][SQL][HIVE] Remove workaround for FileSinkDesc
  • [SPARK-43107] [SC-128533][SQL] Coalesce buckets in join applied on broadcast join stream side
  • [SPARK-43195] [SC-128922][CORE] Remove unnecessary serializable wrapper in HadoopFSUtils
  • [SPARK-43137] [SC-128828][SQL] Improve ArrayInsert if the position is foldable and positive.
  • [SPARK-37829] [SC-128827][SQL] Dataframe.joinWith outer-join should return a null value for unmatched row
  • [SPARK-43042] [SC-128602] [SS] [Connect] Add table() API support for DataStreamReader
  • [SPARK-43153] [SC-128753][CONNECT] Skip Spark execution when the dataframe is local
  • [SPARK-43064] [SC-128496][SQL] Spark SQL CLI SQL tab should only show once statement once
  • [SPARK-43126] [SC-128447][SQL] Mark two Hive UDF expressions as stateful
  • [SPARK-43111] [SC-128750][PS][CONNECT][PYTHON] Merge nested if statements into single if statements
  • [SPARK-43113] [SC-128749][SQL] Evaluate stream-side variables when generating code for a bound condition
  • [SPARK-42895] [SC-127258][CONNECT] Improve error messages for stopped Spark sessions
  • [SPARK-42884] [SC-126662][CONNECT] Add Ammonite REPL integration
  • [SPARK-43168] [SC-128674][SQL] Remove get PhysicalDataType method from Datatype class
  • [SPARK-43121] [SC-128455][SQL] Use BytesWritable.copyBytes instead of manual copy in `HiveInspectors
  • [SPARK-42916] [SC-128389][SQL] JDBCTableCatalog Keeps Char/Varchar meta on the read-side
  • [SPARK-43050] [SC-128550][SQL] Fix construct aggregate expressions by replacing grouping functions
  • [SPARK-43095] [SC-128549][SQL] Avoid Once strategy’s idempotence is broken for batch: Infer Filters
  • [SPARK-43130] [SC-128597][SQL] Move InternalType to PhysicalDataType
  • [SPARK-43105] [SC-128456][CONNECT] Abbreviate Bytes and Strings in proto message
  • [SPARK-43099] [SC-128596][SQL] Use getName instead of getCanonicalName to get builder class name when registering udf to FunctionRegistry
  • [SPARK-42994] [SC-128586][ML][CONNECT] PyTorch Distributor support Local Mode
  • [SPARK-42859] Revert “[SC-127935][CONNECT][PS] Basic support for pandas API on Spark Connect”
  • [SPARK-43021] [SC-128472][SQL] CoalesceBucketsInJoin not work when using AQE
  • [SPARK-43125] [SC-128477][CONNECT] Fix Connect Server Can’t Handle Exception With Null Message
  • [SPARK-43147] [SC-128594] fix flake8 lint for local check
  • [SPARK-43031] [SC-128360] [SS] [Connect] Enable unit test and doctest for streaming
  • [SPARK-43039] [LC-67] Support custom fields in the file source _metadata column.
  • [SPARK-43120] [SC-128407][SS] Add support for tracking pinned blocks memory usage for RocksDB state store
  • [SPARK-43110] [SC-128381][SQL] Move asIntegral to PhysicalDataType
  • [SPARK-43118] [SC-128398][SS] Remove unnecessary assert for UninterruptibleThread in KafkaMicroBatchStream
  • [SPARK-43055] [SC-128331][CONNECT][PYTHON] Support duplicated nested field names
  • [SPARK-42437] [SC-128339][PYTHON][CONNECT] PySpark catalog.cacheTable will allow to specify storage level
  • [SPARK-42985] [SC-128332][CONNECT][PYTHON] Fix createDataFrame to respect the SQL configs
  • [SPARK-39696] [SC-127830][CORE] Fix data race in access to TaskMetrics.externalAccums
  • [SPARK-43103] [SC-128335][SQL] Moving Integral to PhysicalDataType
  • [SPARK-42741] [SC-125547][SQL] Do not unwrap casts in binary comparison when literal is null
  • [SPARK-43057] [SC-127948][CONNECT][PYTHON] Migrate Spark Connect Column errors into error class
  • [SPARK-42859] [SC-127935][CONNECT][PS] Basic support for pandas API on Spark Connect
  • [SPARK-43013] [SC-127773][PYTHON] Migrate ValueError from DataFrame into PySparkValueError.
  • [SPARK-43089] [SC-128051][CONNECT] Redact debug string in UI
  • [SPARK-43028] [SC-128070][SQL] Add error class SQL_CONF_NOT_FOUND
  • [SPARK-42999] [SC-127842][Connect] Dataset#foreach, foreachPartition
  • [SPARK-43066] [SC-127937][SQL] Add test for dropDuplicates in JavaDatasetSuite
  • [SPARK-43075] [SC-127939][CONNECT] Change gRPC to grpcio when it is not installed.
  • [SPARK-42953] [SC-127809][Connect] Typed filter, map, flatMap, mapPartitions
  • [SPARK-42597] [SC-125506][SQL] Support unwrap date type to timestamp type
  • [SPARK-42931] [SC-127933][SS] Introduce dropDuplicatesWithinWatermark
  • [SPARK-43073] [SC-127943][CONNECT] Add proto data types constants
  • [SPARK-43077] [SC-128050][SQL] Improve the error message of UNRECOGNIZED_SQL_TYPE
  • [SPARK-42951] [SC-128030][SS][Connect] DataStreamReader APIs
  • [SPARK-43049] [SC-127846][SQL] Use CLOB instead of VARCHAR(255) for StringType for Oracle JDBC
  • [SPARK-43018] [SC-127762][SQL] Fix bug for INSERT commands with timestamp literals
  • [SPARK-42855] [SC-127722][SQL] Use runtime null checks in TableOutputResolver
  • [SPARK-43030] [SC-127847][SQL] Deduplicate relations with metadata columns
  • [SPARK-42993] [SC-127829][ML][CONNECT] Make PyTorch Distributor compatible with Spark Connect
  • [SPARK-43058] [SC-128072][SQL] Move Numeric and Fractional to PhysicalDataType
  • [SPARK-43056] [SC-127946][SS] RocksDB state store commit should continue background work only if its paused
  • [SPARK-43059] [SC-127947][CONNECT][PYTHON] Migrate TypeError from DataFrame(Reader|Writer) into error class
  • [SPARK-43071] [SC-128018][SQL] Support SELECT DEFAULT with ORDER BY, LIMIT, OFFSET for INSERT source relation
  • [SPARK-43061] [SC-127956][CORE][SQL] Introduce PartitionEvaluator for SQL operator execution
  • [SPARK-43067] [SC-127938][SS] Correct the location of error class resource file in Kafka connector
  • [SPARK-43019] [SC-127844][SQL] Move Ordering to PhysicalDataType
  • [SPARK-43010] [SC-127759][PYTHON] Migrate Column errors into error class
  • [SPARK-42840] [SC-127782][SQL] Change _LEGACY_ERROR_TEMP_2004 error to internal error
  • [SPARK-43041] [SC-127765][SQL] Restore constructors of exceptions for compatibility in connector API
  • [SPARK-42939] [SC-127761][SS][CONNECT] Core streaming Python API for Spark Connect
  • [SPARK-42844] [SC-127766][SQL] Update the error class _LEGACY_ERROR_TEMP_2008 to INVALID_URL
  • [SPARK-42316] [SC-127720][SQL] Assign name to _LEGACY_ERROR_TEMP_2044
  • [SPARK-42995] [SC-127723][CONNECT][PYTHON] Migrate Spark Connect DataFrame errors into error class
  • [SPARK-42983] [SC-127717][CONNECT][PYTHON] Fix createDataFrame to handle 0-dim numpy array properly
  • [SPARK-42955] [SC-127476][SQL] Skip classifyException and wrap AnalysisException for SparkThrowable
  • [SPARK-42949] [SC-127255][SQL] Simplify code for NAAJ
  • [SPARK-43011] [SC-127577][SQL] array_insert should fail with 0 index
  • [SPARK-42974] [SC-127487][CORE] Restore Utils.createTempDir to use the ShutdownHookManager and clean up JavaUtils.createTempDir method.
  • [SPARK-42964] [SC-127585][SQL] PosgresDialect ‘42P07’ also means table already exists
  • [SPARK-42978] [SC-127351][SQL] Derby&PG: RENAME cannot qualify a new-table-Name with a schema-Name
  • [SPARK-37980] [SC-127668][SQL] Access row_index via _metadata if possible in tests
  • [SPARK-42655] [SC-127591][SQL] Incorrect ambiguous column reference error
  • [SPARK-43009] [SC-127596][SQL] Parameterized sql() with Any constants
  • [SPARK-43026] [SC-127590][SQL] Apply AQE with non-exchange table cache
  • [SPARK-42963] [SC-127576][SQL] Extend SparkSessionExtensions to inject rules into AQE query stage optimizer
  • [SPARK-42918] [SC-127357] Generalize handling of metadata attributes in FileSourceStrategy
  • [SPARK-42806] [SC-127452][SPARK-42811][CONNECT] Add Catalog support
  • [SPARK-42997] [SC-127535][SQL] TableOutputResolver must use correct column paths in error messages for arrays and maps
  • [SPARK-43006] [SC-127486][PYSPARK] Fix typo in StorageLevel eq()
  • [SPARK-43005] [SC-127485][PYSPARK] Fix typo in pyspark/pandas/config.py
  • [SPARK-43004] [SC-127457][CORE] Fix typo in ResourceRequest.equals()
  • [SPARK-42907] [SC-126984][CONNECT][PYTHON] Implement Avro functions
  • [SPARK-42979] [SC-127272][SQL] Define literal constructors as keywords
  • [SPARK-42946] [SC-127252][SQL] Redact sensitive data which is nested by variable substitution
  • [SPARK-42952] [SC-127260][SQL] Simplify the parameter of analyzer rule PreprocessTableCreation and DataSourceAnalysis
  • [SPARK-42683] [LC-75] Automatically rename conflicting metadata columns
  • [SPARK-42853] [SC-126101][FollowUp] Fix conflicts
  • [SPARK-42929] [SC-126748][CONNECT] make mapInPandas / mapInArrow support “is_barrier”
  • [SPARK-42968] [SC-127271][SS] Add option to skip commit coordinator as part of StreamingWrite API for DSv2 sources/sinks
  • [SPARK-42954] [SC-127261][PYTHON][CONNECT] Add YearMonthIntervalType to PySpark and Spark Connect Python Client
  • [SPARK-41359] [SC-127256][SQL] Use PhysicalDataType instead of DataType in UnsafeRow
  • [SPARK-42873] [SC-127262][SQL] Define Spark SQL types as keywords
  • [SPARK-42808] [SC-126302][CORE] Avoid getting availableProcessors every time in MapOutputTrackerMaster#getStatistics
  • [SPARK-42937] [SC-126880][SQL] PlanSubqueries should set InSubqueryExec#shouldBroadcast to true
  • [SPARK-42896] [SC-126729][SQL][PYTHON] Make mapInPandas / mapInArrow support barrier mode execution
  • [SPARK-42874] [SC-126442][SQL] Enable new golden file test framework for analysis for all input files
  • [SPARK-42922] [SC-126850][SQL] Move from Random to SecureRandom
  • [SPARK-42753] [SC-126369] ReusedExchange refers to non-existent nodes
  • [SPARK-40822] [SC-126274][SQL] Stable derived column aliases
  • [SPARK-42908] [SC-126856][PYTHON] Raise RuntimeError when SparkContext is required but not initialized
  • [SPARK-42779] [SC-126042][SQL] Allow V2 writes to indicate advisory shuffle partition size
  • [SPARK-42914] [SC-126727][PYTHON] Reuse transformUnregisteredFunction for DistributedSequenceID.
  • [SPARK-42878] [SC-126882][CONNECT] The table API in DataFrameReader could also accept options
  • [SPARK-42927] [SC-126883][CORE] Change the access scope of o.a.spark.util.Iterators#size to private[util]
  • [SPARK-42943] [SC-126879][SQL] Use LONGTEXT instead of TEXT for StringType for effective length
  • [SPARK-37677] [SC-126855][CORE] Unzip could keep file permissions
  • [SPARK-42891] [13.x][SC-126458][CONNECT][PYTHON] Implement CoGrouped Map API
  • [SPARK-41876] [SC-126849][CONNECT][PYTHON] Implement DataFrame.toLocalIterator
  • [SPARK-42930] [SC-126761][CORE][SQL] Change the access scope of ProtobufSerDe related implementations to private[protobuf]
  • [SPARK-42819] [SC-125879][SS] Add support for setting max_write_buffer_number and write_buffer_size for RocksDB used in streaming
  • [SPARK-42924] [SC-126737][SQL][CONNECT][PYTHON] Clarify the comment of parameterized SQL args
  • [SPARK-42748] [SC-126455][CONNECT] Server-side Artifact Management
  • [SPARK-42816] [SC-126365][CONNECT] Support Max Message size up to 128MB
  • [SPARK-42850] [SC-126109][SQL] Remove duplicated rule CombineFilters in Optimizer
  • [SPARK-42662] [SC-126355][CONNECT][PS] Add proto message for pandas API on Spark default index
  • [SPARK-42720] [SC-126136][PS][SQL] Uses expression for distributed-sequence default index instead of plan
  • [SPARK-42790] [SC-126174][SQL] Abstract the excluded method for better test for JDBC docker tests.
  • [SPARK-42900] [SC-126473][CONNECT][PYTHON] Fix createDataFrame to respect inference and column names
  • [SPARK-42917] [SC-126657][SQL] Correct getUpdateColumnNullabilityQuery for DerbyDialect
  • [SPARK-42684] [SC-125157][SQL] v2 catalog should not allow column default value by default
  • [SPARK-42861] [SC-126635][SQL] Use private[sql] instead of protected[sql] to avoid generating API doc
  • [SPARK-42920] [SC-126728][CONNECT][PYTHON] Enable tests for UDF with UDT
  • [SPARK-42791] [SC-126617][SQL] Create a new golden file test framework for analysis
  • [SPARK-42911] [SC-126652][PYTHON] Introduce more basic exceptions
  • [SPARK-42904] [SC-126634][SQL] Char/Varchar Support for JDBC Catalog
  • [SPARK-42901] [SC-126459][CONNECT][PYTHON] Move StorageLevel into a separate file to avoid potential file recursively imports
  • [SPARK-42894] [SC-126451][CONNECT] Support cache/persist/unpersist/storageLevel for Spark connect jvm client
  • [SPARK-42792] [SC-125852][SS] Add support for WRITE_FLUSH_BYTES for RocksDB used in streaming stateful operators
  • [SPARK-41233] [SC-126441][CONNECT][PYTHON] Add array_prepend to Spark Connect Python client
  • [SPARK-42681] [SC-125149][SQL] Relax ordering constraint for ALTER TABLE ADD|REPLACE column descriptor
  • [SPARK-42889] [SC-126367][CONNECT][PYTHON] Implement cache, persist, unpersist, and storageLevel
  • [SPARK-42824] [SC-125985][CONNECT][PYTHON] Provide a clear error message for unsupported JVM attributes
  • [SPARK-42340] [SC-126131][CONNECT][PYTHON] Implement Grouped Map API
  • [SPARK-42892] [SC-126454][SQL] Move sameType and relevant methods out of DataType
  • [SPARK-42827] [SC-126126][CONNECT] Support functions#array_prepend for Scala connect client
  • [SPARK-42823] [SC-125987][SQL] spark-sql shell supports multipart namespaces for initialization
  • [SPARK-42817] [SC-125960][CORE] Logging the shuffle service name once in ApplicationMaster
  • [SPARK-42786] [SC-126438][Connect] Typed Select
  • [SPARK-42800] [SC-125868][CONNECT][PYTHON][ML] Implement ml function {array_to_vector, vector_to_array}
  • [SPARK-42052] [SC-126439][SQL] Codegen Support for HiveSimpleUDF
  • [SPARK-41233] [SC-126110][SQL][PYTHON] Add array_prepend function
  • [SPARK-42864] [SC-126268][ML][3.4] Make IsotonicRegression.PointsAccumulator private
  • [SPARK-42876] [SC-126281][SQL] DataType’s physicalDataType should be private[sql]
  • [SPARK-42101] [SC-125437][SQL] Make AQE support InMemoryTableScanExec
  • [SPARK-41290] [SC-124030][SQL] Support GENERATED ALWAYS AS expressions for columns in create/replace table statements
  • [SPARK-42870] [SC-126220][CONNECT] Move toCatalystValue to connect-common
  • [SPARK-42247] [SC-126107][CONNECT][PYTHON] Fix UserDefinedFunction to have returnType
  • [SPARK-42875] [SC-126258][CONNECT][PYTHON] Fix toPandas to handle timezone and map types properly
  • [SPARK-42757] [SC-125626][CONNECT] Implement textFile for DataFrameReader
  • [SPARK-42803] [SC-126081][CORE][SQL][ML] Use getParameterCount function instead of getParameterTypes.length
  • [SPARK-42833] [SC-126043][SQL] Refactor applyExtensions in SparkSession
  • [SPARK-41765] Revert “[SC-123550][SQL] Pull out v1 write metrics…
  • [SPARK-42848] [SC-126105][CONNECT][PYTHON] Implement DataFrame.registerTempTable
  • [SPARK-42020] [SC-126103][CONNECT][PYTHON] Support UserDefinedType in Spark Connect
  • [SPARK-42818] [SC-125861][CONNECT][PYTHON] Implement DataFrameReader/Writer.jdbc
  • [SPARK-42812] [SC-125867][CONNECT] Add client_type to AddArtifactsRequest protobuf message
  • [SPARK-42772] [SC-125860][SQL] Change the default value of JDBC options about push down to true
  • [SPARK-42771] [SC-125855][SQL] Refactor HiveGenericUDF
  • [SPARK-25050] [SC-123839][SQL] Avro: writing complex unions
  • [SPARK-42765] [SC-125850][CONNECT][PYTHON] Enable importing pandas_udf from pyspark.sql.connect.functions
  • [SPARK-42719] [SC-125225][CORE] MapOutputTracker#getMapLocation should respect spark.shuffle.reduceLocality.enabled
  • [SPARK-42480] [SC-125173][SQL] Improve the performance of drop partitions
  • [SPARK-42689] [SC-125195][CORE][SHUFFLE] Allow ShuffleDriverComponent to declare if shuffle data is reliably stored
  • [SPARK-42726] [SC-125279][CONNECT][PYTHON] Implement DataFrame.mapInArrow
  • [SPARK-41765] [SC-123550][SQL] Pull out v1 write metrics to WriteFiles
  • [SPARK-41171] [SC-124191][SQL] Infer and push down window limit through window if partitionSpec is empty
  • [SPARK-42686] [SC-125292][CORE] Defer formatting for debug messages in TaskMemoryManager
  • [SPARK-42756] [SC-125443][CONNECT][PYTHON] Helper function to convert proto literal to value in Python Client
  • [SPARK-42793] [SC-125627][CONNECT] connect module requires build_profile_flags
  • [SPARK-42701] [SC-125192][SQL] Add the try_aes_decrypt() function
  • [SPARK-42679] [SC-125438][CONNECT][PYTHON] createDataFrame doesn’t work with non-nullable schema
  • [SPARK-42733] [SC-125542][CONNECT][Followup] Write without path or table
  • [SPARK-42777] [SC-125525][SQL] Support converting TimestampNTZ catalog stats to plan stats
  • [SPARK-42770] [SC-125558][CONNECT] Add truncatedTo(ChronoUnit.MICROS) to make SQLImplicitsTestSuite in Java 17 daily test GA task pass
  • [SPARK-42752] [SC-125550][PYSPARK][SQL] Make PySpark exceptions printable during initialization
  • [SPARK-42732] [SC-125544][PYSPARK][CONNECT] Support spark connect session getActiveSession method
  • [SPARK-42755] [SC-125442][CONNECT] Factor literal value conversion out to connect-common
  • [SPARK-42747] [SC-125399][ML] Fix incorrect internal status of LoR and AFT
  • [SPARK-42740] [SC-125439][SQL] Fix the bug that pushdown offset or paging is invalid for some built-in dialect
  • [SPARK-42745] [SC-125332][SQL] Improved AliasAwareOutputExpression works with DSv2
  • [SPARK-42743] [SC-125330][SQL] Support analyze TimestampNTZ columns
  • [SPARK-42721] [SC-125371][CONNECT] RPC logging interceptor
  • [SPARK-42691] [SC-125397][CONNECT][PYTHON] Implement Dataset.semanticHash
  • [SPARK-42688] [SC-124922][CONNECT] Rename Connect proto Request client_id to session_id
  • [SPARK-42310] [SC-122792][SQL] Assign name to _LEGACY_ERROR_TEMP_1289
  • [SPARK-42685] [SC-125339][CORE] Optimize Utils.bytesToString routines
  • [SPARK-42725] [SC-125296][CONNECT][PYTHON] Make LiteralExpression support array params
  • [SPARK-42702] [SC-125293][SPARK-42623][SQL] Support parameterized query in subquery and CTE
  • [SPARK-42697] [SC-125189][WEBUI] Fix /api/v1/applications to return total uptime instead of 0 for the duration field
  • [SPARK-42733] [SC-125278][CONNECT][PYTHON] Fix DataFrameWriter.save to work without path parameter
  • [SPARK-42376] [SC-124928][SS] Introduce watermark propagation among operators
  • [SPARK-42710] [SC-125205][CONNECT][PYTHON] Rename FrameMap proto to MapPartitions
  • [SPARK-37099] [SC-123542][SQL] Introduce the group limit of Window for rank-based filter to optimize top-k computation
  • [SPARK-42630] [SC-125207][CONNECT][PYTHON] Introduce UnparsedDataType and delay parsing DDL string until SparkConnectClient is available
  • [SPARK-42690] [SC-125193][CONNECT] Implement CSV/JSON parsing functions for Scala client
  • [SPARK-42709] [SC-125172][PYTHON] Remove the assumption of __file__ being available
  • [SPARK-42318] [SC-122648][SPARK-42319][SQL] Assign name to LEGACY_ERROR_TEMP(2123|2125)
  • [SPARK-42723] [SC-125183][SQL] Support parser data type json “timestamp_ltz” as TimestampType
  • [SPARK-42722] [SC-125175][CONNECT][PYTHON] Python Connect def schema() should not cache the schema
  • [SPARK-42643] [SC-125152][CONNECT][PYTHON] Register Java (aggregate) user-defined functions
  • [SPARK-42656] [SC-125177][CONNECT][Followup] Fix the spark-connect script
  • [SPARK-41516] [SC-123899] [SQL] Allow jdbc dialects to override the query used to create a table
  • [SPARK-41725] [SC-124396][CONNECT] Eager Execution of DF.sql()
  • [SPARK-42687] [SC-124896][SS] Better error message for the unsupport pivot operation in Streaming
  • [SPARK-42676] [SC-124809][SS] Write temp checkpoints for streaming queries to local filesystem even if default FS is set differently
  • [SPARK-42303] [SC-122644][SQL] Assign name to _LEGACY_ERROR_TEMP_1326
  • [SPARK-42553] [SC-124560][SQL] Ensure at least one time unit after “interval”
  • [SPARK-42649] [SC-124576][CORE] Remove the standard Apache License header from the top of third-party source files
  • [SPARK-42611] [SC-124395][SQL] Insert char/varchar length checks for inner fields during resolution
  • [SPARK-42419] [SC-124019][CONNECT][PYTHON] Migrate into error framework for Spark Connect Column API.
  • [SPARK-42637] [SC-124522][CONNECT] Add SparkSession.stop()
  • [SPARK-42647] [SC-124647][PYTHON] Change alias for numpy deprecated and removed types
  • [SPARK-42616] [SC-124389][SQL] SparkSQLCLIDriver shall only close started hive sessionState
  • [SPARK-42593] [SC-124405][PS] Deprecate & remove the APIs that will be removed in pandas 2.0.
  • [SPARK-41870] [SC-124402][CONNECT][PYTHON] Fix createDataFrame to handle duplicated column names
  • [SPARK-42569] [SC-124379][CONNECT] Throw exceptions for unsupported session API
  • [SPARK-42631] [SC-124526][CONNECT] Support custom extensions in Scala client
  • [SPARK-41868] [SC-124387][CONNECT][PYTHON] Fix createDataFrame to support durations
  • [SPARK-42572] [SC-124171][SQL][SS] Fix behavior for StateStoreProvider.validateStateRowFormat

Maintenance updates

See Databricks Runtime 13.1 maintenance updates.

System environment

  • Operating System: Ubuntu 22.04.2 LTS
  • Java: Zulu 8.70.0.23-CA-linux64
  • Scala: 2.12.15
  • Python: 3.10.12
  • R: 4.2.2
  • Delta Lake: 2.4.0

Installed Python libraries

Library Version Library Version Library Version
appdirs 1.4.4 argon2-cffi 21.3.0 argon2-cffi-bindings 21.2.0
asttokens 2.2.1 attrs 21.4.0 backcall 0.2.0
beautifulsoup4 4.11.1 black 22.6.0 bleach 4.1.0
blinker 1.4 boto3 1.24.28 botocore 1.27.28
certifi 2022.9.14 cffi 1.15.1 chardet 4.0.0
charset-normalizer 2.0.4 click 8.0.4 cryptography 37.0.1
cycler 0.11.0 Cython 0.29.32 dbus-python 1.2.18
debugpy 1.5.1 decorator 5.1.1 defusedxml 0.7.1
distlib 0.3.6 docstring-to-markdown 0.12 entrypoints 0.4
executing 1.2.0 facets-overview 1.0.3 fastjsonschema 2.16.3
filelock 3.12.0 fonttools 4.25.0 googleapis-common-protos 1.56.4
grpcio 1.48.1 grpcio-status 1.48.1 httplib2 0.20.2
idna 3.3 importlib-metadata 4.6.4 ipykernel 6.17.1
ipython 8.10.0 ipython-genutils 0.2.0 ipywidgets 7.7.2
jedi 0.18.1 jeepney 0.7.1 Jinja2 2.11.3
jmespath 0.10.0 joblib 1.2.0 jsonschema 4.16.0
jupyter-client 7.3.4 jupyter_core 4.11.2 jupyterlab-pygments 0.1.2
jupyterlab-widgets 1.0.0 keyring 23.5.0 kiwisolver 1.4.2
launchpadlib 1.10.16 lazr.restfulclient 0.14.4 lazr.uri 1.0.6
MarkupSafe 2.0.1 matplotlib 3.5.2 matplotlib-inline 0.1.6
mccabe 0.7.0 mistune 0.8.4 more-itertools 8.10.0
mypy-extensions 0.4.3 nbclient 0.5.13 nbconvert 6.4.4
nbformat 5.5.0 nest-asyncio 1.5.5 nodeenv 1.7.0
notebook 6.4.12 numpy 1.21.5 oauthlib 3.2.0
packaging 21.3 pandas 1.4.4 pandocfilters 1.5.0
parso 0.8.3 pathspec 0.9.0 patsy 0.5.2
pexpect 4.8.0 pickleshare 0.7.5 Pillow 9.2.0
pip 22.2.2 platformdirs 2.5.2 plotly 5.9.0
pluggy 1.0.0 prometheus-client 0.14.1 prompt-toolkit 3.0.36
protobuf 3.19.4 psutil 5.9.0 psycopg2 2.9.3
ptyprocess 0.7.0 pure-eval 0.2.2 pyarrow 8.0.0
pycparser 2.21 pydantic 1.10.6 pyflakes 3.0.1
Pygments 2.11.2 PyGObject 3.42.1 PyJWT 2.3.0
pyodbc 4.0.32 pyparsing 3.0.9 pyright 1.1.294
pyrsistent 0.18.0 python-dateutil 2.8.2 python-lsp-jsonrpc 1.0.0
python-lsp-server 1.7.1 pytoolconfig 1.2.2 pytz 2022.1
pyzmq 23.2.0 requests 2.28.1 rope 1.7.0
s3transfer 0.6.0 scikit-learn 1.1.1 scipy 1.9.1
seaborn 0.11.2 SecretStorage 3.3.1 Send2Trash 1.8.0
setuptools 63.4.1 six 1.16.0 soupsieve 2.3.1
ssh-import-id 5.11 stack-data 0.6.2 statsmodels 0.13.2
tenacity 8.1.0 terminado 0.13.1 testpath 0.6.0
threadpoolctl 2.2.0 tokenize-rt 4.2.1 tomli 2.0.1
tornado 6.1 traitlets 5.1.1 typing_extensions 4.3.0
ujson 5.4.0 unattended-upgrades 0.1 urllib3 1.26.11
virtualenv 20.16.3 wadllib 1.3.6 wcwidth 0.2.5
webencodings 0.5.1 whatthepatch 1.0.2 wheel 0.37.1
widgetsnbextension 3.6.1 yapf 0.31.0 zipp 1.0.0

Installed R libraries

R libraries are installed from the Microsoft CRAN snapshot on 2023-02-10.

Library Version Library Version Library Version
arrow 10.0.1 askpass 1.1 assertthat 0.2.1
backports 1.4.1 base 4.2.2 base64enc 0.1-3
bit 4.0.5 bit64 4.0.5 blob 1.2.3
boot 1.3-28 brew 1.0-8 brio 1.1.3
broom 1.0.3 bslib 0.4.2 cachem 1.0.6
callr 3.7.3 caret 6.0-93 cellranger 1.1.0
chron 2.3-59 class 7.3-21 cli 3.6.0
clipr 0.8.0 clock 0.6.1 cluster 2.1.4
codetools 0.2-19 colorspace 2.1-0 commonmark 1.8.1
compiler 4.2.2 config 0.3.1 cpp11 0.4.3
crayon 1.5.2 credentials 1.3.2 curl 5.0.0
data.table 1.14.6 datasets 4.2.2 DBI 1.1.3
dbplyr 2.3.0 desc 1.4.2 devtools 2.4.5
diffobj 0.3.5 digest 0.6.31 downlit 0.4.2
dplyr 1.1.0 dtplyr 1.2.2 e1071 1.7-13
ellipsis 0.3.2 evaluate 0.20 fansi 1.0.4
farver 2.1.1 fastmap 1.1.0 fontawesome 0.5.0
forcats 1.0.0 foreach 1.5.2 foreign 0.8-82
forge 0.2.0 fs 1.6.1 future 1.31.0
future.apply 1.10.0 gargle 1.3.0 generics 0.1.3
gert 1.9.2 ggplot2 3.4.0 gh 1.3.1
gitcreds 0.1.2 glmnet 4.1-6 globals 0.16.2
glue 1.6.2 googledrive 2.0.0 googlesheets4 1.0.1
gower 1.0.1 graphics 4.2.2 grDevices 4.2.2
grid 4.2.2 gridExtra 2.3 gsubfn 0.7
gtable 0.3.1 hardhat 1.2.0 haven 2.5.1
highr 0.10 hms 1.1.2 htmltools 0.5.4
htmlwidgets 1.6.1 httpuv 1.6.8 httr 1.4.4
ids 1.0.1 ini 0.3.1 ipred 0.9-13
isoband 0.2.7 iterators 1.0.14 jquerylib 0.1.4
jsonlite 1.8.4 KernSmooth 2.23-20 knitr 1.42
labeling 0.4.2 later 1.3.0 lattice 0.20-45
lava 1.7.1 lifecycle 1.0.3 listenv 0.9.0
lubridate 1.9.1 magrittr 2.0.3 markdown 1.5
MASS 7.3-58.2 Matrix 1.5-1 memoise 2.0.1
methods 4.2.2 mgcv 1.8-41 mime 0.12
miniUI 0.1.1.1 ModelMetrics 1.2.2.2 modelr 0.1.10
munsell 0.5.0 nlme 3.1-162 nnet 7.3-18
numDeriv 2016.8-1.1 openssl 2.0.5 parallel 4.2.2
parallelly 1.34.0 pillar 1.8.1 pkgbuild 1.4.0
pkgconfig 2.0.3 pkgdown 2.0.7 pkgload 1.3.2
plogr 0.2.0 plyr 1.8.8 praise 1.0.0
prettyunits 1.1.1 pROC 1.18.0 processx 3.8.0
prodlim 2019.11.13 profvis 0.3.7 progress 1.2.2
progressr 0.13.0 promises 1.2.0.1 proto 1.0.0
proxy 0.4-27 ps 1.7.2 purrr 1.0.1
r2d3 0.2.6 R6 2.5.1 ragg 1.2.5
randomForest 4.7-1.1 rappdirs 0.3.3 rcmdcheck 1.4.0
RColorBrewer 1.1-3 Rcpp 1.0.10 RcppEigen 0.3.3.9.3
readr 2.1.3 readxl 1.4.2 recipes 1.0.4
rematch 1.0.1 rematch2 2.1.2 remotes 2.4.2
reprex 2.0.2 reshape2 1.4.4 rlang 1.0.6
rmarkdown 2.20 RODBC 1.3-20 roxygen2 7.2.3
rpart 4.1.19 rprojroot 2.0.3 Rserve 1.8-12
RSQLite 2.2.20 rstudioapi 0.14 rversions 2.1.2
rvest 1.0.3 sass 0.4.5 scales 1.2.1
selectr 0.4-2 sessioninfo 1.2.2 shape 1.4.6
shiny 1.7.4 sourcetools 0.1.7-1 sparklyr 1.7.9
SparkR 3.4.0 spatial 7.3-15 splines 4.2.2
sqldf 0.4-11 SQUAREM 2021.1 stats 4.2.2
stats4 4.2.2 stringi 1.7.12 stringr 1.5.0
survival 3.5-3 sys 3.4.1 systemfonts 1.0.4
tcltk 4.2.2 testthat 3.1.6 textshaping 0.3.6
tibble 3.1.8 tidyr 1.3.0 tidyselect 1.2.0
tidyverse 1.3.2 timechange 0.2.0 timeDate 4022.108
tinytex 0.44 tools 4.2.2 tzdb 0.3.0
urlchecker 1.0.1 usethis 2.1.6 utf8 1.2.3
utils 4.2.2 uuid 1.1-0 vctrs 0.5.2
viridisLite 0.4.1 vroom 1.6.1 waldo 0.4.0
whisker 0.4.1 withr 2.5.0 xfun 0.37
xml2 1.3.3 xopen 1.0.0 xtable 1.8-4
yaml 2.3.7 zip 2.2.2

Installed Java and Scala libraries (Scala 2.12 cluster version)

Group ID Artifact ID Version
antlr antlr 2.7.7
com.amazonaws amazon-kinesis-client 1.12.0
com.amazonaws aws-java-sdk-autoscaling 1.12.390
com.amazonaws aws-java-sdk-cloudformation 1.12.390
com.amazonaws aws-java-sdk-cloudfront 1.12.390
com.amazonaws aws-java-sdk-cloudhsm 1.12.390
com.amazonaws aws-java-sdk-cloudsearch 1.12.390
com.amazonaws aws-java-sdk-cloudtrail 1.12.390
com.amazonaws aws-java-sdk-cloudwatch 1.12.390
com.amazonaws aws-java-sdk-cloudwatchmetrics 1.12.390
com.amazonaws aws-java-sdk-codedeploy 1.12.390
com.amazonaws aws-java-sdk-cognitoidentity 1.12.390
com.amazonaws aws-java-sdk-cognitosync 1.12.390
com.amazonaws aws-java-sdk-config 1.12.390
com.amazonaws aws-java-sdk-core 1.12.390
com.amazonaws aws-java-sdk-datapipeline 1.12.390
com.amazonaws aws-java-sdk-directconnect 1.12.390
com.amazonaws aws-java-sdk-directory 1.12.390
com.amazonaws aws-java-sdk-dynamodb 1.12.390
com.amazonaws aws-java-sdk-ec2 1.12.390
com.amazonaws aws-java-sdk-ecs 1.12.390
com.amazonaws aws-java-sdk-efs 1.12.390
com.amazonaws aws-java-sdk-elasticache 1.12.390
com.amazonaws aws-java-sdk-elasticbeanstalk 1.12.390
com.amazonaws aws-java-sdk-elasticloadbalancing 1.12.390
com.amazonaws aws-java-sdk-elastictranscoder 1.12.390
com.amazonaws aws-java-sdk-emr 1.12.390
com.amazonaws aws-java-sdk-glacier 1.12.390
com.amazonaws aws-java-sdk-glue 1.12.390
com.amazonaws aws-java-sdk-iam 1.12.390
com.amazonaws aws-java-sdk-importexport 1.12.390
com.amazonaws aws-java-sdk-kinesis 1.12.390
com.amazonaws aws-java-sdk-kms 1.12.390
com.amazonaws aws-java-sdk-lambda 1.12.390
com.amazonaws aws-java-sdk-logs 1.12.390
com.amazonaws aws-java-sdk-machinelearning 1.12.390
com.amazonaws aws-java-sdk-opsworks 1.12.390
com.amazonaws aws-java-sdk-rds 1.12.390
com.amazonaws aws-java-sdk-redshift 1.12.390
com.amazonaws aws-java-sdk-route53 1.12.390
com.amazonaws aws-java-sdk-s3 1.12.390
com.amazonaws aws-java-sdk-ses 1.12.390
com.amazonaws aws-java-sdk-simpledb 1.12.390
com.amazonaws aws-java-sdk-simpleworkflow 1.12.390
com.amazonaws aws-java-sdk-sns 1.12.390
com.amazonaws aws-java-sdk-sqs 1.12.390
com.amazonaws aws-java-sdk-ssm 1.12.390
com.amazonaws aws-java-sdk-storagegateway 1.12.390
com.amazonaws aws-java-sdk-sts 1.12.390
com.amazonaws aws-java-sdk-support 1.12.390
com.amazonaws aws-java-sdk-swf-libraries 1.11.22
com.amazonaws aws-java-sdk-workspaces 1.12.390
com.amazonaws jmespath-java 1.12.390
com.clearspring.analytics stream 2.9.6
com.databricks Rserve 1.8-3
com.databricks jets3t 0.7.1-0
com.databricks.scalapb compilerplugin_2.12 0.4.15-10
com.databricks.scalapb scalapb-runtime_2.12 0.4.15-10
com.esotericsoftware kryo-shaded 4.0.2
com.esotericsoftware minlog 1.3.0
com.fasterxml classmate 1.3.4
com.fasterxml.jackson.core jackson-annotations 2.14.2
com.fasterxml.jackson.core jackson-core 2.14.2
com.fasterxml.jackson.core jackson-databind 2.14.2
com.fasterxml.jackson.dataformat jackson-dataformat-cbor 2.14.2
com.fasterxml.jackson.datatype jackson-datatype-joda 2.14.2
com.fasterxml.jackson.datatype jackson-datatype-jsr310 2.13.4
com.fasterxml.jackson.module jackson-module-paranamer 2.14.2
com.fasterxml.jackson.module jackson-module-scala_2.12 2.14.2
com.github.ben-manes.caffeine caffeine 2.9.3
com.github.fommil jniloader 1.1
com.github.fommil.netlib native_ref-java 1.1
com.github.fommil.netlib native_ref-java 1.1-natives
com.github.fommil.netlib native_system-java 1.1
com.github.fommil.netlib native_system-java 1.1-natives
com.github.fommil.netlib netlib-native_ref-linux-x86_64 1.1-natives
com.github.fommil.netlib netlib-native_system-linux-x86_64 1.1-natives
com.github.luben zstd-jni 1.5.2-5
com.github.wendykierp JTransforms 3.1
com.google.code.findbugs jsr305 3.0.0
com.google.code.gson gson 2.8.9
com.google.crypto.tink tink 1.7.0
com.google.errorprone error_prone_annotations 2.10.0
com.google.flatbuffers flatbuffers-java 1.12.0
com.google.guava guava 15.0
com.google.protobuf protobuf-java 2.6.1
com.h2database h2 2.1.214
com.helger profiler 1.1.1
com.jcraft jsch 0.1.55
com.jolbox bonecp 0.8.0.RELEASE
com.lihaoyi sourcecode_2.12 0.1.9
com.microsoft.azure azure-data-lake-store-sdk 2.3.9
com.microsoft.sqlserver mssql-jdbc 11.2.2.jre8
com.ning compress-lzf 1.1.2
com.sun.mail javax.mail 1.5.2
com.sun.xml.bind jaxb-core 2.2.11
com.sun.xml.bind jaxb-impl 2.2.11
com.tdunning json 1.8
com.thoughtworks.paranamer paranamer 2.8
com.trueaccord.lenses lenses_2.12 0.4.12
com.twitter chill-java 0.10.0
com.twitter chill_2.12 0.10.0
com.twitter util-app_2.12 7.1.0
com.twitter util-core_2.12 7.1.0
com.twitter util-function_2.12 7.1.0
com.twitter util-jvm_2.12 7.1.0
com.twitter util-lint_2.12 7.1.0
com.twitter util-registry_2.12 7.1.0
com.twitter util-stats_2.12 7.1.0
com.typesafe config 1.2.1
com.typesafe.scala-logging scala-logging_2.12 3.7.2
com.uber h3 3.7.0
com.univocity univocity-parsers 2.9.1
com.zaxxer HikariCP 4.0.3
commons-cli commons-cli 1.5.0
commons-codec commons-codec 1.15
commons-collections commons-collections 3.2.2
commons-dbcp commons-dbcp 1.4
commons-fileupload commons-fileupload 1.5
commons-httpclient commons-httpclient 3.1
commons-io commons-io 2.11.0
commons-lang commons-lang 2.6
commons-logging commons-logging 1.1.3
commons-pool commons-pool 1.5.4
dev.ludovic.netlib arpack 3.0.3
dev.ludovic.netlib blas 3.0.3
dev.ludovic.netlib lapack 3.0.3
info.ganglia.gmetric4j gmetric4j 1.0.10
io.airlift aircompressor 0.21
io.delta delta-sharing-spark_2.12 0.6.4
io.dropwizard.metrics metrics-core 4.2.10
io.dropwizard.metrics metrics-graphite 4.2.10
io.dropwizard.metrics metrics-healthchecks 4.2.10
io.dropwizard.metrics metrics-jetty9 4.2.10
io.dropwizard.metrics metrics-jmx 4.2.10
io.dropwizard.metrics metrics-json 4.2.10
io.dropwizard.metrics metrics-jvm 4.2.10
io.dropwizard.metrics metrics-servlets 4.2.10
io.netty netty-all 4.1.87.Final
io.netty netty-buffer 4.1.87.Final
io.netty netty-codec 4.1.87.Final
io.netty netty-codec-http 4.1.87.Final
io.netty netty-codec-http2 4.1.87.Final
io.netty netty-codec-socks 4.1.87.Final
io.netty netty-common 4.1.87.Final
io.netty netty-handler 4.1.87.Final
io.netty netty-handler-proxy 4.1.87.Final
io.netty netty-resolver 4.1.87.Final
io.netty netty-transport 4.1.87.Final
io.netty netty-transport-classes-epoll 4.1.87.Final
io.netty netty-transport-classes-kqueue 4.1.87.Final
io.netty netty-transport-native-epoll 4.1.87.Final
io.netty netty-transport-native-epoll 4.1.87.Final-linux-aarch_64
io.netty netty-transport-native-epoll 4.1.87.Final-linux-x86_64
io.netty netty-transport-native-kqueue 4.1.87.Final-osx-aarch_64
io.netty netty-transport-native-kqueue 4.1.87.Final-osx-x86_64
io.netty netty-transport-native-unix-common 4.1.87.Final
io.prometheus simpleclient 0.7.0
io.prometheus simpleclient_common 0.7.0
io.prometheus simpleclient_dropwizard 0.7.0
io.prometheus simpleclient_pushgateway 0.7.0
io.prometheus simpleclient_servlet 0.7.0
io.prometheus.jmx collector 0.12.0
jakarta.annotation jakarta.annotation-api 1.3.5
jakarta.servlet jakarta.servlet-api 4.0.3
jakarta.validation jakarta.validation-api 2.0.2
jakarta.ws.rs jakarta.ws.rs-api 2.1.6
javax.activation activation 1.1.1
javax.el javax.el-api 2.2.4
javax.jdo jdo-api 3.0.1
javax.transaction jta 1.1
javax.transaction transaction-api 1.1
javax.xml.bind jaxb-api 2.2.11
javolution javolution 5.5.1
jline jline 2.14.6
joda-time joda-time 2.12.1
ml.combust.mleap mleap-databricks-runtime_2.12 v0.20.0-db2
net.java.dev.jna jna 5.8.0
net.razorvine pickle 1.3
net.sf.jpam jpam 1.1
net.sf.opencsv opencsv 2.3
net.sf.supercsv super-csv 2.2.0
net.snowflake snowflake-ingest-sdk 0.9.6
net.snowflake snowflake-jdbc 3.13.22
net.sourceforge.f2j arpack_combined_all 0.1
org.acplt.remotetea remotetea-oncrpc 1.1.2
org.antlr ST4 4.0.4
org.antlr antlr-runtime 3.5.2
org.antlr antlr4-runtime 4.9.3
org.antlr stringtemplate 3.2.1
org.apache.ant ant 1.9.16
org.apache.ant ant-jsch 1.9.16
org.apache.ant ant-launcher 1.9.16
org.apache.arrow arrow-format 11.0.0
org.apache.arrow arrow-memory-core 11.0.0
org.apache.arrow arrow-memory-netty 11.0.0
org.apache.arrow arrow-vector 11.0.0
org.apache.avro avro 1.11.1
org.apache.avro avro-ipc 1.11.1
org.apache.avro avro-mapred 1.11.1
org.apache.commons commons-collections4 4.4
org.apache.commons commons-compress 1.21
org.apache.commons commons-crypto 1.1.0
org.apache.commons commons-lang3 3.12.0
org.apache.commons commons-math3 3.6.1
org.apache.commons commons-text 1.10.0
org.apache.curator curator-client 2.13.0
org.apache.curator curator-framework 2.13.0
org.apache.curator curator-recipes 2.13.0
org.apache.datasketches datasketches-java 3.1.0
org.apache.datasketches datasketches-memory 2.0.0
org.apache.derby derby 10.14.2.0
org.apache.hadoop hadoop-client-runtime 3.3.4
org.apache.hive hive-beeline 2.3.9
org.apache.hive hive-cli 2.3.9
org.apache.hive hive-jdbc 2.3.9
org.apache.hive hive-llap-client 2.3.9
org.apache.hive hive-llap-common 2.3.9
org.apache.hive hive-serde 2.3.9
org.apache.hive hive-shims 2.3.9
org.apache.hive hive-storage-api 2.8.1
org.apache.hive.shims hive-shims-0.23 2.3.9
org.apache.hive.shims hive-shims-common 2.3.9
org.apache.hive.shims hive-shims-scheduler 2.3.9
org.apache.httpcomponents httpclient 4.5.14
org.apache.httpcomponents httpcore 4.4.16
org.apache.ivy ivy 2.5.1
org.apache.logging.log4j log4j-1.2-api 2.19.0
org.apache.logging.log4j log4j-api 2.19.0
org.apache.logging.log4j log4j-core 2.19.0
org.apache.logging.log4j log4j-slf4j2-impl 2.19.0
org.apache.mesos mesos 1.11.0-shaded-protobuf
org.apache.orc orc-core 1.8.3-shaded-protobuf
org.apache.orc orc-mapreduce 1.8.3-shaded-protobuf
org.apache.orc orc-shims 1.8.3
org.apache.thrift libfb303 0.9.3
org.apache.thrift libthrift 0.12.0
org.apache.xbean xbean-asm9-shaded 4.22
org.apache.yetus audience-annotations 0.13.0
org.apache.zookeeper zookeeper 3.6.3
org.apache.zookeeper zookeeper-jute 3.6.3
org.checkerframework checker-qual 3.19.0
org.codehaus.jackson jackson-core-asl 1.9.13
org.codehaus.jackson jackson-mapper-asl 1.9.13
org.codehaus.janino commons-compiler 3.0.16
org.codehaus.janino janino 3.0.16
org.datanucleus datanucleus-api-jdo 4.2.4
org.datanucleus datanucleus-core 4.1.17
org.datanucleus datanucleus-rdbms 4.1.19
org.datanucleus javax.jdo 3.2.0-m3
org.eclipse.jetty jetty-client 9.4.50.v20221201
org.eclipse.jetty jetty-continuation 9.4.50.v20221201
org.eclipse.jetty jetty-http 9.4.50.v20221201
org.eclipse.jetty jetty-io 9.4.50.v20221201
org.eclipse.jetty jetty-jndi 9.4.50.v20221201
org.eclipse.jetty jetty-plus 9.4.50.v20221201
org.eclipse.jetty jetty-proxy 9.4.50.v20221201
org.eclipse.jetty jetty-security 9.4.50.v20221201
org.eclipse.jetty jetty-server 9.4.50.v20221201
org.eclipse.jetty jetty-servlet 9.4.50.v20221201
org.eclipse.jetty jetty-servlets 9.4.50.v20221201
org.eclipse.jetty jetty-util 9.4.50.v20221201
org.eclipse.jetty jetty-util-ajax 9.4.50.v20221201
org.eclipse.jetty jetty-webapp 9.4.50.v20221201
org.eclipse.jetty jetty-xml 9.4.50.v20221201
org.eclipse.jetty.websocket websocket-api 9.4.50.v20221201
org.eclipse.jetty.websocket websocket-client 9.4.50.v20221201
org.eclipse.jetty.websocket websocket-common 9.4.50.v20221201
org.eclipse.jetty.websocket websocket-server 9.4.50.v20221201
org.eclipse.jetty.websocket websocket-servlet 9.4.50.v20221201
org.fusesource.leveldbjni leveldbjni-all 1.8
org.glassfish.hk2 hk2-api 2.6.1
org.glassfish.hk2 hk2-locator 2.6.1
org.glassfish.hk2 hk2-utils 2.6.1
org.glassfish.hk2 osgi-resource-locator 1.0.3
org.glassfish.hk2.external aopalliance-repackaged 2.6.1
org.glassfish.hk2.external jakarta.inject 2.6.1
org.glassfish.jersey.containers jersey-container-servlet 2.36
org.glassfish.jersey.containers jersey-container-servlet-core 2.36
org.glassfish.jersey.core jersey-client 2.36
org.glassfish.jersey.core jersey-common 2.36
org.glassfish.jersey.core jersey-server 2.36
org.glassfish.jersey.inject jersey-hk2 2.36
org.hibernate.validator hibernate-validator 6.1.7.Final
org.javassist javassist 3.25.0-GA
org.jboss.logging jboss-logging 3.3.2.Final
org.jdbi jdbi 2.63.1
org.jetbrains annotations 17.0.0
org.joda joda-convert 1.7
org.jodd jodd-core 3.5.2
org.json4s json4s-ast_2.12 3.7.0-M11
org.json4s json4s-core_2.12 3.7.0-M11
org.json4s json4s-jackson_2.12 3.7.0-M11
org.json4s json4s-scalap_2.12 3.7.0-M11
org.lz4 lz4-java 1.8.0
org.mariadb.jdbc mariadb-java-client 2.7.4
org.mlflow mlflow-spark 2.2.0
org.objenesis objenesis 2.5.1
org.postgresql postgresql 42.3.8
org.roaringbitmap RoaringBitmap 0.9.39
org.roaringbitmap shims 0.9.39
org.rocksdb rocksdbjni 7.8.3
org.rosuda.REngine REngine 2.1.0
org.scala-lang scala-compiler_2.12 2.12.15
org.scala-lang scala-library_2.12 2.12.15
org.scala-lang scala-reflect_2.12 2.12.15
org.scala-lang.modules scala-collection-compat_2.12 2.4.3
org.scala-lang.modules scala-parser-combinators_2.12 1.1.2
org.scala-lang.modules scala-xml_2.12 1.2.0
org.scala-sbt test-interface 1.0
org.scalacheck scalacheck_2.12 1.14.2
org.scalactic scalactic_2.12 3.2.15
org.scalanlp breeze-macros_2.12 2.1.0
org.scalanlp breeze_2.12 2.1.0
org.scalatest scalatest-compatible 3.2.15
org.scalatest scalatest-core_2.12 3.2.15
org.scalatest scalatest-diagrams_2.12 3.2.15
org.scalatest scalatest-featurespec_2.12 3.2.15
org.scalatest scalatest-flatspec_2.12 3.2.15
org.scalatest scalatest-freespec_2.12 3.2.15
org.scalatest scalatest-funspec_2.12 3.2.15
org.scalatest scalatest-funsuite_2.12 3.2.15
org.scalatest scalatest-matchers-core_2.12 3.2.15
org.scalatest scalatest-mustmatchers_2.12 3.2.15
org.scalatest scalatest-propspec_2.12 3.2.15
org.scalatest scalatest-refspec_2.12 3.2.15
org.scalatest scalatest-shouldmatchers_2.12 3.2.15
org.scalatest scalatest-wordspec_2.12 3.2.15
org.scalatest scalatest_2.12 3.2.15
org.slf4j jcl-over-slf4j 2.0.6
org.slf4j jul-to-slf4j 2.0.6
org.slf4j slf4j-api 2.0.6
org.threeten threeten-extra 1.7.1
org.tukaani xz 1.9
org.typelevel algebra_2.12 2.0.1
org.typelevel cats-kernel_2.12 2.1.1
org.typelevel spire-macros_2.12 0.17.0
org.typelevel spire-platform_2.12 0.17.0
org.typelevel spire-util_2.12 0.17.0
org.typelevel spire_2.12 0.17.0
org.wildfly.openssl wildfly-openssl 1.1.3.Final
org.xerial sqlite-jdbc 3.42.0.0
org.xerial.snappy snappy-java 1.1.8.4
org.yaml snakeyaml 1.33
oro oro 2.0.8
pl.edu.icm JLargeArrays 1.5
software.amazon.cryptools AmazonCorrettoCryptoProvider 1.6.1-linux-x86_64
software.amazon.ion ion-java 1.0.2
stax stax-api 1.0.1