Databricks Runtime 13.3 LTS

Article
11/07/2024

The following release notes provide information about Databricks Runtime 13.3 LTS, powered by Apache Spark 3.4.1.

Databricks released this version in August 2023.

Note

LTS means this version is under long-term support. See Databricks Runtime LTS version lifecycle.

Tip

To see release notes for Databricks Runtime versions that have reached end-of-support (EoS), see End-of-support Databricks Runtime release notes. The EoS Databricks Runtime versions have been retired and might not be updated.

New features and improvements

Scala support for shared clusters
Allowlist for init scripts, JARs, and Maven coordinates on Unity Catalog shared clusters is in Public Preview
Volumes support for init scripts and JARs is in Public Preview
Safe casts enabled by default for Delta Lake operations
Introducing tags with Unity Catalog
Databricks Runtime returns an error if a file is modified between query planning and invocation
Block schema overwrite when using dynamic partition overwrites
Integrate copyFile API into dbutils.fs.cp
Databricks ODBC/JDBC driver support
Support for unnamed (positional) parameter markers
Enable time series column labeling
New bitmap SQL functions
Improved encryption functions
Unity Catalog support for REFRESH FOREIGN
INSERT BY NAME
Share materialized views with Delta Sharing

Scala support for shared clusters

Unity Catalog-enabled shared clusters now support Scala, when using Databricks Runtime 13.3 and above.

Allowlist for init scripts, JARs, and Maven coordinates on Unity Catalog shared clusters is in Public Preview

You can now use the Unity Catalog allowlist to control installation of init scripts, JARs, and Maven coordinates on compute with shared access mode running Databricks Runtime 13.3 and above. See Allowlist libraries and init scripts on shared compute.

Volumes support for init scripts and JARs is in Public Preview

You can now use Unity Catalog volumes to store init scripts and JARs on compute with assigned or shared access modes running Databricks Runtime 13.3 and above. See Where can init scripts be installed? and Cluster-scoped libraries.

Safe casts enabled by default for Delta Lake operations

Delta UPDATE and MERGE operations now result in an error when values cannot be safely cast to the type in the target table schema. All implicit casts in Delta now follow spark.sql.storeAssignmentPolicy instead of spark.sql.ansi.enabled. See ANSI compliance in Databricks Runtime.

Introducing tags with Unity Catalog

With Databricks Runtime 13.3 LTS and later, you can apply tags to your securable objects. Tags help you organize securable objects and simplify search and discovery of your data assets. See Apply tags to Unity Catalog securable objects.

Databricks Runtime returns an error if a file is modified between query planning and invocation

Databricks Runtime queries now return an error if a file is updated between query planning and invocation. Before this change, Databricks Runtime would read a file between these stages, which occasionally lead to unpredictable results.

Block schema overwrite when using dynamic partition overwrites

You cannot set overwriteSchema to true in combination with dynamic partition overwrites in Delta Lake. This prevents possible table corruption due to schema mismatch. See Dynamic partition overwrites.

Integrate copyFile API into dbutils.fs.cp

The dbutils copy command, dbutils.fs.cp is now optimized for faster copying. With this improvement, copy operations can be up to 100x faster, depending on the file size. The feature is available across all file systems accessible in Azure Databricks, including for Unity Catalog Volumes and Databricks Filesystem mounts.

Databricks ODBC/JDBC driver support

Azure Databricks supports ODBC and JDBC drivers released in the past 2 years. Download the latest driver at Download ODBC Driver and Download JDBC Driver.

Support for unnamed (positional) parameter markers

The SparkSession.sql() API now supports parameterized queries using the ? syntax in addition to parameterized queries using the :parm syntax. See parameter markers.

Enable time series column labeling

You can now label time series columns when creating primary key constraints.

New bitmap SQL functions

Azure Databricks now provided a set of functions which can be used to map integral numeric values onto a bitmap and aggregate them. This can be used to pre-compute number of distinct values.

Improved encryption functions

The aes_encrypt, aes_decrypt, and try_aes_decrypt functions have been improved with support for initialization vectors (IV) and authenticated additional data (AAD).

Unity Catalog support for REFRESH FOREIGN

You can now update foreign catalogs, schemas, and tables in Unity Catalog using REFRESH FOREIGN CATALOG, REFRESH FOREIGN SCHEMA, and REFRESH FOREIGN TABLE. See REFRESH FOREIGN (CATALOG, SCHEMA, and TABLE).

INSERT BY NAME

When inserting rows into a table you can now let Azure Databricks map columns and fields in the source query automatically to the table’s columns by their name. See INSERT INTO.

The ALTER SHARE statement now supports materialized views.

Breaking changes

Changes to implicit casting when storing rows in a table with MERGE INTO and UPDATE

Azure Databricks now follows the configuration spark.sql.storeAssignmentPolicy for implicit casting when storing rows in a table. The default value ANSI throws an error when storing values that overflow. Previously, values would be stored as NULL by default.

As in the following example, you can address overflows by rewriting the table to use a broader type for the column causing the error.

-- Enable column mapping
ALTER TABLE MyTable SET TBLPROPERTIES (
    'delta.minReaderVersion' = '2',
    'delta.minWriterVersion' = '5',
    'delta.columnMapping.mode' = 'name'
  )

-- Rename the old column and create a new column with the new type using the old name
ALTER TABLE MyTable RENAME ID to ID_old
ALTER TABLE MyTable ADD COLUMN ID BIGINT

-- Rewrite the entire table
UPDATE MyTable SET ID = ID_old

-- Drop the old column
ALTER TABLE MyTable DROP COLUMN ID_old

Parquet Schema Inference Changes

When inferring schemas from Parquet files not written by Spark, int64 timestamp columns annotated with isAdjustedToUTC=false will now default to TIMESTAMP_NTZ type. Previously, these were inferred as TIMESTAMP type. This update enhances the accuracy of mapping Parquet types to Spark SQL types.

As a result, reading data from external Parquet files into a Delta table may lead to an error if the timestampNtz feature is not enabled. The error will display as follows:

Fatal exception of the update:
com.databricks.sql.transaction.tahoe.DeltaTableFeatureException
Your table schema requires manual enablement of the following table feature(s): timestampNtz.

To maintain the previous behavior and prevent this error, you can set the Spark configuration parameter spark.sql.parquet.inferTimestampNTZ.enabled to false.

Library upgrades

Upgraded Python libraries:
- debugpy from 1.5.1 to 1.6.0
- distlib from 0.3.6 to 0.3.7
- fastjsonschema from 2.16.3 to 2.18.0
- filelock from 3.12.0 to 3.12.2
Upgraded R libraries:
- SparkR from 3.4.0 to 3.4.1
Upgraded Java libraries:
- org.apache.orc.orc-core from 1.8.3-shaded-protobuf to 1.8.4-shaded-protobuf
- org.apache.orc.orc-mapreduce from 1.8.3-shaded-protobuf to 1.8.4-shaded-protobuf
- org.apache.orc.orc-shims from 1.8.3 to 1.8.4
- org.eclipse.jetty.jetty-client from 9.4.50.v20221201 to 9.4.51.v20230217
- org.eclipse.jetty.jetty-continuation from 9.4.50.v20221201 to 9.4.51.v20230217
- org.eclipse.jetty.jetty-http from 9.4.50.v20221201 to 9.4.51.v20230217
- org.eclipse.jetty.jetty-io from 9.4.50.v20221201 to 9.4.51.v20230217
- org.eclipse.jetty.jetty-jndi from 9.4.50.v20221201 to 9.4.51.v20230217
- org.eclipse.jetty.jetty-plus from 9.4.50.v20221201 to 9.4.51.v20230217
- org.eclipse.jetty.jetty-proxy from 9.4.50.v20221201 to 9.4.51.v20230217
- org.eclipse.jetty.jetty-security from 9.4.50.v20221201 to 9.4.51.v20230217
- org.eclipse.jetty.jetty-server from 9.4.50.v20221201 to 9.4.51.v20230217
- org.eclipse.jetty.jetty-servlet from 9.4.50.v20221201 to 9.4.51.v20230217
- org.eclipse.jetty.jetty-servlets from 9.4.50.v20221201 to 9.4.51.v20230217
- org.eclipse.jetty.jetty-util from 9.4.50.v20221201 to 9.4.51.v20230217
- org.eclipse.jetty.jetty-util-ajax from 9.4.50.v20221201 to 9.4.51.v20230217
- org.eclipse.jetty.jetty-webapp from 9.4.50.v20221201 to 9.4.51.v20230217
- org.eclipse.jetty.jetty-xml from 9.4.50.v20221201 to 9.4.51.v20230217
- org.eclipse.jetty.websocket.websocket-api from 9.4.50.v20221201 to 9.4.51.v20230217
- org.eclipse.jetty.websocket.websocket-client from 9.4.50.v20221201 to 9.4.51.v20230217
- org.eclipse.jetty.websocket.websocket-common from 9.4.50.v20221201 to 9.4.51.v20230217
- org.eclipse.jetty.websocket.websocket-server from 9.4.50.v20221201 to 9.4.51.v20230217
- org.eclipse.jetty.websocket.websocket-servlet from 9.4.50.v20221201 to 9.4.51.v20230217
- org.xerial.snappy.snappy-java from 1.1.8.4 to 1.1.10.1

Apache Spark

Databricks Runtime 13.3 includes Apache Spark 3.4.1. This release includes all Spark fixes and improvements included in Databricks Runtime 13.2 (EoS), as well as the following additional bug fixes and improvements made to Spark:

You can now set cluster environment variable SNOWFLAKE_SPARK_CONNECTOR_VERSION=2.12 to use Spark-snowflake connector v2.12.0.
[SPARK-43380] [DBRRM-383] Revert “[SC-133100][SQL] Fix Avro data type conver…
[SPARK-44504] [Backport][13.2][13.x][13.3] Unload provider thereby forcing DB instance close and releasing resources on maintenance task error
[SPARK-44059] [SC-137023] Add analyzer support of named arguments for built-in functions
[SPARK-44485] [SC-137653][SQL] Optimize TreeNode.generateTreeString
[SPARK-42944] [SC-137164][SS][PYTHON] Streaming ForeachBatch in Python
[SPARK-44446] [SC-136994][PYTHON] Add checks for expected list type special cases
[SPARK-44439] [SC-136973][CONNECT][SS]Fixed listListeners to only send ids back to client
[SPARK-44216] [SC-136941] [PYTHON] Make assertSchemaEqual API public
[SPARK-44201] [SC-136778][CONNECT][SS]Add support for Streaming Listener in Scala for Spark Connect
[SPARK-43915] [SC-134766][SQL] Assign names to the error class LEGACY_ERROR_TEMP[2438-2445]
[SPARK-43798] Revert “[SC-133990][SQL][PYTHON] Support Python user-defined table functions”
[SPARK-44398] [SC-136720][CONNECT] Scala foreachBatch API
[SPARK-44476] [SC-137169][CORE][CONNECT] Fix population of artifacts for a JobArtifactState with no associated artifacts
[SPARK-44269] [SC-135817][SQL] Assign names to the error class LEGACY_ERROR_TEMP[2310-2314]
[SPARK-44395] [SC-136744][SQL] Update TVF arguments to require parentheses around identifier after TABLE keyword
[SPARK-43915] [SC-134766][SQL] Assign names to the error class LEGACY_ERROR_TEMP[2438-2445]
[SPARK-43995] [SC-136794][SPARK-43996][CONNECT] Add support for UDFRegistration to the Connect Scala Client
[SPARK-44109] [SC-134755][CORE] Remove duplicate preferred locations of each RDD partition
[SPARK-44169] [SC-135497][SQL] Assign names to the error class LEGACY_ERROR_TEMP[2300-2304]
[SPARK-41487] [SC-135636][SQL] Assign name to _LEGACY_ERROR_TEMP_1020
[SPARK-44398] [SC-136720][CONNECT] Scala foreachBatch API
[SPARK-44153] [SC-134977][CORE][UI] Support Heap Histogram column in Executors tab
[SPARK-44044] [SC-135695][SS] Improve Error message for Window functions with streaming
[SPARK-43914] [SC-135414][SQL] Assign names to the error class LEGACY_ERROR_TEMP[2433-2437]
[SPARK-44217] [SC-136805][PYTHON] Allow custom precision for fp approx equality
[SPARK-44370] [SC-136575][CONNECT] Migrate Buf remote generation alpha to remote plugins
[SPARK-44410] [SC-136930][PYTHON][CONNECT] Set active session in create, not just getOrCreate
[SPARK-44348] [SAS-1910][SC-136644][13.X][CORE][CONNECT][PYTHON] Reenable test_artifact with relevant changes
[SPARK-44145] [SC-136698][SQL] Callback when ready for execution
[SPARK-44264] [SC-136870][PYTHON][ML] FunctionPickler Class
[SPARK-43986] [SC-135623][SQL] Create error classes for HyperLogLog function call failures
[SPARK-44139] [SC-134967][SQL] Discard completely pushed down filters in group-based MERGE operations
[SPARK-44338] [SC-136718][SQL] Fix view schema mismatch error message
[SPARK-44208] [SC-135505][CORE][SQL] Assign clear error class names for some logic that directly uses exceptions
[SPARK-44364] [SC-136773] [PYTHON] Add support for List[Row] data type for expected
[SPARK-44180] [SC-136792][SQL] DistributionAndOrderingUtils should apply ResolveTimeZone
[SPARK-43389] [SC-136772][SQL] Added a null check for lineSep option
[SPARK-44388] [SC-136695][CONNECT] Fix protobuf cast issue when UDF instance is updated
[SPARK-44352] [SC-136373][CONNECT] Put back sameType and friends in DataType.
[SPARK-43975] [SC-134265][SQL] DataSource V2: Handle UPDATE commands for group-based sources
[SPARK-44360] [SC-136461][SQL] Support schema pruning in delta-based MERGE operations
[SPARK-44138] [SC-134907][SQL] Prohibit non-deterministic expressions, subqueries and aggregates in MERGE conditions
[SPARK-43321] [SC-136678][Connect][Followup] Better names for APIs used in Scala Client joinWith
[SPARK-44263] [SC-136447][CONNECT] Custom Interceptors Support
[SPARK-39851] [SC-123383][SQL] Improve join stats estimation if one side can keep uniqueness
[SPARK-44390] [SC-136677][CORE][SQL] Rename SparkSerDerseUtils to SparkSerDeUtils
[SPARK-44004] [SC-134751][SQL] Assign name & improve error message for frequent LEGACY errors.
[SPARK-44322] [SC-136204][CONNECT] Make parser use SqlApiConf instead of SQLConf.
[SPARK-44271] [SC-136347][SQL] Move default values functions from StructType to ResolveDefaultColumns
[SPARK-44332] [SC-136413][CORE][WEBUI] Fix the sorting error of Executor ID Column on Executors UI Page
[SPARK-44282] [SC-135948][CONNECT] Prepare DataType parsing for use in Spark Connect Scala Client
[SPARK-44349] [SC-136581][R] Add math functions to SparkR
[SPARK-44397] [SC-136647][PYTHON] Expose assertDataFrameEqual in pyspark.testing.utils
[SPARK-42981] [SC-136620][CONNECT] Add direct arrow serialization
[SPARK-44373] [SC-136577][SQL] Wrap withActive for Dataset API w/ parse logic to make parser related configuration work
[SPARK-44340] [SC-136574][SQL] Define the computing logic through PartitionEvaluator API and use it in WindowGroupLimitExec
[SPARK-43665] [SC-136573][CONNECT][PS] Enable PandasSQLStringFormatter.vformat to work with Spark Connect
[SPARK-44325] [SC-136572][SQL] Use PartitionEvaluator API in SortMergeJoinExec
[SPARK-44315] [SC-136171][SQL][CONNECT] Move DefinedByConstructorParams to sql/api
[SPARK-44275] [SC-136184][CONNECT] Add configurable retry mechanism to Scala Spark Connect
[SPARK-44326] [SC-136208][SQL][CONNECT] Move utils that are used from Scala client to the common modules
[SPARK-43321] [SC-136156][Connect] Dataset#Joinwith
[SPARK-44343] [SC-136372][CONNECT] Prepare ScalaReflection to the move to SQL/API
[SPARK-44313] [SC-136081][SQL] Fix generated column expression validation when there is a char/varchar column in the schema
[SPARK-43963] [SC-134145][SQL] DataSource V2: Handle MERGE commands for group-based sources
[SPARK-44363] [SC-136433] [PYTHON] Display percent of unequal rows in DataFrame comparison
[SPARK-44251] [SC-136432][SQL] Set nullable correctly on coalesced join key in full outer USING join
[SPARK-43948] [SC-133366][SQL] Assign names to the error class LEGACY_ERROR_TEMP[0050|0057|0058|0059]
[SPARK-44329] [SC-136326][CONNECT][PYTHON] Add hll_sketch_agg, hll_union_agg, to_varchar, try_aes_decrypt to Scala and Python
[SPARK-44351] [SC-136353][SQL] Make some syntactic simplification
[SPARK-44281] [SC-135963][SQL] Move QueryCompilation error that used by DataType to sql/api as DataTypeErrors
[SPARK-44283] [SC-136109][CONNECT] Move Origin to SQL/API
[SPARK-43926] [SC-135590][CONNECT][PYTHON] Add array_agg, array_size, cardinality, count_min_sketch,mask,named_struct,json_* to Scala and Python
[SPARK-44327] [SC-136187][SQL][CONNECT] Add functions any and len to Scala
[SPARK-44290] [SC-136300][CONNECT] Session-based files and archives in Spark Connect
[SPARK-44061] [SC-136299][PYTHON] Add assertDataFrameEqual util function
[SPARK-44331] [SC-136324][CONNECT][PYTHON] Add bitmap functions to Scala and Python
[SPARK-44342] [SC-136334][SQL] Replace SQLContext with SparkSession for GenTPCDSData
[SPARK-42583] [SC-124190][SQL] Remove the outer join if they are all distinct aggregate functions
[SPARK-44333] [SC-136206][CONNECT][SQL] Move EnhancedLogicalPlan out of ParserUtils
[SPARK-43824] [SC-132655][SPARK-43825] [SQL] Assign names to the error class _LEGACY_ERROR_TEMP_128[1-2]
[SPARK-43939] [SC-134602][CONNECT][PYTHON] Add try_* functions to Scala and Python
[SPARK-44193] [SC-135886][CONNECT] Implement GRPC exceptions interception for conversion
[SPARK-44284] [SC-136107][CONNECT] Create simple conf system for sql/api
[SPARK-43913] [SC-133368][SQL] Assign names to the error class LEGACY_ERROR_TEMP[2426-2432]
[SPARK-44291] Revert “[SC-135924][SPARK-43416][CONNECT] Fix incorrect schema for range query”
[SPARK-44312] [SC-136185][CONNECT][PYTHON] Allow to set a user agent with an environment variable
[SPARK-44154] [SC-136060] Implement bitmap functions
[SPARK-43885] [SC-133996][SQL] DataSource V2: Handle MERGE commands for delta-based sources
[SPARK-43924] [SC-135161][CONNECT][PYTHON] Add misc functions to Scala and Python
[SPARK-43969] [SC-134556][SQL] Refactor & Assign names to the error class _LEGACY_ERROR_TEMP_1170
[SPARK-44185] [SC-135730][SQL] Fix inconsistent path qualifying between catalog and data operations
[SPARK-44073] [SC-134562][SQL][PYTHON][CONNECT] Add date time functions to Scala, Python and Connect - part 2
[SPARK-43942] [SC-134527][CONNECT][PYTHON] Add string functions to Scala and Python - part 1
[SPARK-44266] [SC-135813][SQL] Move Util.truncatedString to sql/api
[SPARK-44195] [SC-135722][R] Add JobTag APIs to SparkR SparkContext
[SPARK-44200] [SC-135736][SQL] Support TABLE argument parser rule for TableValuedFunction
[SPARK-44254] [SC-135709][SQL] Move QueryExecutionErrors that used by DataType to sql/api as DataTypeErrors
[SPARK-44291] [SC-135924][SPARK-43416][CONNECT] Fix incorrect schema for range query
[SPARK-43922] [SC-135612][SQL] Add named parameter support in parser for function calls
[SPARK-44300] [SC-135882][CONNECT] Fix artifact cleanup to limit deletion scope to session specific artifacts
[SPARK-44293] [SC-135849][CONNECT] Fix invalid URI for custom JARs in Spark Connect
[SPARK-44215] [SC-135850][SHUFFLE] If num chunks are 0, then server should throw a RuntimeException
[SPARK-44245] [SC-135851][PYTHON] pyspark.sql.dataframe sample() doctests should be illustrative-only
[SPARK-44274] [SC-135814][CONNECT] Move out util functions used by ArtifactManager to common/utils
[SPARK-44194] [SC-135815][PYTHON][CORE] Add JobTag APIs to PySpark SparkContext
[SPARK-42828] [SC-135733][PYTHON][SQL] More explicit Python type annotations for GroupedData
[SPARK-44079] [SC-135486][SQL] Fix ArrayIndexOutOfBoundsException when parse array as struct using PERMISSIVE mode with corrupt record
[SPARK-44255] [SC-135809][SQL] Relocate StorageLevel to common/utils
[SPARK-42169] [SC-135795] [SQL] Implement code generation for to_csv function (StructsToCsv)
[SPARK-44249] [SC-135719][SQL][PYTHON] Refactor PythonUDTFRunner to send its return type separately
[SPARK-43353] [SC-132734][PYTHON] Migrate remaining session errors into error class
[SPARK-44211] [SC-135718][PYTHON][CONNECT] Implement SparkSession.is_stopped
[SPARK-42784] [SC-135691] should still create subDir when the number of subDir in merge dir is less than conf
[SPARK-41599] [SC-135616] Memory leak in FileSystem.CACHE when submitting apps to secure cluster using InProcessLauncher
[SPARK-44241] [SC-135613][Core] Mistakenly set io.connectionTimeout/connectionCreationTimeout to zero or negative will cause incessant executor cons/destructions
[SPARK-44133] [SC-134795][13.X][PYTHON] Upgrade MyPy from 0.920 to 0.982
[SPARK-42941] [SC-134707][SS][CONNECT][1/2] StreamingQueryListener - Event Serde in JSON format
[SPARK-44188] [SC-135173][CORE] Remove useless resetAllPartitions method in ActiveJob
[SPARK-43757] [SC-135418][CONNECT] Change client compatibility from allow list to deny list
[SPARK-43474] [SC-135521] [SS] [CONNECT] Add a spark connect access to runtime Dataframes by ID.
[SPARK-44183] [SC-135209][PYTHON] Increate PyArrow minimum version to 4.0.0
[SPARK-44248] [SC-135554][SS][SQL][Kafka] Add preferred location in kafka source v2
[SPARK-43929] [SC-134510][SQL][PYTHON][CONNECT] Add date time functions to Scala, Python and Connect API - part 1
[SPARK-44161] [SC-135326][Connect] Handle Row input for UDFs
[SPARK-44227] [SC-135520][SQL] Extract SchemaUtils from StructField
[SPARK-44064] [SC-135116][CORE][SQL] Add a new apply function to NonFateSharingCache
[SPARK-44171] [SC-135269][SQL] Assign names to the error class LEGACY_ERROR_TEMP[2279-2282] & delete some unused error classes
[SPARK-44056] [SC-134758][SQL] Include UDF name in UDF execution failure error message when available
[SPARK-44205] [SC-135422][SQL] Extract Catalyst Code from DecimalType
[SPARK-44220] [SC-135484][SQL] Move StringConcat to sql/api
[SPARK-44146] [SC-135230][CONNECT] Isolate Spark Connect Session jars and classfiles
[SPARK-44206] [SC-135411][SQL] DataSet.selectExpr scope Session.active
[SPARK-40850] [SC-135417][SQL] Fix test case interpreted queries may execute Codegen
[SPARK-44204] [SC-135253][SQL][HIVE] Add missing recordHiveCall for getPartitionNames
[SPARK-44237] [SC-135485][CORE] Simplify DirectByteBuffer constructor lookup logic
[SPARK-44189] [SC-135170][CONNECT][PYTHON] Support positional parameters by sql()
[SPARK-44140] [SC-134893][SQL][PYTHON] Support positional parameters in Python sql()
[SPARK-44164] [SC-135158][SQL] Extract toAttribute method from StructField to Util class
[SPARK-44178] [SC-135113][CONNECT] Support positional parameters in sql()
[SPARK-43208] [SC-128955][SQL][HIVE] IsolatedClassLoader should close barrier class InputStream after reading
[SPARK-43063] [SC-128382][SQL] df.show handle null should print NULL instead of null
[SPARK-42539] [SC-124464][SQL][HIVE] Eliminate separate classloader when using ‘builtin’ Hive version for metadata client
[SPARK-43992] [SC-133645][SQL][PYTHON][CONNECT] Add optional pattern for Catalog.listFunctions
[SPARK-44066] [SC-134878][SQL] Support positional parameters in Scala/Java sql()
[SPARK-44078] [SC-134869][CONNECT][CORE] Add support for classloader/resource isolation
[SPARK-43470] [SC-135187][CORE] Add OS, Java, Python version information to application log
[SPARK-43136] [SC-135238][CONNECT][Followup] Adding tests for KeyAs
[SPARK-39740] [SC-135093][UI]: Upgrade vis timeline to 7.7.2 to fix CVE-2020-28487
[SPARK-42298] [SC-133952][SQL] Assign name to _LEGACY_ERROR_TEMP_2132
[SPARK-43961] [SC-133459][SQL][PYTHON][CONNECT] Add optional pattern for Catalog.listTables
[SPARK-44026] [SC-135181] Allow providing initial value for SQLMetrics
[SPARK-44202] [SC-135244][CORE] Add JobTag APIs to JavaSparkContext
[SPARK-44030] [SC-135237][SQL] Implement DataTypeExpression to offer Unapply for expression
[SPARK-43876] Revert “[SC-134603][SQL] Enable fast hashmap for distinct queries”
[SPARK-40082] [SC-126437] Schedule mergeFinalize when push merge shuffleMapStage retry but no running tasks
[SPARK-43888] [SC-132893][CORE] Relocate Logging to common/utils
[SPARK-43474] [SC-134400] [SS] [CONNECT] Add SessionHolder to SparkConnectPlanner
[SPARK-43944] [SC-134490][CONNECT][PYTHON] Add string functions to Scala and Python - part 2
[SPARK-44134] [SC-134954][CORE] Fix setting resources (GPU/FPGA) to 0 when they are set in spark-defaults.conf
[SPARK-44142] [SC-134947][PYTHON] Replace type with tpe in utility to convert python types to spark types
[SPARK-43952] [SC-134695][CORE][CONNECT][SQL] Add SparkContext APIs for query cancellation by tag
[SPARK-44000] [SC-133634][SQL] Add hint to disable broadcasting and replicating one side of join
[SPARK-43937] [SC-134362][CONNECT][PYTHON] Add ifnull,isnotnull,equal_null,nullif,nvl,nvl2 to Scala and Python
[SPARK-43624] [13.x][SC-134557][PS][CONNECT] Add EWM to SparkConnectPlanner.
[SPARK-44040] [SC-134366][SQL] Fix compute stats when AggregateExec node above QueryStageExec
[SPARK-43485] [SC-131264][SQL] Fix the error message for the unit argument of the datetime add/diff functions
[SPARK-43794] [SC-132561][SQL] Assign a name to the error class _LEGACY_ERROR_TEMP_1335
[SPARK-43511] [SC-134749][CONNECT][SS]Implemented MapGroupsWithState and FlatMapGroupsWithState APIs for Spark Connect
[SPARK-43529] [SQL] Support CREATE/REPLACE TABLE OPTIONS expressions + code cleanup
[SPARK-44106] [SC-134750][PYTHON][CONNECT] Add __repr__ for GroupedData
[SPARK-42299] [SC-133190] Assign name to _LEGACY_ERROR_TEMP_2206
[SPARK-43290] [SC-134388][SQL] Adds support for aes_encrypt IVs and AAD
[SPARK-43932] [SC-134138][SQL][PYTHON][CONNECT] Add current like functions to Scala and Python
[SPARK-43934] [SC-134108][SQL][PYTHON][CONNECT] Add regexp_* functions to Scala and Python
[SPARK-44107] [SC-134746][CONNECT][PYTHON] Hide unsupported Column methods from auto-completion
[SPARK-42941] [SC-134707][SS][CONNECT][1/2] StreamingQueryListener - Event Serde in JSON format
[SPARK-43773] [SC-132659][CONNECT][PYTHON] Implement ‘levenshtein(str1, str2[, threshold])’ functions in python client
[SPARK-44125] [SC-134745][R] Support Java 21 in SparkR
[SPARK-43919] [SC-133374][SQL] Extract JSON functionality out of Row
[SPARK-43769] [SC-132520][CONNECT] Implement ‘levenshtein(str1, str2[, threshold])’ functions
[SPARK-44012] [SC-134662][SS] KafkaDataConsumer to print some read status
[SPARK-43876] [SC-134603][SQL] Enable fast hashmap for distinct queries
[SPARK-44024] [SC-134497][SQL] Change to use map when unzip only used to extract a single element
[SPARK-43928] [SC-134467][SQL][PYTHON][CONNECT] Add bit operations to Scala, Python and Connect API
[SPARK-44092] [SC-134506][CORE] Add Utils.isJavaVersionAtLeast21 and make core module pass with Java 21
[SPARK-44018] [SC-134512][SQL] Improve the hashCode and toString for some DS V2 Expression
[SPARK-44105] [SC-134582][SQL] LastNonNull should be lazily resolved
[SPARK-44081] [SC-134434] Simplify PartitionedFileUtil API a little
[SPARK-43791] [SC-132531][SQL] Assign a name to the error class _LEGACY_ERROR_TEMP_1336
[SPARK-44071] [SC-134435] Define and use Unresolved[Leaf|Unary]Node traits.
[SPARK-43852] [SC-132890][SPARK-43853][SPARK-43854][SPARK-43855][SPARK-43856] Assign names to the error class _LEGACY_ERROR_TEMP_2418-2425
[SPARK-43742] [SC-132684][SQL] Refactor default column value resolution
[SPARK-43493] [SC-132063][SQL] Add a max distance argument to the levenshtein() function
[SPARK-44075] [SC-134382][CONNECT] Make transformStatCorr lazy
[SPARK-43925] [SC-134381][SQL][PYTHON][CONNECT] Add some, bool_or,bool_and,every to Scala, Python and Connect
[SPARK-43931] [SC-134158][SQL][PYTHON][CONNECT] Add make_* functions to Scala and Python
[SPARK-43762] [SC-132602][SPARK-43763][SPARK-43764][SPARK-43765][SPARK-43766][SQL] Assign names to the error class _LEGACY_ERROR_TEMP_24[06-10]
[SPARK-43962] [SC-133367][SQL] Improve error messages: CANNOT_DECODE_URL, CANNOT_MERGE_INCOMPATIBLE_DATA_TYPE, CANNOT_PARSE_DECIMAL, CANNOT_READ_FILE_FOOTER, CANNOT_RECOGNIZE_HIVE_TYPE.
[SPARK-43938] [SC-133887][CONNECT][PYTHON] Add to_* functions to Scala and Python
[SPARK-44055] [SC-134321][CORE] Remove redundant override functions from CheckpointRDD
[SPARK-43802] [SC-132597][SQL] Fix codegen for unhex and unbase64 with failOnError=true
[SPARK-43798] [SC-133990][SQL][PYTHON] Support Python user-defined table functions
[SPARK-43941] [SC-134286][SQL][PYTHON][CONNECT] Add any_value, approx_percentile, count_if, first_value, histogram_numeric, last_value, reduce to Scala, Python and Connect API
[SPARK-43380] [SC-133100][SQL] Fix Avro data type conversion issues to avoid producing incorrect results
[SPARK-43803] [SC-133729] [SS] [CONNECT] Improve awaitTermination() to handle client disconnects
[SPARK-43179] [SC-129070][SHUFFLE] Allowing apps to control whether their metadata gets saved in the db by the External Shuffle Service

Databricks ODBC/JDBC driver support

Databricks supports ODBC/JDBC drivers released in the past 2 years. Please download the recently released drivers and upgrade (download ODBC, download JDBC).

See Databricks Runtime 13.3 maintenance updates.

System environment

Operating System: Ubuntu 22.04.2 LTS
Java: Zulu 8.70.0.23-CA-linux64
Scala: 2.12.15
Python: 3.10.12
R: 4.2.2
Delta Lake: 2.4.0

Installed Python libraries

Library	Version	Library	Version	Library	Version
appdirs	1.4.4	argon2-cffi	21.3.0	argon2-cffi-bindings	21.2.0
asttokens	2.2.1	attrs	21.4.0	backcall	0.2.0
beautifulsoup4	4.11.1	black	22.6.0	bleach	4.1.0
blinker	1.4	boto3	1.24.28	botocore	1.27.28
certifi	2022.9.14	cffi	1.15.1	chardet	4.0.0
charset-normalizer	2.0.4	click	8.0.4	cryptography	37.0.1
cycler	0.11.0	Cython	0.29.32	databricks-sdk	0.1.6
dbus-python	1.2.18	debugpy	1.6.0	decorator	5.1.1
defusedxml	0.7.1	distlib	0.3.7	docstring-to-markdown	0.12
entrypoints	0.4	executing	1.2.0	facets-overview	1.0.3
fastjsonschema	2.18.0	filelock	3.12.2	fonttools	4.25.0
googleapis-common-protos	1.56.4	grpcio	1.48.1	grpcio-status	1.48.1
httplib2	0.20.2	idna	3.3	importlib-metadata	4.6.4
ipykernel	6.17.1	ipython	8.10.0	ipython-genutils	0.2.0
ipywidgets	7.7.2	jedi	0.18.1	jeepney	0.7.1
Jinja2	2.11.3	jmespath	0.10.0	joblib	1.2.0
jsonschema	4.16.0	jupyter-client	7.3.4	jupyter_core	4.11.2
jupyterlab-pygments	0.1.2	jupyterlab-widgets	1.0.0	keyring	23.5.0
kiwisolver	1.4.2	launchpadlib	1.10.16	lazr.restfulclient	0.14.4
lazr.uri	1.0.6	MarkupSafe	2.0.1	matplotlib	3.5.2
matplotlib-inline	0.1.6	mccabe	0.7.0	mistune	0.8.4
more-itertools	8.10.0	mypy-extensions	0.4.3	nbclient	0.5.13
nbconvert	6.4.4	nbformat	5.5.0	nest-asyncio	1.5.5
nodeenv	1.8.0	notebook	6.4.12	numpy	1.21.5
oauthlib	3.2.0	packaging	21.3	pandas	1.4.4
pandocfilters	1.5.0	parso	0.8.3	pathspec	0.9.0
patsy	0.5.2	pexpect	4.8.0	pickleshare	0.7.5
Pillow	9.2.0	pip	22.2.2	platformdirs	2.5.2
plotly	5.9.0	pluggy	1.0.0	prometheus-client	0.14.1
prompt-toolkit	3.0.36	protobuf	3.19.4	psutil	5.9.0
psycopg2	2.9.3	ptyprocess	0.7.0	pure-eval	0.2.2
pyarrow	8.0.0	pycparser	2.21	pydantic	1.10.6
pyflakes	3.0.1	Pygments	2.11.2	PyGObject	3.42.1
PyJWT	2.3.0	pyodbc	4.0.32	pyparsing	3.0.9
pyright	1.1.294	pyrsistent	0.18.0	python-dateutil	2.8.2
python-lsp-jsonrpc	1.0.0	python-lsp-server	1.7.1	pytoolconfig	1.2.2
pytz	2022.1	pyzmq	23.2.0	requests	2.28.1
rope	1.7.0	s3transfer	0.6.0	scikit-learn	1.1.1
scipy	1.9.1	seaborn	0.11.2	SecretStorage	3.3.1
Send2Trash	1.8.0	setuptools	63.4.1	six	1.16.0
soupsieve	2.3.1	ssh-import-id	5.11	stack-data	0.6.2
statsmodels	0.13.2	tenacity	8.1.0	terminado	0.13.1
testpath	0.6.0	threadpoolctl	2.2.0	tokenize-rt	4.2.1
tomli	2.0.1	tornado	6.1	traitlets	5.1.1
typing_extensions	4.3.0	ujson	5.4.0	unattended-upgrades	0.1
urllib3	1.26.11	virtualenv	20.16.3	wadllib	1.3.6
wcwidth	0.2.5	webencodings	0.5.1	whatthepatch	1.0.2
wheel	0.37.1	widgetsnbextension	3.6.1	yapf	0.31.0
zipp	1.0.0

Installed R libraries

R libraries are installed from the Microsoft CRAN snapshot on 2023-02-10.

Library	Version	Library	Version	Library	Version
arrow	10.0.1	askpass	1.1	assertthat	0.2.1
backports	1.4.1	base	4.2.2	base64enc	0.1-3
bit	4.0.5	bit64	4.0.5	blob	1.2.3
boot	1.3-28	brew	1.0-8	brio	1.1.3
broom	1.0.3	bslib	0.4.2	cachem	1.0.6
callr	3.7.3	caret	6.0-93	cellranger	1.1.0
chron	2.3-59	class	7.3-21	cli	3.6.0
clipr	0.8.0	clock	0.6.1	cluster	2.1.4
codetools	0.2-19	colorspace	2.1-0	commonmark	1.8.1
compiler	4.2.2	config	0.3.1	cpp11	0.4.3
crayon	1.5.2	credentials	1.3.2	curl	5.0.0
data.table	1.14.6	datasets	4.2.2	DBI	1.1.3
dbplyr	2.3.0	desc	1.4.2	devtools	2.4.5
diffobj	0.3.5	digest	0.6.31	downlit	0.4.2
dplyr	1.1.0	dtplyr	1.2.2	e1071	1.7-13
ellipsis	0.3.2	evaluate	0.20	fansi	1.0.4
farver	2.1.1	fastmap	1.1.0	fontawesome	0.5.0
forcats	1.0.0	foreach	1.5.2	foreign	0.8-82
forge	0.2.0	fs	1.6.1	future	1.31.0
future.apply	1.10.0	gargle	1.3.0	generics	0.1.3
gert	1.9.2	ggplot2	3.4.0	gh	1.3.1
gitcreds	0.1.2	glmnet	4.1-6	globals	0.16.2
glue	1.6.2	googledrive	2.0.0	googlesheets4	1.0.1
gower	1.0.1	graphics	4.2.2	grDevices	4.2.2
grid	4.2.2	gridExtra	2.3	gsubfn	0.7
gtable	0.3.1	hardhat	1.2.0	haven	2.5.1
highr	0.10	hms	1.1.2	htmltools	0.5.4
htmlwidgets	1.6.1	httpuv	1.6.8	httr	1.4.4
ids	1.0.1	ini	0.3.1	ipred	0.9-13
isoband	0.2.7	iterators	1.0.14	jquerylib	0.1.4
jsonlite	1.8.4	KernSmooth	2.23-20	knitr	1.42
labeling	0.4.2	later	1.3.0	lattice	0.20-45
lava	1.7.1	lifecycle	1.0.3	listenv	0.9.0
lubridate	1.9.1	magrittr	2.0.3	markdown	1.5
MASS	7.3-58.2	Matrix	1.5-1	memoise	2.0.1
methods	4.2.2	mgcv	1.8-41	mime	0.12
miniUI	0.1.1.1	ModelMetrics	1.2.2.2	modelr	0.1.10
munsell	0.5.0	nlme	3.1-162	nnet	7.3-18
numDeriv	2016.8-1.1	openssl	2.0.5	parallel	4.2.2
parallelly	1.34.0	pillar	1.8.1	pkgbuild	1.4.0
pkgconfig	2.0.3	pkgdown	2.0.7	pkgload	1.3.2
plogr	0.2.0	plyr	1.8.8	praise	1.0.0
prettyunits	1.1.1	pROC	1.18.0	processx	3.8.0
prodlim	2019.11.13	profvis	0.3.7	progress	1.2.2
progressr	0.13.0	promises	1.2.0.1	proto	1.0.0
proxy	0.4-27	ps	1.7.2	purrr	1.0.1
r2d3	0.2.6	R6	2.5.1	ragg	1.2.5
randomForest	4.7-1.1	rappdirs	0.3.3	rcmdcheck	1.4.0
RColorBrewer	1.1-3	Rcpp	1.0.10	RcppEigen	0.3.3.9.3
readr	2.1.3	readxl	1.4.2	recipes	1.0.4
rematch	1.0.1	rematch2	2.1.2	remotes	2.4.2
reprex	2.0.2	reshape2	1.4.4	rlang	1.0.6
rmarkdown	2.20	RODBC	1.3-20	roxygen2	7.2.3
rpart	4.1.19	rprojroot	2.0.3	Rserve	1.8-12
RSQLite	2.2.20	rstudioapi	0.14	rversions	2.1.2
rvest	1.0.3	sass	0.4.5	scales	1.2.1
selectr	0.4-2	sessioninfo	1.2.2	shape	1.4.6
shiny	1.7.4	sourcetools	0.1.7-1	sparklyr	1.7.9
SparkR	3.4.1	spatial	7.3-15	splines	4.2.2
sqldf	0.4-11	SQUAREM	2021.1	stats	4.2.2
stats4	4.2.2	stringi	1.7.12	stringr	1.5.0
survival	3.5-3	sys	3.4.1	systemfonts	1.0.4
tcltk	4.2.2	testthat	3.1.6	textshaping	0.3.6
tibble	3.1.8	tidyr	1.3.0	tidyselect	1.2.0
tidyverse	1.3.2	timechange	0.2.0	timeDate	4022.108
tinytex	0.44	tools	4.2.2	tzdb	0.3.0
urlchecker	1.0.1	usethis	2.1.6	utf8	1.2.3
utils	4.2.2	uuid	1.1-0	vctrs	0.5.2
viridisLite	0.4.1	vroom	1.6.1	waldo	0.4.0
whisker	0.4.1	withr	2.5.0	xfun	0.37
xml2	1.3.3	xopen	1.0.0	xtable	1.8-4
yaml	2.3.7	zip	2.2.2

Installed Java and Scala libraries (Scala 2.12 cluster version)

Group ID	Artifact ID	Version
antlr	antlr	2.7.7
com.amazonaws	amazon-kinesis-client	1.12.0
com.amazonaws	aws-java-sdk-autoscaling	1.12.390
com.amazonaws	aws-java-sdk-cloudformation	1.12.390
com.amazonaws	aws-java-sdk-cloudfront	1.12.390
com.amazonaws	aws-java-sdk-cloudhsm	1.12.390
com.amazonaws	aws-java-sdk-cloudsearch	1.12.390
com.amazonaws	aws-java-sdk-cloudtrail	1.12.390
com.amazonaws	aws-java-sdk-cloudwatch	1.12.390
com.amazonaws	aws-java-sdk-cloudwatchmetrics	1.12.390
com.amazonaws	aws-java-sdk-codedeploy	1.12.390
com.amazonaws	aws-java-sdk-cognitoidentity	1.12.390
com.amazonaws	aws-java-sdk-cognitosync	1.12.390
com.amazonaws	aws-java-sdk-config	1.12.390
com.amazonaws	aws-java-sdk-core	1.12.390
com.amazonaws	aws-java-sdk-datapipeline	1.12.390
com.amazonaws	aws-java-sdk-directconnect	1.12.390
com.amazonaws	aws-java-sdk-directory	1.12.390
com.amazonaws	aws-java-sdk-dynamodb	1.12.390
com.amazonaws	aws-java-sdk-ec2	1.12.390
com.amazonaws	aws-java-sdk-ecs	1.12.390
com.amazonaws	aws-java-sdk-efs	1.12.390
com.amazonaws	aws-java-sdk-elasticache	1.12.390
com.amazonaws	aws-java-sdk-elasticbeanstalk	1.12.390
com.amazonaws	aws-java-sdk-elasticloadbalancing	1.12.390
com.amazonaws	aws-java-sdk-elastictranscoder	1.12.390
com.amazonaws	aws-java-sdk-emr	1.12.390
com.amazonaws	aws-java-sdk-glacier	1.12.390
com.amazonaws	aws-java-sdk-glue	1.12.390
com.amazonaws	aws-java-sdk-iam	1.12.390
com.amazonaws	aws-java-sdk-importexport	1.12.390
com.amazonaws	aws-java-sdk-kinesis	1.12.390
com.amazonaws	aws-java-sdk-kms	1.12.390
com.amazonaws	aws-java-sdk-lambda	1.12.390
com.amazonaws	aws-java-sdk-logs	1.12.390
com.amazonaws	aws-java-sdk-machinelearning	1.12.390
com.amazonaws	aws-java-sdk-opsworks	1.12.390
com.amazonaws	aws-java-sdk-rds	1.12.390
com.amazonaws	aws-java-sdk-redshift	1.12.390
com.amazonaws	aws-java-sdk-route53	1.12.390
com.amazonaws	aws-java-sdk-s3	1.12.390
com.amazonaws	aws-java-sdk-ses	1.12.390
com.amazonaws	aws-java-sdk-simpledb	1.12.390
com.amazonaws	aws-java-sdk-simpleworkflow	1.12.390
com.amazonaws	aws-java-sdk-sns	1.12.390
com.amazonaws	aws-java-sdk-sqs	1.12.390
com.amazonaws	aws-java-sdk-ssm	1.12.390
com.amazonaws	aws-java-sdk-storagegateway	1.12.390
com.amazonaws	aws-java-sdk-sts	1.12.390
com.amazonaws	aws-java-sdk-support	1.12.390
com.amazonaws	aws-java-sdk-swf-libraries	1.11.22
com.amazonaws	aws-java-sdk-workspaces	1.12.390
com.amazonaws	jmespath-java	1.12.390
com.clearspring.analytics	stream	2.9.6
com.databricks	Rserve	1.8-3
com.databricks	databricks-sdk-java	0.2.0
com.databricks	jets3t	0.7.1-0
com.databricks.scalapb	compilerplugin_2.12	0.4.15-10
com.databricks.scalapb	scalapb-runtime_2.12	0.4.15-10
com.esotericsoftware	kryo-shaded	4.0.2
com.esotericsoftware	minlog	1.3.0
com.fasterxml	classmate	1.3.4
com.fasterxml.jackson.core	jackson-annotations	2.14.2
com.fasterxml.jackson.core	jackson-core	2.14.2
com.fasterxml.jackson.core	jackson-databind	2.14.2
com.fasterxml.jackson.dataformat	jackson-dataformat-cbor	2.14.2
com.fasterxml.jackson.datatype	jackson-datatype-joda	2.14.2
com.fasterxml.jackson.datatype	jackson-datatype-jsr310	2.13.4
com.fasterxml.jackson.module	jackson-module-paranamer	2.14.2
com.fasterxml.jackson.module	jackson-module-scala_2.12	2.14.2
com.github.ben-manes.caffeine	caffeine	2.9.3
com.github.fommil	jniloader	1.1
com.github.fommil.netlib	native_ref-java	1.1
com.github.fommil.netlib	native_ref-java	1.1-natives
com.github.fommil.netlib	native_system-java	1.1
com.github.fommil.netlib	native_system-java	1.1-natives
com.github.fommil.netlib	netlib-native_ref-linux-x86_64	1.1-natives
com.github.fommil.netlib	netlib-native_system-linux-x86_64	1.1-natives
com.github.luben	zstd-jni	1.5.2-5
com.github.wendykierp	JTransforms	3.1
com.google.code.findbugs	jsr305	3.0.0
com.google.code.gson	gson	2.8.9
com.google.crypto.tink	tink	1.7.0
com.google.errorprone	error_prone_annotations	2.10.0
com.google.flatbuffers	flatbuffers-java	1.12.0
com.google.guava	guava	15.0
com.google.protobuf	protobuf-java	2.6.1
com.h2database	h2	2.1.214
com.helger	profiler	1.1.1
com.jcraft	jsch	0.1.55
com.jolbox	bonecp	0.8.0.RELEASE
com.lihaoyi	sourcecode_2.12	0.1.9
com.microsoft.azure	azure-data-lake-store-sdk	2.3.9
com.microsoft.sqlserver	mssql-jdbc	11.2.2.jre8
com.ning	compress-lzf	1.1.2
com.sun.mail	javax.mail	1.5.2
com.sun.xml.bind	jaxb-core	2.2.11
com.sun.xml.bind	jaxb-impl	2.2.11
com.tdunning	json	1.8
com.thoughtworks.paranamer	paranamer	2.8
com.trueaccord.lenses	lenses_2.12	0.4.12
com.twitter	chill-java	0.10.0
com.twitter	chill_2.12	0.10.0
com.twitter	util-app_2.12	7.1.0
com.twitter	util-core_2.12	7.1.0
com.twitter	util-function_2.12	7.1.0
com.twitter	util-jvm_2.12	7.1.0
com.twitter	util-lint_2.12	7.1.0
com.twitter	util-registry_2.12	7.1.0
com.twitter	util-stats_2.12	7.1.0
com.typesafe	config	1.2.1
com.typesafe.scala-logging	scala-logging_2.12	3.7.2
com.uber	h3	3.7.0
com.univocity	univocity-parsers	2.9.1
com.zaxxer	HikariCP	4.0.3
commons-cli	commons-cli	1.5.0
commons-codec	commons-codec	1.15
commons-collections	commons-collections	3.2.2
commons-dbcp	commons-dbcp	1.4
commons-fileupload	commons-fileupload	1.5
commons-httpclient	commons-httpclient	3.1
commons-io	commons-io	2.11.0
commons-lang	commons-lang	2.6
commons-logging	commons-logging	1.1.3
commons-pool	commons-pool	1.5.4
dev.ludovic.netlib	arpack	3.0.3
dev.ludovic.netlib	blas	3.0.3
dev.ludovic.netlib	lapack	3.0.3
info.ganglia.gmetric4j	gmetric4j	1.0.10
io.airlift	aircompressor	0.21
io.delta	delta-sharing-spark_2.12	0.7.1
io.dropwizard.metrics	metrics-core	4.2.10
io.dropwizard.metrics	metrics-graphite	4.2.10
io.dropwizard.metrics	metrics-healthchecks	4.2.10
io.dropwizard.metrics	metrics-jetty9	4.2.10
io.dropwizard.metrics	metrics-jmx	4.2.10
io.dropwizard.metrics	metrics-json	4.2.10
io.dropwizard.metrics	metrics-jvm	4.2.10
io.dropwizard.metrics	metrics-servlets	4.2.10
io.netty	netty-all	4.1.87.Final
io.netty	netty-buffer	4.1.87.Final
io.netty	netty-codec	4.1.87.Final
io.netty	netty-codec-http	4.1.87.Final
io.netty	netty-codec-http2	4.1.87.Final
io.netty	netty-codec-socks	4.1.87.Final
io.netty	netty-common	4.1.87.Final
io.netty	netty-handler	4.1.87.Final
io.netty	netty-handler-proxy	4.1.87.Final
io.netty	netty-resolver	4.1.87.Final
io.netty	netty-transport	4.1.87.Final
io.netty	netty-transport-classes-epoll	4.1.87.Final
io.netty	netty-transport-classes-kqueue	4.1.87.Final
io.netty	netty-transport-native-epoll	4.1.87.Final
io.netty	netty-transport-native-epoll	4.1.87.Final-linux-aarch_64
io.netty	netty-transport-native-epoll	4.1.87.Final-linux-x86_64
io.netty	netty-transport-native-kqueue	4.1.87.Final-osx-aarch_64
io.netty	netty-transport-native-kqueue	4.1.87.Final-osx-x86_64
io.netty	netty-transport-native-unix-common	4.1.87.Final
io.prometheus	simpleclient	0.7.0
io.prometheus	simpleclient_common	0.7.0
io.prometheus	simpleclient_dropwizard	0.7.0
io.prometheus	simpleclient_pushgateway	0.7.0
io.prometheus	simpleclient_servlet	0.7.0
io.prometheus.jmx	collector	0.12.0
jakarta.annotation	jakarta.annotation-api	1.3.5
jakarta.servlet	jakarta.servlet-api	4.0.3
jakarta.validation	jakarta.validation-api	2.0.2
jakarta.ws.rs	jakarta.ws.rs-api	2.1.6
javax.activation	activation	1.1.1
javax.el	javax.el-api	2.2.4
javax.jdo	jdo-api	3.0.1
javax.transaction	jta	1.1
javax.transaction	transaction-api	1.1
javax.xml.bind	jaxb-api	2.2.11
javolution	javolution	5.5.1
jline	jline	2.14.6
joda-time	joda-time	2.12.1
ml.combust.mleap	mleap-databricks-runtime_2.12	v0.20.0-db2
net.java.dev.jna	jna	5.8.0
net.razorvine	pickle	1.3
net.sf.jpam	jpam	1.1
net.sf.opencsv	opencsv	2.3
net.sf.supercsv	super-csv	2.2.0
net.snowflake	snowflake-ingest-sdk	0.9.6
net.snowflake	snowflake-jdbc	3.13.33
net.sourceforge.f2j	arpack_combined_all	0.1
org.acplt.remotetea	remotetea-oncrpc	1.1.2
org.antlr	ST4	4.0.4
org.antlr	antlr-runtime	3.5.2
org.antlr	antlr4-runtime	4.9.3
org.antlr	stringtemplate	3.2.1
org.apache.ant	ant	1.9.16
org.apache.ant	ant-jsch	1.9.16
org.apache.ant	ant-launcher	1.9.16
org.apache.arrow	arrow-format	11.0.0
org.apache.arrow	arrow-memory-core	11.0.0
org.apache.arrow	arrow-memory-netty	11.0.0
org.apache.arrow	arrow-vector	11.0.0
org.apache.avro	avro	1.11.1
org.apache.avro	avro-ipc	1.11.1
org.apache.avro	avro-mapred	1.11.1
org.apache.commons	commons-collections4	4.4
org.apache.commons	commons-compress	1.21
org.apache.commons	commons-crypto	1.1.0
org.apache.commons	commons-lang3	3.12.0
org.apache.commons	commons-math3	3.6.1
org.apache.commons	commons-text	1.10.0
org.apache.curator	curator-client	2.13.0
org.apache.curator	curator-framework	2.13.0
org.apache.curator	curator-recipes	2.13.0
org.apache.datasketches	datasketches-java	3.1.0
org.apache.datasketches	datasketches-memory	2.0.0
org.apache.derby	derby	10.14.2.0
org.apache.hadoop	hadoop-client-runtime	3.3.4
org.apache.hive	hive-beeline	2.3.9
org.apache.hive	hive-cli	2.3.9
org.apache.hive	hive-jdbc	2.3.9
org.apache.hive	hive-llap-client	2.3.9
org.apache.hive	hive-llap-common	2.3.9
org.apache.hive	hive-serde	2.3.9
org.apache.hive	hive-shims	2.3.9
org.apache.hive	hive-storage-api	2.8.1
org.apache.hive.shims	hive-shims-0.23	2.3.9
org.apache.hive.shims	hive-shims-common	2.3.9
org.apache.hive.shims	hive-shims-scheduler	2.3.9
org.apache.httpcomponents	httpclient	4.5.14
org.apache.httpcomponents	httpcore	4.4.16
org.apache.ivy	ivy	2.5.1
org.apache.logging.log4j	log4j-1.2-api	2.19.0
org.apache.logging.log4j	log4j-api	2.19.0
org.apache.logging.log4j	log4j-core	2.19.0
org.apache.logging.log4j	log4j-slf4j2-impl	2.19.0
org.apache.mesos	mesos	1.11.0-shaded-protobuf
org.apache.orc	orc-core	1.8.4-shaded-protobuf
org.apache.orc	orc-mapreduce	1.8.4-shaded-protobuf
org.apache.orc	orc-shims	1.8.4
org.apache.thrift	libfb303	0.9.3
org.apache.thrift	libthrift	0.12.0
org.apache.xbean	xbean-asm9-shaded	4.22
org.apache.yetus	audience-annotations	0.13.0
org.apache.zookeeper	zookeeper	3.6.3
org.apache.zookeeper	zookeeper-jute	3.6.3
org.checkerframework	checker-qual	3.19.0
org.codehaus.jackson	jackson-core-asl	1.9.13
org.codehaus.jackson	jackson-mapper-asl	1.9.13
org.codehaus.janino	commons-compiler	3.0.16
org.codehaus.janino	janino	3.0.16
org.datanucleus	datanucleus-api-jdo	4.2.4
org.datanucleus	datanucleus-core	4.1.17
org.datanucleus	datanucleus-rdbms	4.1.19
org.datanucleus	javax.jdo	3.2.0-m3
org.eclipse.jetty	jetty-client	9.4.51.v20230217
org.eclipse.jetty	jetty-continuation	9.4.51.v20230217
org.eclipse.jetty	jetty-http	9.4.51.v20230217
org.eclipse.jetty	jetty-io	9.4.51.v20230217
org.eclipse.jetty	jetty-jndi	9.4.51.v20230217
org.eclipse.jetty	jetty-plus	9.4.51.v20230217
org.eclipse.jetty	jetty-proxy	9.4.51.v20230217
org.eclipse.jetty	jetty-security	9.4.51.v20230217
org.eclipse.jetty	jetty-server	9.4.51.v20230217
org.eclipse.jetty	jetty-servlet	9.4.51.v20230217
org.eclipse.jetty	jetty-servlets	9.4.51.v20230217
org.eclipse.jetty	jetty-util	9.4.51.v20230217
org.eclipse.jetty	jetty-util-ajax	9.4.51.v20230217
org.eclipse.jetty	jetty-webapp	9.4.51.v20230217
org.eclipse.jetty	jetty-xml	9.4.51.v20230217
org.eclipse.jetty.websocket	websocket-api	9.4.51.v20230217
org.eclipse.jetty.websocket	websocket-client	9.4.51.v20230217
org.eclipse.jetty.websocket	websocket-common	9.4.51.v20230217
org.eclipse.jetty.websocket	websocket-server	9.4.51.v20230217
org.eclipse.jetty.websocket	websocket-servlet	9.4.51.v20230217
org.fusesource.leveldbjni	leveldbjni-all	1.8
org.glassfish.hk2	hk2-api	2.6.1
org.glassfish.hk2	hk2-locator	2.6.1
org.glassfish.hk2	hk2-utils	2.6.1
org.glassfish.hk2	osgi-resource-locator	1.0.3
org.glassfish.hk2.external	aopalliance-repackaged	2.6.1
org.glassfish.hk2.external	jakarta.inject	2.6.1
org.glassfish.jersey.containers	jersey-container-servlet	2.36
org.glassfish.jersey.containers	jersey-container-servlet-core	2.36
org.glassfish.jersey.core	jersey-client	2.36
org.glassfish.jersey.core	jersey-common	2.36
org.glassfish.jersey.core	jersey-server	2.36
org.glassfish.jersey.inject	jersey-hk2	2.36
org.hibernate.validator	hibernate-validator	6.1.7.Final
org.ini4j	ini4j	0.5.4
org.javassist	javassist	3.25.0-GA
org.jboss.logging	jboss-logging	3.3.2.Final
org.jdbi	jdbi	2.63.1
org.jetbrains	annotations	17.0.0
org.joda	joda-convert	1.7
org.jodd	jodd-core	3.5.2
org.json4s	json4s-ast_2.12	3.7.0-M11
org.json4s	json4s-core_2.12	3.7.0-M11
org.json4s	json4s-jackson_2.12	3.7.0-M11
org.json4s	json4s-scalap_2.12	3.7.0-M11
org.lz4	lz4-java	1.8.0
org.mariadb.jdbc	mariadb-java-client	2.7.4
org.mlflow	mlflow-spark	2.2.0
org.objenesis	objenesis	2.5.1
org.postgresql	postgresql	42.3.8
org.roaringbitmap	RoaringBitmap	0.9.39
org.roaringbitmap	shims	0.9.39
org.rocksdb	rocksdbjni	7.8.3
org.rosuda.REngine	REngine	2.1.0
org.scala-lang	scala-compiler_2.12	2.12.15
org.scala-lang	scala-library_2.12	2.12.15
org.scala-lang	scala-reflect_2.12	2.12.15
org.scala-lang.modules	scala-collection-compat_2.12	2.4.3
org.scala-lang.modules	scala-parser-combinators_2.12	1.1.2
org.scala-lang.modules	scala-xml_2.12	1.2.0
org.scala-sbt	test-interface	1.0
org.scalacheck	scalacheck_2.12	1.14.2
org.scalactic	scalactic_2.12	3.2.15
org.scalanlp	breeze-macros_2.12	2.1.0
org.scalanlp	breeze_2.12	2.1.0
org.scalatest	scalatest-compatible	3.2.15
org.scalatest	scalatest-core_2.12	3.2.15
org.scalatest	scalatest-diagrams_2.12	3.2.15
org.scalatest	scalatest-featurespec_2.12	3.2.15
org.scalatest	scalatest-flatspec_2.12	3.2.15
org.scalatest	scalatest-freespec_2.12	3.2.15
org.scalatest	scalatest-funspec_2.12	3.2.15
org.scalatest	scalatest-funsuite_2.12	3.2.15
org.scalatest	scalatest-matchers-core_2.12	3.2.15
org.scalatest	scalatest-mustmatchers_2.12	3.2.15
org.scalatest	scalatest-propspec_2.12	3.2.15
org.scalatest	scalatest-refspec_2.12	3.2.15
org.scalatest	scalatest-shouldmatchers_2.12	3.2.15
org.scalatest	scalatest-wordspec_2.12	3.2.15
org.scalatest	scalatest_2.12	3.2.15
org.slf4j	jcl-over-slf4j	2.0.6
org.slf4j	jul-to-slf4j	2.0.6
org.slf4j	slf4j-api	2.0.6
org.threeten	threeten-extra	1.7.1
org.tukaani	xz	1.9
org.typelevel	algebra_2.12	2.0.1
org.typelevel	cats-kernel_2.12	2.1.1
org.typelevel	spire-macros_2.12	0.17.0
org.typelevel	spire-platform_2.12	0.17.0
org.typelevel	spire-util_2.12	0.17.0
org.typelevel	spire_2.12	0.17.0
org.wildfly.openssl	wildfly-openssl	1.1.3.Final
org.xerial	sqlite-jdbc	3.42.0.0
org.xerial.snappy	snappy-java	1.1.10.1
org.yaml	snakeyaml	1.33
oro	oro	2.0.8
pl.edu.icm	JLargeArrays	1.5
software.amazon.cryptools	AmazonCorrettoCryptoProvider	1.6.1-linux-x86_64
software.amazon.ion	ion-java	1.0.2
stax	stax-api	1.0.1

Share via

Databricks Runtime 13.3 LTS

New features and improvements

Scala support for shared clusters

Allowlist for init scripts, JARs, and Maven coordinates on Unity Catalog shared clusters is in Public Preview

Volumes support for init scripts and JARs is in Public Preview

Safe casts enabled by default for Delta Lake operations

Introducing tags with Unity Catalog

Databricks Runtime returns an error if a file is modified between query planning and invocation

Block schema overwrite when using dynamic partition overwrites

Integrate copyFile API into dbutils.fs.cp

Databricks ODBC/JDBC driver support

Support for unnamed (positional) parameter markers

Enable time series column labeling

New bitmap SQL functions

Improved encryption functions

Unity Catalog support for REFRESH FOREIGN

INSERT BY NAME

Breaking changes

Changes to implicit casting when storing rows in a table with MERGE INTO and UPDATE

Parquet Schema Inference Changes

Library upgrades

Apache Spark

Databricks ODBC/JDBC driver support

System environment

Installed Python libraries

Installed R libraries

Installed Java and Scala libraries (Scala 2.12 cluster version)

Feedback

Additional resources

Share via

Databricks Runtime 13.3 LTS

New features and improvements

Scala support for shared clusters

Allowlist for init scripts, JARs, and Maven coordinates on Unity Catalog shared clusters is in Public Preview

Volumes support for init scripts and JARs is in Public Preview

Safe casts enabled by default for Delta Lake operations

Introducing tags with Unity Catalog

Databricks Runtime returns an error if a file is modified between query planning and invocation

Block schema overwrite when using dynamic partition overwrites

Integrate copyFile API into dbutils.fs.cp

Databricks ODBC/JDBC driver support

Support for unnamed (positional) parameter markers

Enable time series column labeling

New bitmap SQL functions

Improved encryption functions

Unity Catalog support for REFRESH FOREIGN

INSERT BY NAME

Share materialized views with Delta Sharing

Breaking changes

Changes to implicit casting when storing rows in a table with MERGE INTO and UPDATE

Parquet Schema Inference Changes

Library upgrades

Apache Spark

Databricks ODBC/JDBC driver support

System environment

Installed Python libraries

Installed R libraries

Installed Java and Scala libraries (Scala 2.12 cluster version)

Feedback

Additional resources