Databricks Runtime 10.3 (EoS)
Note
Support for this Databricks Runtime version has ended. For the end-of-support date, see End-of-support history. For all supported Databricks Runtime versions, see Databricks Runtime release notes versions and compatibility.
The following release notes provide information about Databricks Runtime 10.3 and Databricks Runtime 10.3 Photon, powered by Apache Spark 3.2.1. Photon is in Public Preview. Databricks released this version in February 2022.
New features and improvements
- Parquet and Avro writers now use the session time zone when rebasing timestamps in Parquet data sources
- New functions in Spark SQL
- Low shuffle merge is now generally available
- New COPY INTO features: validation mode and schema evolution
- Identity columns are now supported in Delta tables (Public Preview)
Parquet and Avro writers now use the session time zone when rebasing timestamps in Parquet data sources
Parquet and Avro writers now use the session time zone while timestamp rebasing in legacy mode instead of using the default JVM time zone. The session time zone is set to the JVM time zone by default.
New functions in Spark SQL
The following Spark SQL functions are now supported in this release:
- aes_decrypt: decrypts input by using AES.
- aes_encrypt: encrypts input by using AES.
- contains: determines whether an expression contains another expression.
- endswith: determines whether an expression ends with another expression.
- percentile_cont: returns a percentile value.
- startswith: determines whether an expression starts with another expression.
Low shuffle merge is now generally available
The Delta MERGE INTO command uses a technique called low shuffle merge, which reduces shuffling of unmodified rows. This improves performance of the command and helps to preserve existing clustering on the table, such as Z-ordering. This functionality was introduced in Databricks Runtime 9.0 for Public Preview and has now reached general availability. To enable low shuffle merge, set spark.databricks.delta.merge.enableLowShuffle to true. See Low shuffle merge on Azure Databricks.
New COPY INTO features: validation mode and schema evolution
Validation mode: COPY INTO now supports a VALIDATE mode where the data that is to be loaded into a table is validated but not written to the table. These validations include:
- Whether the data can be parsed.
- Whether the schema matches that of the table or if the schema needs to be evolved.
- Whether all nullability and check constraints are met.
Example usage:
COPY INTO <table>
FROM <path-to-files>
FILEFORMAT = <format>
VALIDATE [ALL | <num-rows> ROWS]
The default is to validate all of the data that is to be loaded. You can provide a number of rows to be validated with the ROWS keyword, such as VALIDATE 15 ROWS. The COPY INTO statement returns a preview of the data of 50 rows or less, when a number of less than 50 is used with the ROWS keyword).
Schema evolution: COPY INTO now supports schema evolution with the mergeSchema option in COPY_OPTIONS. To evolve the schema of a table, you must have OWN permissions on the table.
Example usage:
COPY INTO <table>
FROM <path-to-files>
FILEFORMAT = <format>
COPY_OPTIONS ('mergeSchema' = 'true')
See COPY INTO.
Identity columns are now supported in Delta tables (Public Preview)
Delta Lake now supports identity columns. When you write to a Delta table that defines an identity column, and you do not provide values for that column, Delta now automatically assigns a unique and statistically increasing or decreasing value. See CREATE TABLE [USING].
Library upgrades
- Upgraded Python libraries:
- distlib from 0.3.3 to 0.3.4
- filelock from 3.4.0 to 3.4.2
- numpy from 1.19.2 to 1.20.1
- plotly from 5.3.1 to 5.5.0
- Upgraded R libraries:
- backports from 1.4.0 to 1.4.1
- brio from 1.1.2 to 1.1.3
- broom from 0.7.10 to 0.7.11
- class from 7.3-19 to 7.3-20
- cpp11 from 0.4.1 to 0.4.2
- DBI from 1.1.1 to 1.1.2
- devtools from 2.4.2 to 2.4.3
- digest from 0.6.28 to 0.6.29
- dtplyr from 1.1.0 to 1.2.0
- fansi from 0.5.0 to 1.0.2
- foreign from 0.8-81 to 0.8-82
- fs from 1.5.0 to 1.5.2
- gert from 1.4.3 to 1.5.0
- glue from 1.5.0 to 1.6.0
- httpuv from 1.6.3 to 1.6.5
- jsonlite from 1.7.2 to 1.7.3
- knitr from 1.36 to 1.37
- MASS from 7.3-54 to 7.3-55
- Matrix from 1.3-4 to 1.4-0
- nlme from 3.1-152 to 3.1-155
- nnet frm 7.3-16 to 7.3-17
- openssl from 1.4.5 to 1.4.6
- parallelly from 1.29.0 to 1.30.0
- pkgbuild from 1.2.0 to 1.3.1
- pkgload from 1.2.3 to 1.2.4
- progressr from 0.9.0 to 0.10.0
- Rcpp from 1.0.7 to 1.0.8
- readr from 2.1.0 to 2.1.1
- remotes from 2.4.1 to 2.4.2
- RSQLite from 2.2.8 to 2.2.9
- sessioninfo from 1.2.1 to 1.2.2
- sparklyr from 1.7.2 to 1.7.4
- testthat from 3.1.0 to 3.1.1
- tinytex from 0.35 to 0.36
- usethis from 2.1.3 to 2.1.5
- vroom from 1.5.6 to 1.5.7
- withr from 2.4.2 to 2.4.3
- xfun from 0.28 to 0.29
- xml2 from 1.3.2 to 1.3.3
- Upgraded Java libraries:
- dev.ludovic.netlib.arpack from 1.3.2 to 2.2.1
- dev.ludovic.netlib.blas from 1.3.2 to 2.2.1
- dev.ludovic.netlib.lapack from 1.3.2 to 2.2.1
- io.delta.delta-sharing-spark_2.12 from 0.2.0 to 0.3.0
Apache Spark
Databricks Runtime 10.3 includes Apache Spark 3.2.1. This release includes all Spark fixes and improvements included in Databricks Runtime 10.2 (EoS), as well as the following additional bug fixes and improvements made to Spark:
- [SPARK-37896] [SQL] Implement a ConstantColumnVector and improve performance of the hidden file metadata
- [SPARK-37924] [SQL] Sort table properties by key in SHOW CREATE TABLE on VIEW (v1)
- [SPARK-37636] [SQL] Migrate CREATE NAMESPACE to use V2 command by default
- [SPARK-37381] [SQL] Unify v1 and v2 SHOW CREATE TABLE tests
- [SPARK-37876] [CORE][SQL] Move
SpecificParquetRecordReaderBase.listDirectorytoTestUtils - [SPARK-37854] [CORE] Replace type check with pattern matching in Spark code
- [SPARK-36967] [CORE] Report accurate shuffle block size if its skewed
- [SPARK-37864] [SQL] Support vectorized read boolean values use RLE encoding with Parquet DataPage V2
- [SPARK-37862] [SQL] RecordBinaryComparator should fast skip the check of aligning with unaligned platform
- [SPARK-37855] [SQL] IllegalStateException when transforming an array inside a nested struct
- [SPARK-37858] [SQL] Throw
SparkRuntimeExceptionwith error classes from AES functions - [SPARK-37841] [SQL] BasicWriteTaskStatsTracker should not try get status for a skipped file
- [SPARK-37860] [UI] Fix taskindex in the stage page task event timeline
- [SPARK-36879] [SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path
- [SPARK-37828] [SQL] Push down filters through RebalancePartitions
- [SPARK-37802] [SQL][3.2] Composite field name should work with Aggregate push down
- [SPARK-37826] [SQL] Use zstd codec name in ORC file names for hive orc impl
- [SPARK-37820] [SQL] Replace ApacheCommonBase64 with JavaBase64 for string funcs
- [SPARK-37832] [SQL] Orc struct converter should use an array to look up field converters rather than a linked list
- [SPARK-37786] [SQL] StreamingQueryListener support use SQLConf.get to get corresponding SessionState’s SQLConf
- [SPARK-37527] [SQL] Translate more standard aggregate functions for pushdown
- [SPARK-37777] [SQL] Update the SQL syntax of SHOW FUNCTIONS
- [SPARK-8582] [CORE] Checkpoint eagerly when asked to do so for real
- [SPARK-37822] [SQL] StringSplit should return an array on non-null elements
- [SPARK-37812] [SQL] When deserializing an Orc struct, reuse the result row when possible
- [SPARK-37367] [SQL] Reenable exception test in DDLParserSuite.create view – basic
- [SPARK-37796] [SQL] ByteArrayMethods arrayEquals should fast skip the check of aligning with unaligned platform
- [SPARK-37817] [SQL][Minor] Remove unreachable code in complexTypeExtractors.scala
- [SPARK-37807] [SQL] Fix a typo in HttpAuthenticationException message
- [SPARK-37750] [SQL] ANSI mode: optionally return null result if element not exists in array/map
- [SPARK-37789] [SQL] Add a class to represent general aggregate functions in DS V2
- [SPARK-37785] [SQL][CORE] Add Utils.isInRunningSparkTask
- [SPARK-37780] [SQL] QueryExecutionListener support use SQLConf.get to get corresponding SessionState’s SQLConf
- [SPARK-37783] [SS][SQL][CORE] Enable tail-recursion wherever possible
- [SPARK-37726] [SQL] Add spill size metrics for sort merge join
- [SPARK-37800] [SQL] TreeNode.argString incorrectly formats arguments of type Set[_]
- [SPARK-37716] [SQL] Improve error messages when a LateralJoin has non-deterministic expressions
- [SPARK-37779] [SQL] Make ColumnarToRowExec plan canonicalizable after (de)serialization
- [SPARK-37578] [SQL] Update task metrics from ds v2 custom metrics
- [SPARK-37676] [SQL] Support ANSI Aggregation Function: percentile_cont
- [SPARK-37724] [SQL] ANSI mode: disable ANSI reserved keywords by default
- [SPARK-37705] [SQL][3.2] Rebase timestamps in the session time zone saved in Parquet/Avro metadata
- [SPARK-34755] [SQL] Support the utils for transform number format
- [SPARK-36703] [SQL] Remove the Sort if it is the child of RepartitionByExpression
- [SPARK-37369] [SQL] Avoid redundant ColumnarToRow transition on InMemoryTableScan
- [SPARK-37450] [SQL] Prune unnecessary fields from Generate
- [SPARK-37728] [SQL] Reading nested columns with ORC vectorized reader can cause ArrayIndexOutOfBoundsException
- [SPARK-37564] [SQL] Add code-gen for sort aggregate without grouping keys
- [SPARK-37659] [UI][3.2] Fix FsHistoryProvider race condition between list and delet log info
- [SPARK-37727] [SQL] Show ignored confs & hide warnings for conf already set in SparkSession.builder.getOrCreate
- [SPARK-37391] [SQL]JdbcConnectionProvider tells if it modifies security context
- [SPARK-37273] [SQL] Support hidden file metadata columns in Spark SQL
- [SPARK-36876] [SQL] Support Dynamic Partition pruning for HiveTableScanExec
- [SPARK-37546] [SQL] V2 ReplaceTableAsSelect command should qualify location
- [SPARK-37545] [SQL] V2 CreateTableAsSelect command should qualify location
- [SPARK-37707] [SQL] Allow store assignment and implicit cast among datetime types
- [SPARK-37644] [SQL] Support datasource v2 complete aggregate pushdown
- [SPARK-37695] [CORE][SHUFFLE] Skip diagnosis ob merged blocks from push-based shuffle
- [SPARK-35295] [ML] Replace fully com.github.fommil.netlib by dev.ludovic.netlib:2.0
- [SPARK-37694] [SQL] Disallow delete resources in spark sql cli
- [SPARK-37330] [SQL] Migrate ReplaceTableStatement to v2 command
- [SPARK-27974] [SQL] Support ANSI Aggregate Function: array_agg
- [SPARK-36902] [SQL] Migrate CreateTableAsSelectStatement to v2 command
- [SPARK-37678] [PYTHON] Fix _cleanup_and_return signature
- [SPARK-37613] [SQL] Support ANSI Aggregate Function: regr_count
- [SPARK-37689] [SQL] Expand should be supported in PropagateEmptyRelation
- [SPARK-37654] [SQL] Fix NPE in Row.getSeq when field is Null
- [SPARK-37496] [SQL] Migrate ReplaceTableAsSelectStatement to v2 command
- [SPARK-37483] [SQL] Support push down top N to JDBC data source V2
- [SPARK-37577] [SQL] Fix ClassCastException: ArrayType cannot be cast to StructType for Generate Pruning
- [SPARK-37310] [SQL] Migrate ALTER NAMESPACE … SET PROPERTIES to use V2 command by default
- [SPARK-37501] [SQL] CREATE/REPLACE TABLE should qualify location for v2 command
- [SPARK-37343] [SQL] Implement createIndex, IndexExists and dropIndex in JDBC (Postgres dialect)
- [SPARK-37060] [CORE][3.2] Handle driver status response from backup masters
- [SPARK-37629] [SQL] Speed up Expression.canonicalized
- [SPARK-37633] [SQL] Unwrap cast should skip if downcast failed with an…
- [SPARK-37631] [SQL] Code clean up on promoting strings to double type in math functions
- [SPARK-37569] [SQL] Don’t mark nested view fields as nullable
- [SPARK-37557] [SQL] Replace object hash with sort aggregate if child is already sorted
- [SPARK-36850] [SQL] Migrate CreateTableStatement to v2 command framework
- [SPARK-37592] [SQL] Improve performance of
JoinSelection - [SPARK-37217] [SQL] The number of dynamic partitions should early check when writing to external tables
- [SPARK-37584] [SQL] New SQL function: map_contains_key
- [SPARK-37455] [SQL] Replace hash with sort aggregate if child is already sorted
- [SPARK-34735] [SQL][UI] Add modified configs for SQL execution in UI
- [SPARK-37540] [SQL] Detect more unsupported time travel
- [SPARK-37376] [SQL] Introduce a new DataSource V2 interface HasPartitionKey
- [SPARK-37666] [SQL] Set
GCMas the default mode inaes_encrypt()/aes_decrypt() - [SPARK-37573] [SQL] IsolatedClientLoader should fallback to related version, not always hadoop-2.7.4
- [SPARK-37561] [SQL] Avoid loading all functions when obtaining hive’s DelegationToken
- [SPARK-37370] [SQL] Add SQL configs to control newly added join code-gen in 3.3
- [SPARK-37311] [SQL] Migrate ALTER NAMESPACE … SET LOCATION to use V2 command by default
- [SPARK-37192] [SQL] Migrate SHOW TBLPROPERTIES to use V2 command by default
- [SPARK-37219] [SQL] Add AS OF syntax support
- [SPARK-37316] [SQL] Add code-gen for existence sort merge join
- [SPARK-37591] [SQL] Support the GCM mode by
aes_encrypt()/aes_decrypt() - [SPARK-37586] [SQL] Add the
modeandpaddingargs toaes_encrypt()/aes_decrypt() - [SPARK-12567] [SQL] Add aes_encrypt and aes_decrypt builtin functions
- [SPARK-37392] [SQL] Fix the performance bug when inferring constraints for Generate
- [SPARK-37481] [Core][WebUI] Fix disappearance of skipped stages after they retry
- [SPARK-37533] [SQL] New SQL function: try_element_at
- [SPARK-37451] [SQL] Fix cast string type to decimal type if spark.sql.legacy.allowNegativeScaleOfDecimal is enabled
- [SPARK-37062] [SS] Introduce a new data source for providing consistent set of rows per microbatch
- [SPARK-37552] [SQL] Add the
convert_timezone()function - [SPARK-37555] [SQL] spark-sql should pass last unclosed comment to backend
- [SPARK-37286] [SQL] Move compileAggregates from JDBCRDD to JdbcDialect
- [SPARK-37485] [CORE][SQL] Replace
mapwith expressions which produce no result withforeach - [SPARK-37556] [SQL] Deser void class fail with Java serialization
- [SPARK-37520] [SQL] Add the
startswith()andendswith()string functions - [SPARK-37471] [SQL] spark-sql support
;in nested bracketed comment - [SPARK-37360] [SQL] Support TimestampNTZ in JSON data source
- [SPARK-37508] [SQL] Add CONTAINS() string function
- [SPARK-37494] [SQL] Unify v1 and v2 options output of
SHOW CREATE TABLEcommand - [SPARK-37524] [SQL] We should drop all tables after testing dynamic partition pruning.
- [SPARK-37389] [SQL] Check unclosed bracketed comments
- [SPARK-37220] [SQL] Do not split input file for Parquet reader with aggregate push down
- [SPARK-37326] [SQL] Support TimestampNTZ in CSV data source
- [SPARK-35867] [SQL] Enable vectorized read for VectorizedPlainValuesReader.readBooleans
- [SPARK-37484] [CORE][SQL] Replace
getandgetOrElsewithgetOrElse - [SPARK-37492] [SQL] Optimize Orc test code with withAllNativeOrcReaders
- [SPARK-37458] [SS] Remove unnecessary SerializeFromObject from the plan of foreachBatch
Maintenance updates
See Databricks Runtime 10.3 maintenance updates.
System environment
- Operating System: Ubuntu 20.04.3 LTS
- Java: Zulu 8.56.0.21-CA-linux64
- Scala: 2.12.14
- Python: 3.8.10
- R: 4.1.2
- Delta Lake: 1.1.0
Installed Python libraries
| Library | Version | Library | Version | Library | Version |
|---|---|---|---|---|---|
| Antergos Linux | 2015.10 (ISO-Rolling) | appdirs | 1.4.4 | argon2-cffi | 20.1.0 |
| async-generator | 1.10 | attrs | 20.3.0 | backcall | 0.2.0 |
| bidict | 0.21.4 | bleach | 3.3.0 | boto3 | 1.16.7 |
| botocore | 1.19.7 | certifi | 2020.12.5 | cffi | 1.14.5 |
| chardet | 4.0.0 | cycler | 0.10.0 | Cython | 0.29.23 |
| dbus-python | 1.2.16 | decorator | 5.0.6 | defusedxml | 0.7.1 |
| distlib | 0.3.4 | distro-info | 0.23ubuntu1 | entrypoints | 0.3 |
| facets-overview | 1.0.0 | filelock | 3.4.2 | idna | 2.10 |
| ipykernel | 5.3.4 | ipython | 7.22.0 | ipython-genutils | 0.2.0 |
| ipywidgets | 7.6.3 | jedi | 0.17.2 | Jinja2 | 2.11.3 |
| jmespath | 0.10.0 | joblib | 1.0.1 | jsonschema | 3.2.0 |
| jupyter-client | 6.1.12 | jupyter-core | 4.7.1 | jupyterlab-pygments | 0.1.2 |
| jupyterlab-widgets | 1.0.0 | kiwisolver | 1.3.1 | koalas | 1.8.2 |
| MarkupSafe | 2.0.1 | matplotlib | 3.4.2 | mistune | 0.8.4 |
| nbclient | 0.5.3 | nbconvert | 6.0.7 | nbformat | 5.1.3 |
| nest-asyncio | 1.5.1 | notebook | 6.3.0 | numpy | 1.20.1 |
| packaging | 20.9 | pandas | 1.2.4 | pandocfilters | 1.4.3 |
| parso | 0.7.0 | patsy | 0.5.1 | pexpect | 4.8.0 |
| pickleshare | 0.7.5 | Pillow | 8.2.0 | pip | 21.0.1 |
| plotly | 5.5.0 | prometheus-client | 0.10.1 | prompt-toolkit | 3.0.17 |
| protobuf | 3.17.2 | psycopg2 | 2.8.5 | ptyprocess | 0.7.0 |
| pyarrow | 4.0.0 | pycparser | 2.20 | Pygments | 2.8.1 |
| PyGObject | 3.36.0 | pyparsing | 2.4.7 | pyrsistent | 0.17.3 |
| python-apt | 2.0.0+ubuntu0.20.4.6 | python-dateutil | 2.8.1 | python-engineio | 4.3.0 |
| python-socketio | 5.4.1 | pytz | 2020.5 | pyzmq | 20.0.0 |
| requests | 2.25.1 | requests-unixsocket | 0.2.0 | s3transfer | 0.3.7 |
| scikit-learn | 0.24.1 | scipy | 1.6.2 | seaborn | 0.11.1 |
| Send2Trash | 1.5.0 | setuptools | 52.0.0 | six | 1.15.0 |
| ssh-import-id | 5.10 | statsmodels | 0.12.2 | tenacity | 8.0.1 |
| terminado | 0.9.4 | testpath | 0.4.4 | threadpoolctl | 2.1.0 |
| tornado | 6.1 | traitlets | 5.0.5 | unattended-upgrades | 0.1 |
| urllib3 | 1.25.11 | virtualenv | 20.4.1 | wcwidth | 0.2.5 |
| webencodings | 0.5.1 | wheel | 0.36.2 | widgetsnbextension | 3.5.1 |
Installed R libraries
R libraries are installed from the Microsoft CRAN snapshot on 2022-01-19.
| Library | Version | Library | Version | Library | Version |
|---|---|---|---|---|---|
| askpass | 1.1 | assertthat | 0.2.1 | backports | 1.4.1 |
| base | 4.1.2 | base64enc | 0.1-3 | bit | 4.0.4 |
| bit64 | 4.0.5 | blob | 1.2.2 | boot | 1.3-28 |
| brew | 1.0-6 | brio | 1.1.3 | broom | 0.7.11 |
| bslib | 0.3.1 | cachem | 1.0.6 | callr | 3.7.0 |
| caret | 6.0-90 | cellranger | 1.1.0 | chron | 2.3-56 |
| class | 7.3-20 | cli | 3.1.0 | clipr | 0.7.1 |
| cluster | 2.1.2 | codetools | 0.2-18 | colorspace | 2.0-2 |
| commonmark | 1.7 | compiler | 4.1.2 | config | 0.3.1 |
| cpp11 | 0.4.2 | crayon | 1.4.2 | credentials | 1.3.2 |
| curl | 4.3.2 | data.table | 1.14.2 | datasets | 4.1.2 |
| DBI | 1.1.2 | dbplyr | 2.1.1 | desc | 1.4.0 |
| devtools | 2.4.3 | diffobj | 0.3.5 | digest | 0.6.29 |
| dplyr | 1.0.7 | dtplyr | 1.2.0 | e1071 | 1.7-9 |
| ellipsis | 0.3.2 | evaluate | 0.14 | fansi | 1.0.2 |
| farver | 2.1.0 | fastmap | 1.1.0 | fontawesome | 0.2.2 |
| forcats | 0.5.1 | foreach | 1.5.1 | foreign | 0.8-82 |
| forge | 0.2.0 | fs | 1.5.2 | future | 1.23.0 |
| future.apply | 1.8.1 | gargle | 1.2.0 | generics | 0.1.1 |
| gert | 1.5.0 | ggplot2 | 3.3.5 | gh | 1.3.0 |
| gitcreds | 0.1.1 | glmnet | 4.1-3 | globals | 0.14.0 |
| glue | 1.6.0 | googledrive | 2.0.0 | googlesheets4 | 1.0.0 |
| gower | 0.2.2 | graphics | 4.1.2 | grDevices | 4.1.2 |
| grid | 4.1.2 | gridExtra | 2.3 | gsubfn | 0.7 |
| gtable | 0.3.0 | haven | 2.4.3 | highr | 0.9 |
| hms | 1.1.1 | htmltools | 0.5.2 | htmlwidgets | 1.5.4 |
| httpuv | 1.6.5 | httr | 1.4.2 | hwriter | 1.3.2 |
| hwriterPlus | 1.0-3 | ids | 1.0.1 | ini | 0.3.1 |
| ipred | 0.9-12 | isoband | 0.2.5 | iterators | 1.0.13 |
| jquerylib | 0.1.4 | jsonlite | 1.7.3 | KernSmooth | 2.23-20 |
| knitr | 1.37 | labeling | 0.4.2 | later | 1.3.0 |
| lattice | 0.20-45 | lava | 1.6.10 | lifecycle | 1.0.1 |
| listenv | 0.8.0 | lubridate | 1.8.0 | magrittr | 2.0.1 |
| markdown | 1.1 | MASS | 7.3-55 | Matrix | 1.4-0 |
| memoise | 2.0.1 | methods | 4.1.2 | mgcv | 1.8-38 |
| mime | 0.12 | ModelMetrics | 1.2.2.2 | modelr | 0.1.8 |
| munsell | 0.5.0 | nlme | 3.1-155 | nnet | 7.3-17 |
| numDeriv | 2016.8-1.1 | openssl | 1.4.6 | parallel | 4.1.2 |
| parallelly | 1.30.0 | pillar | 1.6.4 | pkgbuild | 1.3.1 |
| pkgconfig | 2.0.3 | pkgload | 1.2.4 | plogr | 0.2.0 |
| plyr | 1.8.6 | praise | 1.0.0 | prettyunits | 1.1.1 |
| pROC | 1.18.0 | processx | 3.5.2 | prodlim | 2019.11.13 |
| progress | 1.2.2 | progressr | 0.10.0 | promises | 1.2.0.1 |
| proto | 1.0.0 | proxy | 0.4-26 | ps | 1.6.0 |
| purrr | 0.3.4 | r2d3 | 0.2.5 | R6 | 2.5.1 |
| randomForest | 4.6-14 | rappdirs | 0.3.3 | rcmdcheck | 1.4.0 |
| RColorBrewer | 1.1-2 | Rcpp | 1.0.8 | RcppEigen | 0.3.3.9.1 |
| readr | 2.1.1 | readxl | 1.3.1 | recipes | 0.1.17 |
| rematch | 1.0.1 | rematch2 | 2.1.2 | remotes | 2.4.2 |
| reprex | 2.0.1 | reshape2 | 1.4.4 | rlang | 0.4.12 |
| rmarkdown | 2.11 | RODBC | 1.3-19 | roxygen2 | 7.1.2 |
| rpart | 4.1-15 | rprojroot | 2.0.2 | Rserve | 1.8-10 |
| RSQLite | 2.2.9 | rstudioapi | 0.13 | rversions | 2.1.1 |
| rvest | 1.0.2 | sass | 0.4.0 | scales | 1.1.1 |
| selectr | 0.4-2 | sessioninfo | 1.2.2 | shape | 1.4.6 |
| shiny | 1.7.1 | sourcetools | 0.1.7 | sparklyr | 1.7.4 |
| SparkR | 3.2.0 | spatial | 7.3-11 | splines | 4.1.2 |
| sqldf | 0.4-11 | SQUAREM | 2021.1 | stats | 4.1.2 |
| stats4 | 4.1.2 | stringi | 1.7.6 | stringr | 1.4.0 |
| survival | 3.2-13 | sys | 3.4 | tcltk | 4.1.2 |
| TeachingDemos | 2.10 | testthat | 3.1.1 | tibble | 3.1.6 |
| tidyr | 1.1.4 | tidyselect | 1.1.1 | tidyverse | 1.3.1 |
| timeDate | 3043.102 | tinytex | 0.36 | tools | 4.1.2 |
| tzdb | 0.2.0 | usethis | 2.1.5 | utf8 | 1.2.2 |
| utils | 4.1.2 | uuid | 1.0-3 | vctrs | 0.3.8 |
| viridisLite | 0.4.0 | vroom | 1.5.7 | waldo | 0.3.1 |
| whisker | 0.4 | withr | 2.4.3 | xfun | 0.29 |
| xml2 | 1.3.3 | xopen | 1.0.0 | xtable | 1.8-4 |
| yaml | 2.2.1 | zip | 2.2.0 |
Installed Java and Scala libraries (Scala 2.12 cluster version)
| Group ID | Artifact ID | Version |
|---|---|---|
| antlr | antlr | 2.7.7 |
| com.amazonaws | amazon-kinesis-client | 1.12.0 |
| com.amazonaws | aws-java-sdk-autoscaling | 1.11.655 |
| com.amazonaws | aws-java-sdk-cloudformation | 1.11.655 |
| com.amazonaws | aws-java-sdk-cloudfront | 1.11.655 |
| com.amazonaws | aws-java-sdk-cloudhsm | 1.11.655 |
| com.amazonaws | aws-java-sdk-cloudsearch | 1.11.655 |
| com.amazonaws | aws-java-sdk-cloudtrail | 1.11.655 |
| com.amazonaws | aws-java-sdk-cloudwatch | 1.11.655 |
| com.amazonaws | aws-java-sdk-cloudwatchmetrics | 1.11.655 |
| com.amazonaws | aws-java-sdk-codedeploy | 1.11.655 |
| com.amazonaws | aws-java-sdk-cognitoidentity | 1.11.655 |
| com.amazonaws | aws-java-sdk-cognitosync | 1.11.655 |
| com.amazonaws | aws-java-sdk-config | 1.11.655 |
| com.amazonaws | aws-java-sdk-core | 1.11.655 |
| com.amazonaws | aws-java-sdk-datapipeline | 1.11.655 |
| com.amazonaws | aws-java-sdk-directconnect | 1.11.655 |
| com.amazonaws | aws-java-sdk-directory | 1.11.655 |
| com.amazonaws | aws-java-sdk-dynamodb | 1.11.655 |
| com.amazonaws | aws-java-sdk-ec2 | 1.11.655 |
| com.amazonaws | aws-java-sdk-ecs | 1.11.655 |
| com.amazonaws | aws-java-sdk-efs | 1.11.655 |
| com.amazonaws | aws-java-sdk-elasticache | 1.11.655 |
| com.amazonaws | aws-java-sdk-elasticbeanstalk | 1.11.655 |
| com.amazonaws | aws-java-sdk-elasticloadbalancing | 1.11.655 |
| com.amazonaws | aws-java-sdk-elastictranscoder | 1.11.655 |
| com.amazonaws | aws-java-sdk-emr | 1.11.655 |
| com.amazonaws | aws-java-sdk-glacier | 1.11.655 |
| com.amazonaws | aws-java-sdk-glue | 1.11.655 |
| com.amazonaws | aws-java-sdk-iam | 1.11.655 |
| com.amazonaws | aws-java-sdk-importexport | 1.11.655 |
| com.amazonaws | aws-java-sdk-kinesis | 1.11.655 |
| com.amazonaws | aws-java-sdk-kms | 1.11.655 |
| com.amazonaws | aws-java-sdk-lambda | 1.11.655 |
| com.amazonaws | aws-java-sdk-logs | 1.11.655 |
| com.amazonaws | aws-java-sdk-machinelearning | 1.11.655 |
| com.amazonaws | aws-java-sdk-opsworks | 1.11.655 |
| com.amazonaws | aws-java-sdk-rds | 1.11.655 |
| com.amazonaws | aws-java-sdk-redshift | 1.11.655 |
| com.amazonaws | aws-java-sdk-route53 | 1.11.655 |
| com.amazonaws | aws-java-sdk-s3 | 1.11.655 |
| com.amazonaws | aws-java-sdk-ses | 1.11.655 |
| com.amazonaws | aws-java-sdk-simpledb | 1.11.655 |
| com.amazonaws | aws-java-sdk-simpleworkflow | 1.11.655 |
| com.amazonaws | aws-java-sdk-sns | 1.11.655 |
| com.amazonaws | aws-java-sdk-sqs | 1.11.655 |
| com.amazonaws | aws-java-sdk-ssm | 1.11.655 |
| com.amazonaws | aws-java-sdk-storagegateway | 1.11.655 |
| com.amazonaws | aws-java-sdk-sts | 1.11.655 |
| com.amazonaws | aws-java-sdk-support | 1.11.655 |
| com.amazonaws | aws-java-sdk-swf-libraries | 1.11.22 |
| com.amazonaws | aws-java-sdk-workspaces | 1.11.655 |
| com.amazonaws | jmespath-java | 1.11.655 |
| com.chuusai | shapeless_2.12 | 2.3.3 |
| com.clearspring.analytics | stream | 2.9.6 |
| com.databricks | Rserve | 1.8-3 |
| com.databricks | jets3t | 0.7.1-0 |
| com.databricks.scalapb | compilerplugin_2.12 | 0.4.15-10 |
| com.databricks.scalapb | scalapb-runtime_2.12 | 0.4.15-10 |
| com.esotericsoftware | kryo-shaded | 4.0.2 |
| com.esotericsoftware | minlog | 1.3.0 |
| com.fasterxml | classmate | 1.3.4 |
| com.fasterxml.jackson.core | jackson-annotations | 2.12.3 |
| com.fasterxml.jackson.core | jackson-core | 2.12.3 |
| com.fasterxml.jackson.core | jackson-databind | 2.12.3 |
| com.fasterxml.jackson.dataformat | jackson-dataformat-cbor | 2.12.3 |
| com.fasterxml.jackson.datatype | jackson-datatype-joda | 2.12.3 |
| com.fasterxml.jackson.module | jackson-module-paranamer | 2.12.3 |
| com.fasterxml.jackson.module | jackson-module-scala_2.12 | 2.12.3 |
| com.github.ben-manes.caffeine | caffeine | 2.3.4 |
| com.github.fommil | jniloader | 1.1 |
| com.github.fommil.netlib | core | 1.1.2 |
| com.github.fommil.netlib | native_ref-java | 1.1 |
| com.github.fommil.netlib | native_ref-java-natives | 1.1 |
| com.github.fommil.netlib | native_system-java | 1.1 |
| com.github.fommil.netlib | native_system-java-natives | 1.1 |
| com.github.fommil.netlib | netlib-native_ref-linux-x86_64-natives | 1.1 |
| com.github.fommil.netlib | netlib-native_system-linux-x86_64-natives | 1.1 |
| com.github.luben | zstd-jni | 1.5.0-4 |
| com.github.wendykierp | JTransforms | 3.1 |
| com.google.code.findbugs | jsr305 | 3.0.0 |
| com.google.code.gson | gson | 2.8.6 |
| com.google.crypto.tink | tink | 1.6.0 |
| com.google.flatbuffers | flatbuffers-java | 1.9.0 |
| com.google.guava | guava | 15.0 |
| com.google.protobuf | protobuf-java | 2.6.1 |
| com.h2database | h2 | 1.4.195 |
| com.helger | profiler | 1.1.1 |
| com.jcraft | jsch | 0.1.50 |
| com.jolbox | bonecp | 0.8.0.RELEASE |
| com.lihaoyi | sourcecode_2.12 | 0.1.9 |
| com.microsoft.azure | azure-data-lake-store-sdk | 2.3.9 |
| com.microsoft.sqlserver | mssql-jdbc | 9.2.1.jre8 |
| com.ning | compress-lzf | 1.0.3 |
| com.sun.istack | istack-commons-runtime | 3.0.8 |
| com.sun.mail | javax.mail | 1.5.2 |
| com.tdunning | json | 1.8 |
| com.thoughtworks.paranamer | paranamer | 2.8 |
| com.trueaccord.lenses | lenses_2.12 | 0.4.12 |
| com.twitter | chill-java | 0.10.0 |
| com.twitter | chill_2.12 | 0.10.0 |
| com.twitter | util-app_2.12 | 7.1.0 |
| com.twitter | util-core_2.12 | 7.1.0 |
| com.twitter | util-function_2.12 | 7.1.0 |
| com.twitter | util-jvm_2.12 | 7.1.0 |
| com.twitter | util-lint_2.12 | 7.1.0 |
| com.twitter | util-registry_2.12 | 7.1.0 |
| com.twitter | util-stats_2.12 | 7.1.0 |
| com.typesafe | config | 1.2.1 |
| com.typesafe.scala-logging | scala-logging_2.12 | 3.7.2 |
| com.univocity | univocity-parsers | 2.9.1 |
| com.zaxxer | HikariCP | 4.0.3 |
| commons-cli | commons-cli | 1.2 |
| commons-codec | commons-codec | 1.15 |
| commons-collections | commons-collections | 3.2.2 |
| commons-dbcp | commons-dbcp | 1.4 |
| commons-fileupload | commons-fileupload | 1.3.3 |
| commons-httpclient | commons-httpclient | 3.1 |
| commons-io | commons-io | 2.8.0 |
| commons-lang | commons-lang | 2.6 |
| commons-logging | commons-logging | 1.1.3 |
| commons-net | commons-net | 3.1 |
| commons-pool | commons-pool | 1.5.4 |
| dev.ludovic.netlib | arpack | 2.2.1 |
| dev.ludovic.netlib | blas | 2.2.1 |
| dev.ludovic.netlib | lapack | 2.2.1 |
| hive-2.3__hadoop-3.2 | jets3t-0.7 | liball_deps_2.12 |
| info.ganglia.gmetric4j | gmetric4j | 1.0.10 |
| io.airlift | aircompressor | 0.21 |
| io.delta | delta-sharing-spark_2.12 | 0.3.0 |
| io.dropwizard.metrics | metrics-core | 4.1.1 |
| io.dropwizard.metrics | metrics-graphite | 4.1.1 |
| io.dropwizard.metrics | metrics-healthchecks | 4.1.1 |
| io.dropwizard.metrics | metrics-jetty9 | 4.1.1 |
| io.dropwizard.metrics | metrics-jmx | 4.1.1 |
| io.dropwizard.metrics | metrics-json | 4.1.1 |
| io.dropwizard.metrics | metrics-jvm | 4.1.1 |
| io.dropwizard.metrics | metrics-servlets | 4.1.1 |
| io.netty | netty-all | 4.1.68.Final |
| io.prometheus | simpleclient | 0.7.0 |
| io.prometheus | simpleclient_common | 0.7.0 |
| io.prometheus | simpleclient_dropwizard | 0.7.0 |
| io.prometheus | simpleclient_pushgateway | 0.7.0 |
| io.prometheus | simpleclient_servlet | 0.7.0 |
| io.prometheus.jmx | collector | 0.12.0 |
| jakarta.annotation | jakarta.annotation-api | 1.3.5 |
| jakarta.servlet | jakarta.servlet-api | 4.0.3 |
| jakarta.validation | jakarta.validation-api | 2.0.2 |
| jakarta.ws.rs | jakarta.ws.rs-api | 2.1.6 |
| javax.activation | activation | 1.1.1 |
| javax.annotation | javax.annotation-api | 1.3.2 |
| javax.el | javax.el-api | 2.2.4 |
| javax.jdo | jdo-api | 3.0.1 |
| javax.transaction | jta | 1.1 |
| javax.transaction | transaction-api | 1.1 |
| javax.xml.bind | jaxb-api | 2.2.2 |
| javax.xml.stream | stax-api | 1.0-2 |
| javolution | javolution | 5.5.1 |
| jline | jline | 2.14.6 |
| joda-time | joda-time | 2.10.10 |
| log4j | apache-log4j-extras | 1.2.17 |
| log4j | log4j | 1.2.17 |
| maven-trees | hive-2.3__hadoop-3.2 | liball_deps_2.12 |
| net.java.dev.jna | jna | 5.8.0 |
| net.razorvine | pyrolite | 4.30 |
| net.sf.jpam | jpam | 1.1 |
| net.sf.opencsv | opencsv | 2.3 |
| net.sf.supercsv | super-csv | 2.2.0 |
| net.snowflake | snowflake-ingest-sdk | 0.9.6 |
| net.snowflake | snowflake-jdbc | 3.13.3 |
| net.snowflake | spark-snowflake_2.12 | 2.9.0-spark_3.1 |
| net.sourceforge.f2j | arpack_combined_all | 0.1 |
| org.acplt.remotetea | remotetea-oncrpc | 1.1.2 |
| org.antlr | ST4 | 4.0.4 |
| org.antlr | antlr-runtime | 3.5.2 |
| org.antlr | antlr4-runtime | 4.8 |
| org.antlr | stringtemplate | 3.2.1 |
| org.apache.ant | ant | 1.9.2 |
| org.apache.ant | ant-jsch | 1.9.2 |
| org.apache.ant | ant-launcher | 1.9.2 |
| org.apache.arrow | arrow-format | 2.0.0 |
| org.apache.arrow | arrow-memory-core | 2.0.0 |
| org.apache.arrow | arrow-memory-netty | 2.0.0 |
| org.apache.arrow | arrow-vector | 2.0.0 |
| org.apache.avro | avro | 1.10.2 |
| org.apache.avro | avro-ipc | 1.10.2 |
| org.apache.avro | avro-mapred | 1.10.2 |
| org.apache.commons | commons-compress | 1.21 |
| org.apache.commons | commons-crypto | 1.1.0 |
| org.apache.commons | commons-lang3 | 3.12.0 |
| org.apache.commons | commons-math3 | 3.4.1 |
| org.apache.commons | commons-text | 1.6 |
| org.apache.curator | curator-client | 2.13.0 |
| org.apache.curator | curator-framework | 2.13.0 |
| org.apache.curator | curator-recipes | 2.13.0 |
| org.apache.derby | derby | 10.14.2.0 |
| org.apache.hadoop | hadoop-client-api | 3.3.1-databricks |
| org.apache.hadoop | hadoop-client-runtime | 3.3.1 |
| org.apache.hive | hive-beeline | 2.3.9 |
| org.apache.hive | hive-cli | 2.3.9 |
| org.apache.hive | hive-jdbc | 2.3.9 |
| org.apache.hive | hive-llap-client | 2.3.9 |
| org.apache.hive | hive-llap-common | 2.3.9 |
| org.apache.hive | hive-serde | 2.3.9 |
| org.apache.hive | hive-shims | 2.3.9 |
| org.apache.hive | hive-storage-api | 2.7.2 |
| org.apache.hive.shims | hive-shims-0.23 | 2.3.9 |
| org.apache.hive.shims | hive-shims-common | 2.3.9 |
| org.apache.hive.shims | hive-shims-scheduler | 2.3.9 |
| org.apache.htrace | htrace-core4 | 4.1.0-incubating |
| org.apache.httpcomponents | httpclient | 4.5.13 |
| org.apache.httpcomponents | httpcore | 4.4.12 |
| org.apache.ivy | ivy | 2.5.0 |
| org.apache.mesos | mesos-shaded-protobuf | 1.4.0 |
| org.apache.orc | orc-core | 1.6.12 |
| org.apache.orc | orc-mapreduce | 1.6.12 |
| org.apache.orc | orc-shims | 1.6.12 |
| org.apache.parquet | parquet-column | 1.12.0-databricks-0003 |
| org.apache.parquet | parquet-common | 1.12.0-databricks-0003 |
| org.apache.parquet | parquet-encoding | 1.12.0-databricks-0003 |
| org.apache.parquet | parquet-format-structures | 1.12.0-databricks-0003 |
| org.apache.parquet | parquet-hadoop | 1.12.0-databricks-0003 |
| org.apache.parquet | parquet-jackson | 1.12.0-databricks-0003 |
| org.apache.thrift | libfb303 | 0.9.3 |
| org.apache.thrift | libthrift | 0.12.0 |
| org.apache.xbean | xbean-asm9-shaded | 4.20 |
| org.apache.yetus | audience-annotations | 0.5.0 |
| org.apache.zookeeper | zookeeper | 3.6.2 |
| org.apache.zookeeper | zookeeper-jute | 3.6.2 |
| org.checkerframework | checker-qual | 3.5.0 |
| org.codehaus.jackson | jackson-core-asl | 1.9.13 |
| org.codehaus.jackson | jackson-mapper-asl | 1.9.13 |
| org.codehaus.janino | commons-compiler | 3.0.16 |
| org.codehaus.janino | janino | 3.0.16 |
| org.datanucleus | datanucleus-api-jdo | 4.2.4 |
| org.datanucleus | datanucleus-core | 4.1.17 |
| org.datanucleus | datanucleus-rdbms | 4.1.19 |
| org.datanucleus | javax.jdo | 3.2.0-m3 |
| org.eclipse.jetty | jetty-client | 9.4.43.v20210629 |
| org.eclipse.jetty | jetty-continuation | 9.4.43.v20210629 |
| org.eclipse.jetty | jetty-http | 9.4.43.v20210629 |
| org.eclipse.jetty | jetty-io | 9.4.43.v20210629 |
| org.eclipse.jetty | jetty-jndi | 9.4.43.v20210629 |
| org.eclipse.jetty | jetty-plus | 9.4.43.v20210629 |
| org.eclipse.jetty | jetty-proxy | 9.4.43.v20210629 |
| org.eclipse.jetty | jetty-security | 9.4.43.v20210629 |
| org.eclipse.jetty | jetty-server | 9.4.43.v20210629 |
| org.eclipse.jetty | jetty-servlet | 9.4.43.v20210629 |
| org.eclipse.jetty | jetty-servlets | 9.4.43.v20210629 |
| org.eclipse.jetty | jetty-util | 9.4.43.v20210629 |
| org.eclipse.jetty | jetty-util-ajax | 9.4.43.v20210629 |
| org.eclipse.jetty | jetty-webapp | 9.4.43.v20210629 |
| org.eclipse.jetty | jetty-xml | 9.4.43.v20210629 |
| org.eclipse.jetty.websocket | websocket-api | 9.4.43.v20210629 |
| org.eclipse.jetty.websocket | websocket-client | 9.4.43.v20210629 |
| org.eclipse.jetty.websocket | websocket-common | 9.4.43.v20210629 |
| org.eclipse.jetty.websocket | websocket-server | 9.4.43.v20210629 |
| org.eclipse.jetty.websocket | websocket-servlet | 9.4.43.v20210629 |
| org.fusesource.leveldbjni | leveldbjni-all | 1.8 |
| org.glassfish.hk2 | hk2-api | 2.6.1 |
| org.glassfish.hk2 | hk2-locator | 2.6.1 |
| org.glassfish.hk2 | hk2-utils | 2.6.1 |
| org.glassfish.hk2 | osgi-resource-locator | 1.0.3 |
| org.glassfish.hk2.external | aopalliance-repackaged | 2.6.1 |
| org.glassfish.hk2.external | jakarta.inject | 2.6.1 |
| org.glassfish.jaxb | jaxb-runtime | 2.3.2 |
| org.glassfish.jersey.containers | jersey-container-servlet | 2.34 |
| org.glassfish.jersey.containers | jersey-container-servlet-core | 2.34 |
| org.glassfish.jersey.core | jersey-client | 2.34 |
| org.glassfish.jersey.core | jersey-common | 2.34 |
| org.glassfish.jersey.core | jersey-server | 2.34 |
| org.glassfish.jersey.inject | jersey-hk2 | 2.34 |
| org.hibernate.validator | hibernate-validator | 6.1.0.Final |
| org.javassist | javassist | 3.25.0-GA |
| org.jboss.logging | jboss-logging | 3.3.2.Final |
| org.jdbi | jdbi | 2.63.1 |
| org.jetbrains | annotations | 17.0.0 |
| org.joda | joda-convert | 1.7 |
| org.jodd | jodd-core | 3.5.2 |
| org.json4s | json4s-ast_2.12 | 3.7.0-M11 |
| org.json4s | json4s-core_2.12 | 3.7.0-M11 |
| org.json4s | json4s-jackson_2.12 | 3.7.0-M11 |
| org.json4s | json4s-scalap_2.12 | 3.7.0-M11 |
| org.lz4 | lz4-java | 1.7.1 |
| org.mariadb.jdbc | mariadb-java-client | 2.2.5 |
| org.objenesis | objenesis | 2.5.1 |
| org.postgresql | postgresql | 42.2.19 |
| org.roaringbitmap | RoaringBitmap | 0.9.14 |
| org.roaringbitmap | shims | 0.9.14 |
| org.rocksdb | rocksdbjni | 6.20.3 |
| org.rosuda.REngine | REngine | 2.1.0 |
| org.scala-lang | scala-compiler_2.12 | 2.12.14 |
| org.scala-lang | scala-library_2.12 | 2.12.14 |
| org.scala-lang | scala-reflect_2.12 | 2.12.14 |
| org.scala-lang.modules | scala-collection-compat_2.12 | 2.4.3 |
| org.scala-lang.modules | scala-parser-combinators_2.12 | 1.1.2 |
| org.scala-lang.modules | scala-xml_2.12 | 1.2.0 |
| org.scala-sbt | test-interface | 1.0 |
| org.scalacheck | scalacheck_2.12 | 1.14.2 |
| org.scalactic | scalactic_2.12 | 3.0.8 |
| org.scalanlp | breeze-macros_2.12 | 1.2 |
| org.scalanlp | breeze_2.12 | 1.2 |
| org.scalatest | scalatest_2.12 | 3.0.8 |
| org.slf4j | jcl-over-slf4j | 1.7.30 |
| org.slf4j | jul-to-slf4j | 1.7.30 |
| org.slf4j | slf4j-api | 1.7.30 |
| org.slf4j | slf4j-log4j12 | 1.7.30 |
| org.spark-project.spark | unused | 1.0.0 |
| org.springframework | spring-core | 4.1.4.RELEASE |
| org.springframework | spring-test | 4.1.4.RELEASE |
| org.threeten | threeten-extra | 1.5.0 |
| org.tukaani | xz | 1.8 |
| org.typelevel | algebra_2.12 | 2.0.1 |
| org.typelevel | cats-kernel_2.12 | 2.1.1 |
| org.typelevel | macro-compat_2.12 | 1.1.1 |
| org.typelevel | spire-macros_2.12 | 0.17.0 |
| org.typelevel | spire-platform_2.12 | 0.17.0 |
| org.typelevel | spire-util_2.12 | 0.17.0 |
| org.typelevel | spire_2.12 | 0.17.0 |
| org.wildfly.openssl | wildfly-openssl | 1.0.7.Final |
| org.xerial | sqlite-jdbc | 3.8.11.2 |
| org.xerial.snappy | snappy-java | 1.1.8.4 |
| org.yaml | snakeyaml | 1.24 |
| oro | oro | 2.0.8 |
| pl.edu.icm | JLargeArrays | 1.5 |
| software.amazon.ion | ion-java | 1.0.2 |
| stax | stax-api | 1.0.1 |