Databricks Runtime 16.0 for Machine Learning

Databricks Runtime 16.0 for Machine Learning provides a ready-to-go environment for machine learning and data science based on Databricks Runtime 16.0. Databricks Runtime ML contains many popular machine learning libraries, including TensorFlow, PyTorch, and XGBoost. Databricks Runtime ML includes AutoML, a tool to automatically train machine learning pipelines. Databricks Runtime ML also supports distributed deep learning training using TorchDistributor, DeepSpeed, and Ray.

Tip

To see release notes for Databricks Runtime versions that have reached end-of-support (EoS), see End-of-support Databricks Runtime release notes. The EoS Databricks Runtime versions have been retired and might not be updated.

New features and improvements

Databricks Runtime 16.0 ML is built on top of Databricks Runtime 16.0. For information on what’s new in Databricks Runtime 16.0, including Apache Spark MLlib and SparkR, see the Databricks Runtime 16.0 release notes.

New Python packages

The following Python packages have been added to Databricks Runtime ML:

  • composer 0.24.1
  • optuna 3.6.1

AutoML sample weights for forecasting

AutoML now supports sample weights for forecasting, letting you adjust the importance of each time-series for training multi-time-series forecasting models. For more information, see the forecasting parameters for the AutoML Python API.

Use a view in Unity Catalog as a feature table

You can now use a view in Unity Catalog as a feature table. See Use an existing view in Unity Catalog as a feature table.

Other changes

Horovod, HorovodRunner, Petastorm, spark-tensorflow-distributor removed

The following packages that were included in previous versions of Databricks Runtime ML are not included in Databricks Runtime 16.0 ML:

  • Horovod
  • HorovodRunner
  • Petastorm
  • spark-tensorflow-distributor

Databricks recommends the following replacements:

  • For distributed deep learning, Databricks recommends using TorchDistributor for distributed training with PyTorch or the tf.distribute.Strategy API for distributed training with TensorFlow.
  • For loading large datasets from cloud storage, Databricks recommends using Mosaic Streaming.
  • For distributed training for a TensorFlow or Keras model, Databricks recommends using Ray. See Ray on Databricks and the Ray documentation.

System environment

The system environment in Databricks Runtime 16.0 ML differs from Databricks Runtime 16.0 as follows:

  • For GPU clusters, Databricks Runtime ML includes the following NVIDIA GPU libraries:
    • CUDA 12.6
    • cublas 12.6.0.22-1
    • cusolver 11.6.4.38-1
    • cupti 12.6.37-1
    • cusparse 12.5.2.23-1
    • cuDNN 9.3.0.75-1
    • NCCL 2.22.3
    • TensorRT 10.2.0.19-1

Libraries

The following sections list the libraries included in Databricks Runtime 16.0 ML that differ from those included in Databricks Runtime 16.0.

In this section:

Top-tier libraries

Databricks Runtime 16.0 ML includes the following top-tier libraries:

Python libraries

Databricks Runtime 16.0 ML uses virtualenv for Python package management and includes many popular ML packages.

In addition to the packages specified in the following sections, Databricks Runtime 16.0 ML also includes the following packages:

  • hyperopt 0.2.7+db5
  • automl 1.29.0

To reproduce the Databricks Runtime ML Python environment in your local Python virtual environment, download the requirements-16.0.txt file and run pip install -r requirements-16.0.txt. This command installs all of the open source libraries that Databricks Runtime ML uses, but does not install libraries developed by Databricks, such as databricks-automl, databricks-feature-engineering, or the Databricks fork of hyperopt.

Python libraries on CPU clusters

Library Version Library Version Library Version
absl-py 1.0.0 accelerate 0.33.0 aiohttp 3.9.5
aiohttp-cors 0.7.0 aiosignal 1.2.0 alembic 1.13.3
annotated-types 0.7.0 anyio 4.2.0 argcomplete 3.5.0
argon2-cffi 21.3.0 argon2-cffi-bindings 21.2.0 arrow 1.2.3
astor 0.8.1 asttokens 2.0.5 astunparse 1.6.3
async-lru 2.0.4 attrs 23.1.0 audioread 3.0.1
autocommand 2.2.2 azure-core 1.31.0 azure-cosmos 4.3.1
azure-identity 1.18.0 azure-storage-blob 12.23.1 azure-storage-file-datalake 12.17.0
Babel 2.11.0 backoff 2.2.1 backports.tarfile 1.2.0
bcrypt 3.2.0 beautifulsoup4 4.12.3 black 24.4.2
bleach 4.1.0 blinker 1.7.0 blis 0.7.11
boto3 1.34.69 botocore 1.34.69 Brotli 1.0.9
cachetools 5.3.3 catalogue 2.0.10 category-encoders 2.6.3
certifi 2024.6.2 cffi 1.16.0 chardet 4.0.0
charset-normalizer 2.0.4 circuitbreaker 2.0.0 click 8.1.7
cloudpathlib 0.19.0 cloudpickle 2.2.1 cmdstanpy 1.2.4
colorful 0.5.6 colorlog 6.8.2 comm 0.2.1
composer 0.24.1 confection 0.1.5 configparser 5.2.0
contourpy 1.2.0 coolname 2.2.0 cryptography 42.0.5
cycler 0.11.0 cymem 2.0.8 Cython 3.0.11
dacite 1.8.1 databricks-automl-runtime 0.2.21 databricks-feature-engineering 0.7.0
databricks-sdk 0.30.0 datasets 2.20.0 dbl-tempo 0.1.26
dbus-python 1.3.2 debugpy 1.6.7 decorator 5.1.1
deepspeed 0.14.4 defusedxml 0.7.1 Deprecated 1.2.14
dill 0.3.8 distlib 0.3.8 dm-tree 0.1.8
docstring-to-markdown 0.11 entrypoints 0.4 evaluate 0.4.2
executing 0.8.3 facets-overview 1.1.1 Farama-Notifications 0.0.4
fastjsonschema 2.20.0 fasttext-wheel 0.9.2 filelock 3.13.1
Flask 2.2.5 flatbuffers 24.3.25 fonttools 4.51.0
fqdn 1.5.1 frozenlist 1.4.0 fsspec 2023.5.0
future 0.18.3 gast 0.4.0 gitdb 4.0.11
GitPython 3.1.37 google-api-core 2.20.0 google-auth 2.21.0
google-auth-oauthlib 1.0.0 google-cloud-core 2.4.1 google-cloud-storage 2.10.0
google-crc32c 1.6.0 google-pasta 0.2.0 google-resumable-media 2.7.2
googleapis-common-protos 1.65.0 gql 3.5.0 graphql-core 3.2.4
greenlet 3.0.1 grpcio 1.60.0 grpcio-status 1.60.0
gunicorn 20.1.0 gviz-api 1.10.0 gymnasium 0.28.1
h11 0.14.0 h5py 3.11.0 hjson 3.1.0
holidays 0.54 htmlmin 0.1.12 httpcore 1.0.5
httplib2 0.20.4 httpx 0.27.2 huggingface-hub 0.24.5
idna 3.7 ImageHash 4.3.1 imageio 2.33.1
imbalanced-learn 0.12.3 importlib-metadata 6.0.0 importlib_resources 6.4.5
inflect 7.3.1 ipyflow-core 0.0.201 ipykernel 6.28.0
ipython 8.25.0 ipython-genutils 0.2.0 ipywidgets 7.7.2
isodate 0.6.1 isoduration 20.11.0 itsdangerous 2.2.0
jaraco.context 5.3.0 jaraco.functools 4.0.1 jaraco.text 3.12.1
jax-jumpy 1.0.0 jedi 0.19.1 Jinja2 3.1.4
jiter 0.5.0 jmespath 1.0.1 joblib 1.4.2
joblibspark 0.5.1 json5 0.9.6 jsonpatch 1.33
jsonpointer 3.0.0 jsonschema 4.19.2 jsonschema-specifications 2023.7.1
jupyter-events 0.10.0 jupyter-lsp 2.2.0 jupyter_client 8.6.0
jupyter_core 5.7.2 jupyter_server 2.14.1 jupyter_server_terminals 0.4.4
jupyterlab 4.0.11 jupyterlab-pygments 0.1.2 jupyterlab_server 2.25.1
keras 3.5.0 kiwisolver 1.4.4 langchain 0.2.12
langchain-core 0.2.41 langchain-text-splitters 0.2.4 langcodes 3.4.1
langsmith 0.1.129 language_data 1.2.0 launchpadlib 1.11.0
lazr.restfulclient 0.14.6 lazr.uri 1.0.6 lazy_loader 0.4
libclang 15.0.6.1 librosa 0.10.2 lightgbm 4.5.0
lightning-utilities 0.11.7 linkify-it-py 2.0.0 llvmlite 0.42.0
lz4 4.3.2 Mako 1.2.0 marisa-trie 1.2.0
Markdown 3.4.1 markdown-it-py 2.2.0 MarkupSafe 2.1.3
matplotlib 3.8.4 matplotlib-inline 0.1.6 mccabe 0.7.0
mdit-py-plugins 0.3.0 mdurl 0.1.0 memray 1.14.0
mistune 2.0.4 ml-dtypes 0.4.1 mlflow-skinny 2.15.1
more-itertools 10.3.0 mosaicml-cli 0.6.41 mosaicml-streaming 0.8.0
mpmath 1.3.0 msal 1.31.0 msal-extensions 1.2.0
msgpack 1.1.0 multidict 6.0.4 multimethod 1.12
multiprocess 0.70.16 murmurhash 1.0.10 mypy 1.10.0
mypy-extensions 1.0.0 namex 0.0.8 nbclient 0.8.0
nbconvert 7.10.0 nbformat 5.9.2 nest-asyncio 1.6.0
networkx 3.2.1 ninja 1.11.1.1 nltk 3.8.1
nodeenv 1.9.1 notebook 7.0.8 notebook_shim 0.2.3
numba 0.59.1 numpy 1.26.4 nvidia-ml-py 12.560.30
oauthlib 3.2.0 oci 2.135.0 openai 1.40.2
opencensus 0.11.4 opencensus-context 0.1.3 opentelemetry-api 1.27.0
opentelemetry-sdk 1.27.0 opentelemetry-semantic-conventions 0.48b0 opt_einsum 3.4.0
optree 0.12.1 optuna 3.6.1 optuna-integration 3.6.0
orjson 3.10.7 overrides 7.4.0 packaging 24.1
pandas 1.5.3 pandocfilters 1.5.0 paramiko 3.4.0
parso 0.8.3 pathspec 0.10.3 patsy 0.5.6
pexpect 4.8.0 phik 0.12.4 pillow 10.3.0
pip 24.2 platformdirs 3.10.0 plotly 5.22.0
pluggy 1.0.0 pmdarima 2.0.4 pooch 1.8.2
portalocker 2.10.1 preshed 3.0.9 prometheus-client 0.14.1
prompt-toolkit 3.0.43 prophet 1.1.5 proto-plus 1.24.0
protobuf 4.24.1 psutil 5.9.0 psycopg2 2.9.3
ptyprocess 0.7.0 pure-eval 0.2.2 py-cpuinfo 9.0.0
py-spy 0.3.14 pyarrow 15.0.2 pyarrow-hotfix 0.6
pyasn1 0.4.8 pyasn1-modules 0.2.8 pybind11 2.13.6
pyccolo 0.0.65 pycparser 2.21 pydantic 2.8.2
pydantic_core 2.20.1 pyflakes 3.2.0 Pygments 2.15.1
PyGObject 3.48.2 PyJWT 2.7.0 PyNaCl 1.5.0
pyodbc 5.0.1 pyOpenSSL 24.0.0 pyparsing 3.0.9
pyright 1.1.294 pytesseract 0.3.10 python-dateutil 2.9.0.post0
python-editor 1.0.4 python-json-logger 2.0.7 python-lsp-jsonrpc 1.1.2
python-lsp-server 1.10.0 python-snappy 0.6.1 pytoolconfig 1.2.6
pytorch-ranger 0.1.1 pytz 2024.1 PyWavelets 1.5.0
PyYAML 6.0.1 pyzmq 25.1.2 questionary 1.10.0
ray 2.35.0 referencing 0.30.2 regex 2023.10.3
requests 2.32.2 requests-oauthlib 1.3.1 rfc3339-validator 0.1.4
rfc3986-validator 0.1.1 rich 13.3.5 rope 1.12.0
rpds-py 0.10.6 rsa 4.9 ruamel.yaml 0.18.6
ruamel.yaml.clib 0.2.8 s3transfer 0.10.2 safetensors 0.4.4
scikit-image 0.23.2 scikit-learn 1.4.2 scipy 1.13.1
seaborn 0.13.2 Send2Trash 1.8.2 sentence-transformers 3.0.1
sentencepiece 0.2.0 setuptools 74.0.0 shap 0.46.0
shellingham 1.5.4 simplejson 3.17.6 six 1.16.0
slicer 0.0.8 smart-open 5.2.1 smmap 5.0.0
sniffio 1.3.0 soundfile 0.12.1 soupsieve 2.5
soxr 0.5.0.post1 spacy 3.7.5 spacy-legacy 3.0.12
spacy-loggers 1.0.5 SQLAlchemy 2.0.30 sqlparse 0.4.2
srsly 2.4.8 ssh-import-id 5.11 stack-data 0.2.0
stanio 0.5.1 statsmodels 0.14.2 sympy 1.12
tabulate 0.9.0 tangled-up-in-unicode 0.2.0 tenacity 8.2.2
tensorboard 2.17.0 tensorboard-data-server 0.7.2 tensorboard-plugin-profile 2.17.0
tensorboardX 2.6.2.2 tensorflow 2.17.0 tensorflow-estimator 2.15.0
termcolor 2.4.0 terminado 0.17.1 textual 0.81.0
tf_keras 2.17.0 thinc 8.2.5 threadpoolctl 2.2.0
tifffile 2023.4.12 tiktoken 0.7.0 tinycss2 1.2.1
tokenize-rt 4.2.1 tokenizers 0.19.1 tomli 2.0.1
torch 2.4.0+cpu torch-optimizer 0.3.0 torcheval 0.0.7
torchmetrics 1.4.0.post0 torchvision 0.19.0+cpu tornado 6.4.1
tqdm 4.66.4 traitlets 5.14.3 transformers 4.44.0
typeguard 4.3.0 typer 0.12.5 types-protobuf 3.20.3
types-psutil 5.9.0 types-pytz 2023.3.1.1 types-PyYAML 6.0.0
types-requests 2.31.0.0 types-setuptools 68.0.0.0 types-six 1.16.0
types-urllib3 1.26.25.14 typing_extensions 4.11.0 uc-micro-py 1.0.1
ujson 5.10.0 unattended-upgrades 0.1 uri-template 1.3.0
urllib3 1.26.16 validators 0.34.0 virtualenv 20.26.2
visions 0.7.5 wadllib 1.3.6 wasabi 1.1.3
wcwidth 0.2.5 weasel 0.4.1 webcolors 24.8.0
webencodings 0.5.1 websocket-client 1.8.0 websockets 11.0.3
Werkzeug 3.0.3 whatthepatch 1.0.2 wheel 0.43.0
wordcloud 1.9.3 wrapt 1.14.1 xgboost 2.0.3
xgboost-ray 0.1.19 xxhash 3.4.1 yapf 0.33.0
yarl 1.9.3 ydata-profiling 4.9.0 zipp 3.17.0
zstd 1.5.5.1

Python libraries on GPU clusters

Library Version Library Version Library Version
absl-py 1.0.0 accelerate 0.33.0 aiohttp 3.9.5
aiohttp-cors 0.7.0 aiosignal 1.2.0 annotated-types 0.7.0
anyio 4.2.0 argcomplete 3.5.0 argon2-cffi 21.3.0
argon2-cffi-bindings 21.2.0 arrow 1.2.3 astor 0.8.1
asttokens 2.0.5 astunparse 1.6.3 async-lru 2.0.4
attrs 23.1.0 audioread 3.0.1 autocommand 2.2.2
azure-core 1.31.0 azure-cosmos 4.3.1 azure-identity 1.18.0
azure-storage-blob 12.23.1 azure-storage-file-datalake 12.17.0 Babel 2.11.0
backoff 2.2.1 backports.tarfile 1.2.0 bcrypt 3.2.0
beautifulsoup4 4.12.3 black 24.4.2 bleach 4.1.0
blinker 1.7.0 blis 0.7.11 boto3 1.34.69
botocore 1.34.69 Brotli 1.0.9 cachetools 5.3.3
catalogue 2.0.10 category-encoders 2.6.3 certifi 2024.6.2
cffi 1.16.0 chardet 4.0.0 charset-normalizer 2.0.4
circuitbreaker 2.0.0 click 8.1.7 cloudpathlib 0.19.0
cloudpickle 2.2.1 cmdstanpy 1.2.4 colorful 0.5.6
colorlog 6.8.2 comm 0.2.1 composer 0.24.1
confection 0.1.5 configparser 5.2.0 contourpy 1.2.0
coolname 2.2.0 cryptography 42.0.5 cycler 0.11.0
cymem 2.0.8 Cython 3.0.11 dacite 1.8.1
databricks-automl-runtime 0.2.21 databricks-feature-engineering 0.7.0 databricks-sdk 0.30.0
datasets 2.20.0 dbl-tempo 0.1.26 dbus-python 1.3.2
debugpy 1.6.7 decorator 5.1.1 deepspeed 0.14.4
defusedxml 0.7.1 Deprecated 1.2.14 dill 0.3.8
distlib 0.3.8 dm-tree 0.1.8 docstring-to-markdown 0.11
einops 0.8.0 entrypoints 0.4 evaluate 0.4.2
executing 0.8.3 facets-overview 1.1.1 Farama-Notifications 0.0.4
fastjsonschema 2.20.0 fasttext-wheel 0.9.2 filelock 3.13.1
flash_attn 2.5.6 Flask 2.2.5 flatbuffers 24.3.25
fonttools 4.51.0 fqdn 1.5.1 frozenlist 1.4.0
fsspec 2023.5.0 future 0.18.3 gast 0.4.0
gitdb 4.0.11 GitPython 3.1.37 google-api-core 2.20.0
google-auth 2.21.0 google-auth-oauthlib 1.0.0 google-cloud-core 2.4.1
google-cloud-storage 2.10.0 google-crc32c 1.6.0 google-pasta 0.2.0
google-resumable-media 2.7.2 googleapis-common-protos 1.65.0 gql 3.5.0
graphql-core 3.2.4 greenlet 3.0.1 grpcio 1.60.0
grpcio-status 1.60.0 gunicorn 20.1.0 gviz-api 1.10.0
gymnasium 0.28.1 h11 0.14.0 h5py 3.11.0
hjson 3.1.0 holidays 0.54 htmlmin 0.1.12
httpcore 1.0.5 httplib2 0.20.4 httpx 0.27.2
huggingface-hub 0.24.5 idna 3.7 ImageHash 4.3.1
imageio 2.33.1 imbalanced-learn 0.12.3 importlib-metadata 6.0.0
importlib_resources 6.4.5 inflect 7.3.1 ipyflow-core 0.0.201
ipykernel 6.28.0 ipython 8.25.0 ipython-genutils 0.2.0
ipywidgets 7.7.2 isodate 0.6.1 isoduration 20.11.0
itsdangerous 2.2.0 jaraco.context 5.3.0 jaraco.functools 4.0.1
jaraco.text 3.12.1 jax-jumpy 1.0.0 jedi 0.19.1
Jinja2 3.1.4 jiter 0.5.0 jmespath 1.0.1
joblib 1.4.2 joblibspark 0.5.1 json5 0.9.6
jsonpatch 1.33 jsonpointer 3.0.0 jsonschema 4.19.2
jsonschema-specifications 2023.7.1 jupyter-events 0.10.0 jupyter-lsp 2.2.0
jupyter_client 8.6.0 jupyter_core 5.7.2 jupyter_server 2.14.1
jupyter_server_terminals 0.4.4 jupyterlab 4.0.11 jupyterlab-pygments 0.1.2
jupyterlab_server 2.25.1 keras 3.5.0 kiwisolver 1.4.4
langchain 0.2.12 langchain-core 0.2.41 langchain-text-splitters 0.2.4
langcodes 3.4.1 langsmith 0.1.129 language_data 1.2.0
launchpadlib 1.11.0 lazr.restfulclient 0.14.6 lazr.uri 1.0.6
lazy_loader 0.4 libclang 15.0.6.1 librosa 0.10.2
lightgbm 4.5.0 lightning-utilities 0.11.7 linkify-it-py 2.0.0
llvmlite 0.42.0 lz4 4.3.2 Mako 1.2.0
marisa-trie 1.2.0 Markdown 3.4.1 markdown-it-py 2.2.0
MarkupSafe 2.1.3 matplotlib 3.8.4 matplotlib-inline 0.1.6
mccabe 0.7.0 mdit-py-plugins 0.3.0 mdurl 0.1.0
memray 1.14.0 mistune 2.0.4 ml-dtypes 0.4.1
mlflow-skinny 2.15.1 more-itertools 10.3.0 mosaicml-cli 0.6.41
mosaicml-streaming 0.8.0 mpmath 1.3.0 msal 1.31.0
msal-extensions 1.2.0 msgpack 1.1.0 multidict 6.0.4
multimethod 1.12 multiprocess 0.70.16 murmurhash 1.0.10
mypy 1.10.0 mypy-extensions 1.0.0 namex 0.0.8
nbclient 0.8.0 nbconvert 7.10.0 nbformat 5.9.2
nest-asyncio 1.6.0 networkx 3.2.1 ninja 1.11.1.1
nltk 3.8.1 nodeenv 1.9.1 notebook 7.0.8
notebook_shim 0.2.3 numba 0.59.1 numpy 1.26.4
nvidia-cublas-cu12 12.4.2.65 nvidia-cuda-cupti-cu12 12.4.99 nvidia-cuda-nvrtc-cu12 12.4.99
nvidia-cuda-runtime-cu12 12.4.99 nvidia-cudnn-cu12 9.1.0.70 nvidia-cufft-cu12 11.2.0.44
nvidia-curand-cu12 10.3.5.119 nvidia-cusolver-cu12 11.6.0.99 nvidia-cusparse-cu12 12.3.0.142
nvidia-ml-py 12.560.30 nvidia-nccl-cu12 2.20.5 nvidia-nvjitlink-cu12 12.4.99
nvidia-nvtx-cu12 12.4.99 oauthlib 3.2.0 oci 2.135.0
openai 1.40.2 opencensus 0.11.4 opencensus-context 0.1.3
opentelemetry-api 1.27.0 opentelemetry-sdk 1.27.0 opentelemetry-semantic-conventions 0.48b0
opt_einsum 3.4.0 optree 0.12.1 optuna 3.6.1
optuna-integration 3.6.0 orjson 3.10.7 overrides 7.4.0
packaging 24.1 pandas 1.5.3 pandocfilters 1.5.0
paramiko 3.4.0 parso 0.8.3 pathspec 0.10.3
patsy 0.5.6 pexpect 4.8.0 phik 0.12.4
pillow 10.3.0 pip 24.2 platformdirs 3.10.0
plotly 5.22.0 pluggy 1.0.0 pmdarima 2.0.4
pooch 1.8.2 portalocker 2.10.1 preshed 3.0.9
prometheus-client 0.14.1 prompt-toolkit 3.0.43 prophet 1.1.5
proto-plus 1.24.0 protobuf 4.24.1 psutil 5.9.0
psycopg2 2.9.3 ptyprocess 0.7.0 pure-eval 0.2.2
py-cpuinfo 9.0.0 py-spy 0.3.14 pyarrow 15.0.2
pyarrow-hotfix 0.6 pyasn1 0.4.8 pyasn1-modules 0.2.8
pybind11 2.13.6 pyccolo 0.0.65 pycparser 2.21
pydantic 2.8.2 pydantic_core 2.20.1 pyflakes 3.2.0
Pygments 2.15.1 PyGObject 3.48.2 PyJWT 2.7.0
PyNaCl 1.5.0 pyodbc 5.0.1 pyOpenSSL 24.0.0
pyparsing 3.0.9 pyright 1.1.294 pytesseract 0.3.10
python-dateutil 2.9.0.post0 python-editor 1.0.4 python-json-logger 2.0.7
python-lsp-jsonrpc 1.1.2 python-lsp-server 1.10.0 python-snappy 0.6.1
pytoolconfig 1.2.6 pytorch-ranger 0.1.1 pytz 2024.1
PyWavelets 1.5.0 PyYAML 6.0.1 pyzmq 25.1.2
questionary 1.10.0 ray 2.35.0 referencing 0.30.2
regex 2023.10.3 requests 2.32.2 requests-oauthlib 1.3.1
rfc3339-validator 0.1.4 rfc3986-validator 0.1.1 rich 13.3.5
rope 1.12.0 rpds-py 0.10.6 rsa 4.9
ruamel.yaml 0.18.6 ruamel.yaml.clib 0.2.8 s3transfer 0.10.2
safetensors 0.4.4 scikit-image 0.23.2 scikit-learn 1.4.2
scipy 1.13.1 seaborn 0.13.2 Send2Trash 1.8.2
sentence-transformers 3.0.1 sentencepiece 0.2.0 setuptools 74.0.0
shap 0.46.0 shellingham 1.5.4 simplejson 3.17.6
six 1.16.0 slicer 0.0.8 smart-open 5.2.1
smmap 5.0.0 sniffio 1.3.0 soundfile 0.12.1
soupsieve 2.5 soxr 0.5.0.post1 spacy 3.7.5
spacy-legacy 3.0.12 spacy-loggers 1.0.5 SQLAlchemy 2.0.30
sqlparse 0.4.2 srsly 2.4.8 ssh-import-id 5.11
stack-data 0.2.0 stanio 0.5.1 statsmodels 0.14.2
sympy 1.12 tabulate 0.9.0 tangled-up-in-unicode 0.2.0
tenacity 8.2.2 tensorboard 2.17.0 tensorboard-data-server 0.7.2
tensorboard-plugin-profile 2.17.0 tensorboardX 2.6.2.2 tensorflow 2.17.0
tensorflow-estimator 2.15.0 termcolor 2.4.0 terminado 0.17.1
textual 0.81.0 tf_keras 2.17.0 thinc 8.2.5
threadpoolctl 2.2.0 tifffile 2023.4.12 tiktoken 0.7.0
tinycss2 1.2.1 tokenize-rt 4.2.1 tokenizers 0.19.1
tomli 2.0.1 torch 2.4.0+cu124 torch-optimizer 0.3.0
torcheval 0.0.7 torchmetrics 1.4.0.post0 torchvision 0.19.0+cu124
tornado 6.4.1 tqdm 4.66.4 traitlets 5.14.3
transformers 4.44.0 triton 3.0.0 typeguard 4.3.0
typer 0.12.5 types-protobuf 3.20.3 types-psutil 5.9.0
types-pytz 2023.3.1.1 types-PyYAML 6.0.0 types-requests 2.31.0.0
types-setuptools 68.0.0.0 types-six 1.16.0 types-urllib3 1.26.25.14
typing_extensions 4.11.0 uc-micro-py 1.0.1 ujson 5.10.0
unattended-upgrades 0.1 uri-template 1.3.0 urllib3 1.26.16
validators 0.34.0 virtualenv 20.26.2 visions 0.7.5
wadllib 1.3.6 wasabi 1.1.3 wcwidth 0.2.5
weasel 0.4.1 webcolors 24.8.0 webencodings 0.5.1
websocket-client 1.8.0 websockets 11.0.3 Werkzeug 3.0.3
whatthepatch 1.0.2 wheel 0.43.0 wordcloud 1.9.3
wrapt 1.14.1 xgboost 2.0.3 xgboost-ray 0.1.19
xxhash 3.4.1 yapf 0.33.0 yarl 1.9.3
ydata-profiling 4.9.0 zipp 3.17.0 zstd 1.5.5.1

R libraries

The R libraries are identical to the R Libraries in Databricks Runtime 16.0.

Java and Scala libraries (Scala 2.12 cluster)

In addition to Java and Scala libraries in Databricks Runtime 16.0, Databricks Runtime 16.0 ML contains the following JARs:

CPU clusters

Group ID Artifact ID Version
com.typesafe.akka akka-actor_2.12 2.5.23
ml.dmlc xgboost4j-spark_2.12 1.7.3
ml.dmlc xgboost4j_2.12 1.7.3
org.graphframes graphframes_2.12 0.8.4-db1-spark3.5
org.mlflow mlflow-client 2.15.1
org.tensorflow spark-tensorflow-connector_2.12 1.15.0

GPU clusters

Group ID Artifact ID Version
com.typesafe.akka akka-actor_2.12 2.5.23
ml.dmlc xgboost4j-gpu_2.12 1.7.3
ml.dmlc xgboost4j-spark-gpu_2.12 1.7.3
org.graphframes graphframes_2.12 0.8.4-db1-spark3.5
org.mlflow mlflow-client 2.15.1
org.tensorflow spark-tensorflow-connector_2.12 1.15.0