Databricks Runtime 15.2 para Machine Learning
O Databricks Runtime 15.2 para Machine Learning fornece um ambiente pronto para uso para aprendizado de máquina e ciência de dados baseado no Databricks Runtime 15.2. O Databricks Runtime ML contém muitas bibliotecas de aprendizado de máquina populares, inclusive TensorFlow, PyTorch e XGBoost. O Databricks Runtime ML inclui o AutoML, uma ferramenta para treinamento automático de pipelines de aprendizado de máquina. O Databricks Runtime ML também oferece suporte ao treinamento de aprendizado profundo distribuído com o uso do Horovod.
Dica
Para ver as notas sobre a versão das versões do Databricks Runtime que chegaram ao fim do suporte (EoS), confira Notas sobre as versões do Databricks Runtime em fim de suporte. As versões do Databricks Runtime EoS foram desativadas e podem não ser atualizadas.
Novos recursos e aprimoramentos
O Databricks Runtime 15.2 ML foi desenvolvido com base no Databricks Runtime 15.2. Para obter informações sobre as novidades do Databricks Runtime 15.2, incluindo Apache Spark MLlib e SparkR, confira as notas sobre a versão do Databricks Runtime 15.2.
Adicionar o StreamingDataset
O StreamingDataset é usado para tornar o treinamento em grandes conjuntos de dados do armazenamento em nuvem o mais rápido, barato e escalonável possível. Ele vem pré-instalado no Databricks Runtime 15.2 ML.
Suporte ao Photon em clusters de CPU do Databricks Runtime ML
A partir do Databricks Runtime 15.2 ML, você pode acelerar suas cargas de trabalho do Spark SQL e do Spark DataFrame habilitando oPhoton em seu cluster de CPU.
Para aplicativos de aprendizado de máquina, o Photon oferece desempenho mais rápido para casos de uso como:
- Preparação de dados usando SQL ou API DataFrame.
- Engenharia de recursos com pesquisa pontual.
- Análise de grafo com GraphFrames.
O Photon não melhora o desempenho de bibliotecas do Python, como TensorFlow, PyTorch e XGBoost.
As APIs do Spark RDD e do Spark MLlib têm compatibilidade limitada com o Photon. Ao processar grandes conjuntos de dados usando o Spark RDD ou o Spark MLlib, você pode enfrentar problemas de memória do Spark. Veja Problemas de memória Spark.
Ambiente do sistema
O ambiente do sistema no Databricks Runtime 15.2 ML difere do Databricks Runtime 15.2 da seguinte forma:
- Para clusters de GPU, o Databricks Runtime ML inclui as seguintes bibliotecas de GPU NVIDIA:
- CUDA 12.1
- cusolver 11.4.5.107-1
- cupti 12.1
- cuDNN 8.9.0.131-1
- NCCL 2.17.1
- TensorRT 8.6.1.6-1
Bibliotecas
As seções a seguir listam as bibliotecas incluídas no Databricks Runtime 15.2 ML que diferem daquelas incluídas no Databricks Runtime 15.2.
Nesta seção:
- Bibliotecas de camada superior
- Bibliotecas do Python
- Bibliotecas do R
- Bibliotecas do Java e do Scala (cluster do Scala 2.12)
Bibliotecas de camada superior
O Databricks Runtime 15.2 ML inclui as seguintes bibliotecas de nível superior:
- datasets
- GraphFrames
- MLflow
- PyTorch
- spark-tensorflow-connector
- Scikit-learn
- TensorFlow
- TensorBoard
- transformadores
Bibliotecas do Python
O Databricks Runtime 15.2 ML usa virtualenv para o gerenciamento de pacotes Python e inclui muitos pacotes populares de ML.
Além dos pacotes especificados nas seções a seguir, o Databricks Runtime 15.2 ML também inclui os seguintes pacotes:
- hyperopt 0.2.7+db3
- sparkdl 3.0.0_db1
- automl 1.26.0
Para reproduzir o ambiente Python do Databricks Runtime ML em seu ambiente virtual Python local, baixe o arquivo requirements-15.2.txt e execute pip install -r requirements-15.2.txt. Esse comando instala todas as bibliotecas código aberto que o Databricks Runtime ML usa, mas não instala bibliotecas desenvolvidas pelo Databricks, como databricks-automl, databricks-feature-engineering, ou o fork do Databricks de hyperopt.
Bibliotecas do Python em clusters de CPU
| Biblioteca | Versão | Biblioteca | Versão | Biblioteca | Versão |
|---|---|---|---|---|---|
| absl-py | 1.0.0 | accelerate | 0.28.0 | aiohttp | 3.8.5 |
| aiohttp-cors | 0.7.0 | aiosignal | 1.2.0 | anyio | 3.5.0 |
| argon2-cffi | 21.3.0 | argon2-cffi-bindings | 21.2.0 | astor | 0.8.1 |
| asttokens | 2.0.5 | astunparse | 1.6.3 | async-timeout | 4.0.2 |
| attrs | 22.1.0 | audioread | 3.0.1 | azure-core | 1.30.1 |
| azure-cosmos | 4.3.1 | azure-identity | 1.16.0 | azure-storage-blob | 12.19.1 |
| azure-storage-file-datalake | 12.14.0 | backcall | 0.2.0 | bcrypt | 3.2.0 |
| beautifulsoup4 | 4.12.2 | black | 23.3.0 | bleach | 4.1.0 |
| blinker | 1.4 | blis | 0.7.11 | boto3 | 1.34.39 |
| botocore | 1.34.39 | Brotli | 1.0.9 | cachetools | 5.3.3 |
| catalogue | 2.0.10 | category-encoders | 2.6.3 | certifi | 2023.7.22 |
| cffi | 1.15.1 | chardet | 4.0.0 | charset-normalizer | 2.0.4 |
| circuitbreaker | 1.4.0 | clique | 8.0.4 | cloudpathlib | 0.16.0 |
| cloudpickle | 2.2.1 | cmdstanpy | 1.2.2 | colorido | 0.5.6 |
| comm | 0.1.2 | confecção | 0.1.4 | configparser | 5.2.0 |
| contourpy | 1.0.5 | criptografia | 41.0.3 | cycler | 0.11.0 |
| cymem | 2.0.8 | Cython | 0.29.32 | dacite | 1.8.1 |
| databricks-automl-runtime | 0.2.21 | databricks-feature-engineering | 0.4.0 | databricks-sdk | 0.20.0 |
| dataclasses-json | 0.6.4 | conjuntos de dados | 2.18.0 | dbl-tempo | 0.1.26 |
| dbus-python | 1.2.18 | debugpy | 1.6.7 | decorator | 5.1.1 |
| deepspeed | 0.14.0 | defusedxml | 0.7.1 | dill | 0.3.6 |
| diskcache | 5.6.3 | distlib | 0.3.8 | dm-tree | 0.1.8 |
| entrypoints | 0,4 | evaluate | 0.4.1 | executando | 0.8.3 |
| facets-overview | 1.1.1 | Farama-Notifications | 0.0.4 | fastjsonschema | 2.19.1 |
| fasttext | 0.9.2 | filelock | 3.9.0 | Flask | 2.2.5 |
| flatbuffers | 24.3.25 | fonttools | 4.25.0 | frozenlist | 1.3.3 |
| fsspec | 2023.5.0 | future | 0.18.3 | gast | 0.4.0 |
| gitdb | 4.0.11 | GitPython | 3.1.27 | google-api-core | 2.18.0 |
| google-auth | 2.21.0 | google-auth-oauthlib | 1.0.0 | google-cloud-core | 2.4.1 |
| google-cloud-storage | 2.10.0 | google-crc32c | 1.5.0 | google-pasta | 0.2.0 |
| google-resumable-media | 2.7.0 | googleapis-common-protos | 1.63.0 | greenlet | 2.0.1 |
| grpcio | 1.60.0 | grpcio-status | 1.60.0 | gunicorn | 20.1.0 |
| gviz-api | 1.10.0 | gymnasium | 0.28.1 | h11 | 0.14.0 |
| h5py | 3.10.0 | hjson | 3.1.0 | feriados | 0,45 |
| horovod | 0.28.1+db1 | htmlmin | 0.1.12 | httpcore | 1.0.5 |
| httplib2 | 0.20.2 | httpx | 0.27.0 | huggingface-hub | 0.20.2 |
| idna | 3.4 | ImageHash | 4.3.1 | imageio | 2.31.1 |
| imbalanced-learn | 0.11.0 | importlib-metadata | 6.0.0 | importlib_resources | 6.4.0 |
| ipyflow-core | 0.0.198 | ipykernel | 6.25.1 | ipython | 8.15.0 |
| ipython-genutils | 0.2.0 | ipywidgets | 8.0.4 | isodate | 0.6.1 |
| itsdangerous | 2.0.1 | jax-jumpy | 1.0.0 | jedi | 0.18.1 |
| jeepney | 0.7.1 | Jinja2 | 3.1.2 | jmespath | 0.10.0 |
| joblib | 1.2.0 | joblibspark | 0.5.1 | jsonpatch | 1.33 |
| jsonpointer | 2.4 | jsonschema | 4.17.3 | jupyter-server | 1.23.4 |
| jupyter_client | 7.4.9 | jupyter_core | 5.3.0 | jupyterlab-pygments | 0.1.2 |
| jupyterlab-widgets | 3.0.5 | keras | 3.1.1 | keyring | 23.5.0 |
| kiwisolver | 1.4.4 | langchain | 0.1.13 | langchain-community | 0.0.32 |
| langchain-core | 0.1.41 | langchain-text-splitters | 0.0.1 | langcodes | 3.3.0 |
| langsmith | 0.1.45 | launchpadlib | 1.10.16 | lazr.restfulclient | 0.14.4 |
| lazr.uri | 1.0.6 | lazy_loader | 0,2 | libclang | 15.0.6.1 |
| librosa | 0.10.1 | lightgbm | 4.3.0 | llvmlite | 0.40.0 |
| lxml | 4.9.2 | lz4 | 4.3.2 | Mako | 1.2.0 |
| Markdown | 3.4.1 | markdown-it-py | 2.2.0 | MarkupSafe | 2.1.1 |
| marshmallow | 3.21.1 | matplotlib | 3.7.2 | matplotlib-inline | 0.1.6 |
| mdurl | 0.1.0 | mistune | 0.8.4 | ml-dtypes | 0.3.2 |
| mlflow-skinny | 2.11.3 | more-itertools | 8.10.0 | mosaicml-streaming | 0.7.4 |
| mpmath | 1.3.0 | msal | 1.28.0 | msal-extensions | 1.1.0 |
| msgpack | 1.0.8 | multidict | 6.0.2 | multimethod | 1.11.2 |
| multiprocess | 0.70.14 | murmurhash | 1.0.10 | mypy-extensions | 0.4.3 |
| namex | 0.0.7 | nbclassic | 0.5.5 | nbclient | 0.5.13 |
| nbconvert | 6.5.4 | nbformat | 5.7.0 | nest-asyncio | 1.5.6 |
| networkx | 3.1 | ninja | 1.11.1.1 | nltk | 3.8.1 |
| notebook | 6.5.4 | notebook_shim | 0.2.2 | numba | 0.57.1 |
| numpy | 1.23.5 | oauthlib | 3.2.0 | oci | 2.125.2 |
| openai | 1.14.3 | opencensus | 0.11.4 | opencensus-context | 0.1.3 |
| opt-einsum | 3.3.0 | optree | 0.11.0 | orjson | 3.10.0 |
| empacotando | 23,2 | pandas | 1.5.3 | pandocfilters | 1.5.0 |
| paramiko | 3.4.0 | parso | 0.8.3 | pathspec | 0.10.3 |
| patsy | 0.5.3 | petastorm | 0.12.1 | pexpect | 4.8.0 |
| phik | 0.12.4 | pickleshare | 0.7.5 | Pillow | 9.4.0 |
| pip | 23.2.1 | platformdirs | 3.10.0 | plotly | 5.9.0 |
| pmdarima | 2.0.4 | pooch | 1.8.1 | portalocker | 2.8.2 |
| preshed | 3.0.9 | prometheus-client | 0.14.1 | prompt-toolkit | 3.0.36 |
| prophet | 1.1.5 | proto-plus | 1.23.0 | protobuf | 4.24.1 |
| psutil | 5.9.0 | psycopg2 | 2.9.3 | ptyprocess | 0.7.0 |
| pure-eval | 0.2.2 | py-cpuinfo | 8.0.0 | py-spy | 0.3.14 |
| pyarrow | 14.0.1 | pyarrow-hotfix | 0,6 | pyasn1 | 0.4.8 |
| pyasn1-modules | 0.2.8 | pybind11 | 2.12.0 | pyccolo | 0.0.52 |
| pycparser | 2.21 | pydantic | 1.10.6 | Pygments | 2.15.1 |
| PyGObject | 3.42.1 | PyJWT | 2.3.0 | PyNaCl | 1.5.0 |
| pynvml | 11.5.0 | pyodbc | 4.0.38 | pyOpenSSL | 23.2.0 |
| pyparsing | 3.0.9 | pyrsistent | 0.18.0 | pytesseract | 0.3.10 |
| python-dateutil | 2.8.2 | python-editor | 1.0.4 | python-lsp-jsonrpc | 1.1.1 |
| python-snappy | 0.6.1 | pytz | 2022.7 | PyWavelets | 1.4.1 |
| PyYAML | 6,0 | pyzmq | 23.2.0 | ray | 2.10.0 |
| regex | 2022.7.9 | solicitações | 2.31.0 | requests-oauthlib | 1.3.1 |
| responses | 0.13.3 | rich | 13.7.1 | rsa | 4.9 |
| s3transfer | 0.10.1 | safetensors | 0.4.2 | scikit-image | 0.20.0 |
| scikit-learn | 1.3.0 | scipy | 1.11.1 | seaborn | 0.12.2 |
| SecretStorage | 3.3.1 | Send2Trash | 1.8.0 | sentence-transformers | 2.6.1 |
| sentencepiece | 0.1.99 | setuptools | 68.0.0 | shap | 0.44.0 |
| simplejson | 3.17.6 | six | 1.16.0 | slicer | 0.0.7 |
| smart-open | 5.2.1 | smmap | 5.0.0 | sniffio | 1.2.0 |
| soundfile | 0.12.1 | soupsieve | 2.4 | soxr | 0.3.7 |
| spacy | 3.7.2 | spacy-legacy | 3.0.12 | spacy-loggers | 1.0.5 |
| spark-tensorflow-distributor | 1.0.0 | SQLAlchemy | 1.4.39 | sqlparse | 0.4.2 |
| srsly | 2.4.8 | ssh-import-id | 5.11 | stack-data | 0.2.0 |
| stanio | 0.5.0 | statsmodels | 0.14.0 | sympy | 1.11.1 |
| tangled-up-in-unicode | 0.2.0 | tenacity | 8.2.2 | tensorboard | 2.16.2 |
| tensorboard-data-server | 0.7.2 | tensorboard_plugin_profile | 2.15.1 | tensorboardX | 2.6.2.2 |
| tensorflow-cpu | 2.16.1 | tensorflow-estimator | 2.15.0 | tensorflow-io-gcs-filesystem | 0.36.0 |
| termcolor | 2.4.0 | terminado | 0.17.1 | thinc | 8.2.3 |
| threadpoolctl | 2.2.0 | tifffile | 2021.7.2 | tiktoken | 0.5.2 |
| tinycss2 | 1.2.1 | tokenize-rt | 4.2.1 | criadores de token | 0.15.0 |
| torch | 2.2.2+cpu | torcheval | 0.0.7 | torchvision | 0.17.2+cpu |
| tornado | 6.3.2 | tqdm | 4.65.0 | traitlets | 5.7.1 |
| transformers | 4.39.2 | typeguard | 2.13.3 | typer | 0.9.4 |
| typing-inspect | 0.9.0 | typing_extensions | 4.10.0 | tzdata | 2022.1 |
| ujson | 5.4.0 | unattended-upgrades | 0,1 | urllib3 | 1.26.16 |
| virtualenv | 20.21.0 | visions | 0.7.5 | wadllib | 1.3.6 |
| wasabi | 1.1.2 | wcwidth | 0.2.5 | weasel | 0.3.4 |
| webencodings | 0.5.1 | websocket-client | 0.58.0 | Werkzeug | 2.2.3 |
| wheel | 0.38.4 | widgetsnbextension | 4.0.5 | wordcloud | 1.9.3 |
| wrapt | 1.14.1 | xgboost | 2.0.3 | xxhash | 3.4.1 |
| yarl | 1.8.1 | ydata-profiling | 4.5.1 | zipp | 3.11.0 |
Bibliotecas do Python em clusters de GPU
| Biblioteca | Versão | Biblioteca | Versão | Biblioteca | Versão |
|---|---|---|---|---|---|
| absl-py | 1.0.0 | accelerate | 0.28.0 | aiohttp | 3.8.5 |
| aiohttp-cors | 0.7.0 | aiosignal | 1.2.0 | anyio | 3.5.0 |
| argon2-cffi | 21.3.0 | argon2-cffi-bindings | 21.2.0 | astor | 0.8.1 |
| asttokens | 2.0.5 | astunparse | 1.6.3 | async-timeout | 4.0.2 |
| attrs | 22.1.0 | audioread | 3.0.1 | azure-core | 1.30.1 |
| azure-cosmos | 4.3.1 | azure-identity | 1.16.0 | azure-storage-blob | 12.19.1 |
| azure-storage-file-datalake | 12.14.0 | backcall | 0.2.0 | bcrypt | 3.2.0 |
| beautifulsoup4 | 4.12.2 | black | 23.3.0 | bleach | 4.1.0 |
| blinker | 1.4 | blis | 0.7.11 | boto3 | 1.34.39 |
| botocore | 1.34.39 | Brotli | 1.0.9 | cachetools | 5.3.3 |
| catalogue | 2.0.10 | category-encoders | 2.6.3 | certifi | 2023.7.22 |
| cffi | 1.15.1 | chardet | 4.0.0 | charset-normalizer | 2.0.4 |
| circuitbreaker | 1.4.0 | clique | 8.0.4 | cloudpathlib | 0.16.0 |
| cloudpickle | 2.2.1 | cmdstanpy | 1.2.2 | colorido | 0.5.6 |
| comm | 0.1.2 | confecção | 0.1.4 | configparser | 5.2.0 |
| contourpy | 1.0.5 | criptografia | 41.0.3 | cycler | 0.11.0 |
| cymem | 2.0.8 | Cython | 0.29.32 | dacite | 1.8.1 |
| databricks-automl-runtime | 0.2.21 | databricks-feature-engineering | 0.4.0 | databricks-sdk | 0.20.0 |
| dataclasses-json | 0.6.4 | conjuntos de dados | 2.18.0 | dbl-tempo | 0.1.26 |
| dbus-python | 1.2.18 | debugpy | 1.6.7 | decorator | 5.1.1 |
| deepspeed | 0.14.0 | defusedxml | 0.7.1 | dill | 0.3.6 |
| diskcache | 5.6.3 | distlib | 0.3.8 | dm-tree | 0.1.8 |
| einops | 0.7.0 | entrypoints | 0,4 | evaluate | 0.4.1 |
| executando | 0.8.3 | facets-overview | 1.1.1 | Farama-Notifications | 0.0.4 |
| fastjsonschema | 2.19.1 | fasttext | 0.9.2 | filelock | 3.9.0 |
| flash-attn | 2.5.6 | Flask | 2.2.5 | flatbuffers | 24.3.25 |
| fonttools | 4.25.0 | frozenlist | 1.3.3 | fsspec | 2023.5.0 |
| future | 0.18.3 | gast | 0.4.0 | gitdb | 4.0.11 |
| GitPython | 3.1.27 | google-api-core | 2.18.0 | google-auth | 2.21.0 |
| google-auth-oauthlib | 1.0.0 | google-cloud-core | 2.4.1 | google-cloud-storage | 2.10.0 |
| google-crc32c | 1.5.0 | google-pasta | 0.2.0 | google-resumable-media | 2.7.0 |
| googleapis-common-protos | 1.63.0 | greenlet | 2.0.1 | grpcio | 1.60.0 |
| grpcio-status | 1.60.0 | gunicorn | 20.1.0 | gviz-api | 1.10.0 |
| gymnasium | 0.28.1 | h11 | 0.14.0 | h5py | 3.10.0 |
| hjson | 3.1.0 | feriados | 0,45 | horovod | 0.28.1+db1 |
| htmlmin | 0.1.12 | httpcore | 1.0.5 | httplib2 | 0.20.2 |
| httpx | 0.27.0 | huggingface-hub | 0.20.2 | idna | 3.4 |
| ImageHash | 4.3.1 | imageio | 2.31.1 | imbalanced-learn | 0.11.0 |
| importlib-metadata | 6.0.0 | importlib_resources | 6.4.0 | ipyflow-core | 0.0.198 |
| ipykernel | 6.25.1 | ipython | 8.15.0 | ipython-genutils | 0.2.0 |
| ipywidgets | 8.0.4 | isodate | 0.6.1 | itsdangerous | 2.0.1 |
| jax-jumpy | 1.0.0 | jedi | 0.18.1 | jeepney | 0.7.1 |
| Jinja2 | 3.1.2 | jmespath | 0.10.0 | joblib | 1.2.0 |
| joblibspark | 0.5.1 | jsonpatch | 1.33 | jsonpointer | 2.4 |
| jsonschema | 4.17.3 | jupyter-server | 1.23.4 | jupyter_client | 7.4.9 |
| jupyter_core | 5.3.0 | jupyterlab-pygments | 0.1.2 | jupyterlab-widgets | 3.0.5 |
| keras | 3.1.1 | keyring | 23.5.0 | kiwisolver | 1.4.4 |
| langchain | 0.1.13 | langchain-community | 0.0.32 | langchain-core | 0.1.41 |
| langchain-text-splitters | 0.0.1 | langcodes | 3.3.0 | langsmith | 0.1.45 |
| launchpadlib | 1.10.16 | lazr.restfulclient | 0.14.4 | lazr.uri | 1.0.6 |
| lazy_loader | 0,2 | libclang | 15.0.6.1 | librosa | 0.10.1 |
| lightgbm | 4.3.0 | llvmlite | 0.40.0 | lxml | 4.9.2 |
| lz4 | 4.3.2 | Mako | 1.2.0 | Markdown | 3.4.1 |
| markdown-it-py | 2.2.0 | MarkupSafe | 2.1.1 | marshmallow | 3.21.1 |
| matplotlib | 3.7.2 | matplotlib-inline | 0.1.6 | mdurl | 0.1.0 |
| mistune | 0.8.4 | ml-dtypes | 0.3.2 | mlflow-skinny | 2.11.3 |
| more-itertools | 8.10.0 | mosaicml-streaming | 0.7.4 | mpmath | 1.3.0 |
| msal | 1.28.0 | msal-extensions | 1.1.0 | msgpack | 1.0.8 |
| multidict | 6.0.2 | multimethod | 1.11.2 | multiprocess | 0.70.14 |
| murmurhash | 1.0.10 | mypy-extensions | 0.4.3 | namex | 0.0.7 |
| nbclassic | 0.5.5 | nbclient | 0.5.13 | nbconvert | 6.5.4 |
| nbformat | 5.7.0 | nest-asyncio | 1.5.6 | networkx | 3.1 |
| ninja | 1.11.1.1 | nltk | 3.8.1 | notebook | 6.5.4 |
| notebook_shim | 0.2.2 | numba | 0.57.1 | numpy | 1.23.5 |
| nvidia-cublas-cu12 | 12.1.3.1 | nvidia-cuda-cupti-cu12 | 12.1.105 | nvidia-cuda-nvrtc-cu12 | 12.1.105 |
| nvidia-cuda-runtime-cu12 | 12.1.105 | nvidia-cudnn-cu12 | 8.9.2.26 | nvidia-cufft-cu12 | 11.0.2.54 |
| nvidia-curand-cu12 | 10.3.2.106 | nvidia-cusolver-cu12 | 11.4.5.107 | nvidia-cusparse-cu12 | 12.1.0.106 |
| nvidia-nccl-cu12 | 2.19.3 | nvidia-nvjitlink-cu12 | 12.4.127 | nvidia-nvtx-cu12 | 12.1.105 |
| oauthlib | 3.2.0 | oci | 2.125.2 | openai | 1.14.3 |
| opencensus | 0.11.4 | opencensus-context | 0.1.3 | opt-einsum | 3.3.0 |
| optree | 0.11.0 | orjson | 3.10.0 | empacotando | 23,2 |
| pandas | 1.5.3 | pandocfilters | 1.5.0 | paramiko | 3.4.0 |
| parso | 0.8.3 | pathspec | 0.10.3 | patsy | 0.5.3 |
| petastorm | 0.12.1 | pexpect | 4.8.0 | phik | 0.12.4 |
| pickleshare | 0.7.5 | Pillow | 9.4.0 | pip | 23.2.1 |
| platformdirs | 3.10.0 | plotly | 5.9.0 | pmdarima | 2.0.4 |
| pooch | 1.8.1 | portalocker | 2.8.2 | preshed | 3.0.9 |
| prometheus-client | 0.14.1 | prompt-toolkit | 3.0.36 | prophet | 1.1.5 |
| proto-plus | 1.23.0 | protobuf | 4.24.1 | psutil | 5.9.0 |
| psycopg2 | 2.9.3 | ptyprocess | 0.7.0 | pure-eval | 0.2.2 |
| py-cpuinfo | 8.0.0 | py-spy | 0.3.14 | pyarrow | 14.0.1 |
| pyarrow-hotfix | 0,6 | pyasn1 | 0.4.8 | pyasn1-modules | 0.2.8 |
| pybind11 | 2.12.0 | pyccolo | 0.0.52 | pycparser | 2.21 |
| pydantic | 1.10.6 | Pygments | 2.15.1 | PyGObject | 3.42.1 |
| PyJWT | 2.3.0 | PyNaCl | 1.5.0 | pynvml | 11.5.0 |
| pyodbc | 4.0.38 | pyOpenSSL | 23.2.0 | pyparsing | 3.0.9 |
| pyrsistent | 0.18.0 | pytesseract | 0.3.10 | python-dateutil | 2.8.2 |
| python-editor | 1.0.4 | python-lsp-jsonrpc | 1.1.1 | python-snappy | 0.6.1 |
| pytz | 2022.7 | PyWavelets | 1.4.1 | PyYAML | 6,0 |
| pyzmq | 23.2.0 | ray | 2.10.0 | regex | 2022.7.9 |
| solicitações | 2.31.0 | requests-oauthlib | 1.3.1 | responses | 0.13.3 |
| rich | 13.7.1 | rsa | 4.9 | s3transfer | 0.10.1 |
| safetensors | 0.4.2 | scikit-image | 0.20.0 | scikit-learn | 1.3.0 |
| scipy | 1.11.1 | seaborn | 0.12.2 | SecretStorage | 3.3.1 |
| Send2Trash | 1.8.0 | sentence-transformers | 2.6.1 | sentencepiece | 0.1.99 |
| setuptools | 68.0.0 | shap | 0.44.0 | simplejson | 3.17.6 |
| six | 1.16.0 | slicer | 0.0.7 | smart-open | 5.2.1 |
| smmap | 5.0.0 | sniffio | 1.2.0 | soundfile | 0.12.1 |
| soupsieve | 2.4 | soxr | 0.3.7 | spacy | 3.7.2 |
| spacy-legacy | 3.0.12 | spacy-loggers | 1.0.5 | spark-tensorflow-distributor | 1.0.0 |
| SQLAlchemy | 1.4.39 | sqlparse | 0.4.2 | srsly | 2.4.8 |
| ssh-import-id | 5.11 | stack-data | 0.2.0 | stanio | 0.5.0 |
| statsmodels | 0.14.0 | sympy | 1.11.1 | tangled-up-in-unicode | 0.2.0 |
| tenacity | 8.2.2 | tensorboard | 2.16.2 | tensorboard-data-server | 0.7.2 |
| tensorboard_plugin_profile | 2.15.1 | tensorboardX | 2.6.2.2 | tensorflow | 2.16.1 |
| tensorflow-estimator | 2.15.0 | tensorflow-io-gcs-filesystem | 0.36.0 | termcolor | 2.4.0 |
| terminado | 0.17.1 | thinc | 8.2.3 | threadpoolctl | 2.2.0 |
| tifffile | 2021.7.2 | tiktoken | 0.5.2 | tinycss2 | 1.2.1 |
| tokenize-rt | 4.2.1 | criadores de token | 0.15.0 | torch | 2.2.2+cu121 |
| torcheval | 0.0.7 | torchvision | 0.17.2+cu121 | tornado | 6.3.2 |
| tqdm | 4.65.0 | traitlets | 5.7.1 | transformers | 4.39.2 |
| triton | 2.2.0 | typeguard | 2.13.3 | typer | 0.9.4 |
| typing-inspect | 0.9.0 | typing_extensions | 4.10.0 | tzdata | 2022.1 |
| ujson | 5.4.0 | unattended-upgrades | 0,1 | urllib3 | 1.26.16 |
| virtualenv | 20.21.0 | visions | 0.7.5 | wadllib | 1.3.6 |
| wasabi | 1.1.2 | wcwidth | 0.2.5 | weasel | 0.3.4 |
| webencodings | 0.5.1 | websocket-client | 0.58.0 | Werkzeug | 2.2.3 |
| wheel | 0.38.4 | widgetsnbextension | 4.0.5 | wordcloud | 1.9.3 |
| wrapt | 1.14.1 | xgboost | 2.0.3 | xxhash | 3.4.1 |
| yarl | 1.8.1 | ydata-profiling | 4.5.1 | zipp | 3.11.0 |
| zstd | 1.5.5.1 |
Bibliotecas do R
As bibliotecas R são idênticas às Bibliotecas R no Databricks Runtime 15.2.
Bibliotecas do Java e do Scala (cluster do Scala 2.12)
Além das bibliotecas Java e Scala no Databricks Runtime 15.2, o Databricks Runtime 15.2 ML contém os seguintes JARs:
Clusters de CPU
| ID do Grupo | Artifact ID | Versão |
|---|---|---|
| com.typesafe.akka | akka-actor_2.12 | 2.5.23 |
| ml.dmlc | xgboost4j-spark_2.12 | 1.7.3 |
| ml.dmlc | xgboost4j_2.12 | 1.7.3 |
| org.graphframes | graphframes_2.12 | 0.8.3-db1-spark3.5 |
| org.mlflow | mlflow-client | 2.11.1 |
| org.scala-lang.modules | scala-java8-compat_2.12 | 0.8.0 |
| org.tensorflow | spark-tensorflow-connector_2.12 | 1.15.0 |
Clusters de GPU
| ID do Grupo | Artifact ID | Versão |
|---|---|---|
| com.typesafe.akka | akka-actor_2.12 | 2.5.23 |
| ml.dmlc | xgboost4j-gpu_2.12 | 1.7.3 |
| ml.dmlc | xgboost4j-spark-gpu_2.12 | 1.7.3 |
| org.graphframes | graphframes_2.12 | 0.8.3-db1-spark3.5 |
| org.mlflow | mlflow-client | 2.11.1 |
| org.scala-lang.modules | scala-java8-compat_2.12 | 0.8.0 |
| org.tensorflow | spark-tensorflow-connector_2.12 | 1.15.0 |