다음을 통해 공유

ML용 Databricks Runtime 8.4(EoS)

  • 아티클

참고 항목

이 Databricks Runtime 버전에 대한 지원이 종료되었습니다. 지원 종료 날짜는 지원 종료 기록을 참조하세요. 지원되는 모든 Databricks Runtime 버전은 Databricks Runtime 릴리스 정보 버전 및 호환성을 참조하세요.

Databricks는 2021년 7월에 이 버전을 릴리스했습니다.

Machine Learning용 Databricks Runtime 8.4은 Databricks Runtime 8.4(EoS)을 기반으로 즉시 사용 가능한 기계 학습 및 데이터 과학 환경을 제공합니다. Databricks Runtime ML에는 TensorFlow, PyTorch 및 XGBoost를 포함하여 널리 사용되는 많은 기계 학습 라이브러리가 포함되어 있습니다. 또한 Horovod를 사용하여 분산 딥 러닝 학습을 지원합니다.

Databricks Runtime ML 클러스터 만들기 지침을 포함한 자세한 내용은 Databricks에서의 AI 및 기계 학습을 참조하세요.

새로운 기능 및 향상 기능

Databricks Runtime 8.4 ML은 Databricks Runtime 8.4를 기반으로 빌드되었습니다. Apache Spark MLlib 및 SparkR을 포함하여 Databricks Runtime 8.4의 새로운 기능에 대한 자세한 내용은 Databricks Runtime 8.4(EoS) 릴리스 정보를 참조하세요.

FeatureStoreClient v0.3.2

  • SQL 예약어와 충돌하는 기능 및 기능 테이블 이름을 허용합니다.
  • 제공된 DataFrame이 PySpark DataFrame(pyspark.sql.dataframe.DataFrame)인지 유효성 검사합니다.

AutoML v1.1.0

  • Databricks Runtime 8.4 ML과 함께 제공되는 업데이트된 버전의 AutoML에는 몇 가지 버그 수정 및 안정성 개선 사항이 포함되어 있습니다.
  • AutoML 분류는 이제 LGBMClassifier로 평가판을 실행합니다.
  • AutoML 회귀는 이제 LGBMRegressor를 사용하여 평가판도 실행합니다.

Databricks Runtime ML Python 환경의 주요 변경 내용

Databricks Runtime Python 환경의 주요 변경 내용은 Databricks Runtime 8.4(EoS)을 참조하세요. 설치된 Python 패키지 및 버전의 전체 목록은 Python 라이브러리를 참조하세요.

업그레이드된 Python 패키지

  • koalas 1.8.0 -> 1.8.1
  • horovod 0.21.3 -> 0.22.1
  • mleap 0.16.1 -> 0.17.0
  • mlflow 1.16.0 -> 1.18.0
  • pandas-profiling 2.11.0 -> 3.0.0
  • petastorm 0.10.0 -> 0.11.1
  • pytorch 1.8.1 -> 1.9.0
  • tensorboard 2.4.1 -> 2.5.0
  • tensorflow 2.4.1 -> 2.5.0
  • torchvision 0.9.1 -> 0.10.0
  • xgboost 1.4.1 -> 1.4.2

사용 중단

다음 변경 내용은 더 이상 사용되지 않으며 Databricks Runtime 9.0에서 제거됩니다.

  • HorovodRunner에서 np=0을 설정합니다. 여기서 np는 Horovod 작업에 사용할 병렬 프로세스 수입니다.
  • Intel MKL(Intel Math Kernel Library)과 그에 의존하는 패키지의 다운스트림 버전.
  • 핵심 Azure 예외 및 모듈용 azure-core python 라이브러리
  • Azure Storage Blob 서비스와 상호 작용하기 위한 azure-storage-blob python 라이브러리 클라이언트
  • AutoRest swagger 생성을 위한 msrest python 라이브러리
  • Docker Engine API용 docker python 라이브러리
  • Python/Django에서 쿼리 구문 분석을 위한 querystring-parser python 라이브러리
  • 멀티스레드 소프트웨어 만들기를 위한 intel-openmp python 라이브러리

시스템 환경

Databricks Runtime 8.4 ML의 시스템 환경은 다음과 같이 Databricks Runtime 8.4와 다릅니다.

라이브러리

다음 섹션에서는 Databricks Runtime 8.4에 포함된 라이브러리와 다른 Databricks Runtime 8.4 ML에 포함된 라이브러리를 나열합니다.

이 섹션의 내용:

최상위 계층 라이브러리

Databricks Runtime 8.4 ML에는 다음과 같은 최상위 라이브러리가 포함됩니다.

Python 라이브러리

Databricks Runtime 8.4 ML은 Python 패키지 관리에 Conda를 사용하며 많은 자주 사용되는 ML 패키지를 포함합니다.

다음 섹션의 Conda 환경에 지정된 패키지 외에도 Databricks Runtime 8.4 ML에는 다음 패키지도 포함됩니다.

  • hyperopt 0.2.5.db2
  • sparkdl 2.1.0.db4
  • feature_store 0.3.2
  • automl 1.1.0

CPU 클러스터의 Python 라이브러리

name: databricks-ml
channels:
  - pytorch
  - defaults
dependencies:
  - _libgcc_mutex=0.1=main
  - absl-py=0.11.0=pyhd3eb1b0_1
  - aiohttp=3.7.4=py38h27cfd23_1
  - asn1crypto=1.4.0=py_0
  - astor=0.8.1=py38h06a4308_0
  - async-timeout=3.0.1=py38h06a4308_0
  - attrs=20.3.0=pyhd3eb1b0_0
  - backcall=0.2.0=pyhd3eb1b0_0
  - bcrypt=3.2.0=py38h7b6447c_0
  - blas=1.0=mkl
  - blinker=1.4=py38h06a4308_0
  - boto3=1.16.7=pyhd3eb1b0_0
  - botocore=1.19.7=pyhd3eb1b0_0
  - brotlipy=0.7.0=py38h27cfd23_1003
  - bzip2=1.0.8=h7b6447c_0
  - ca-certificates=2021.5.25=h06a4308_1
  - cachetools=4.2.2=pyhd3eb1b0_0
  - certifi=2021.5.30=py38h06a4308_0
  - cffi=1.14.3=py38h261ae71_2
  - chardet=3.0.4=py38h06a4308_1003
  - click=7.1.2=pyhd3eb1b0_0
  - cloudpickle=1.6.0=py_0
  - configparser=5.0.1=py_0
  - cpuonly=1.0=0
  - cryptography=3.1.1=py38h1ba5d50_0
  - cycler=0.10.0=py38_0
  - cython=0.29.21=py38h2531618_0
  - decorator=4.4.2=pyhd3eb1b0_0
  - dill=0.3.2=py_0
  - docutils=0.15.2=py38h06a4308_1
  - entrypoints=0.3=py38_0
  - ffmpeg=4.2.2=h20bf706_0
  - flask=1.1.2=pyhd3eb1b0_0
  - freetype=2.10.4=h5ab3b9f_0
  - fsspec=0.8.3=py_0
  - future=0.18.2=py38_1
  - gast=0.4.0=py_0
  - gitdb=4.0.7=pyhd3eb1b0_0
  - gitpython=3.1.12=pyhd3eb1b0_1
  - gmp=6.1.2=h6c8ec71_1
  - gnutls=3.6.15=he1e5248_0
  - google-auth=1.22.1=py_0
  - google-auth-oauthlib=0.4.2=pyhd3eb1b0_2
  - google-pasta=0.2.0=py_0
  - gunicorn=20.0.4=py38h06a4308_0
  - hdf5=1.10.4=hb1b8bf9_0
  - icu=58.2=he6710b0_3
  - idna=2.10=pyhd3eb1b0_0
  - importlib-metadata=2.0.0=py_1
  - intel-openmp=2019.4=243
  - ipykernel=5.3.4=py38h5ca1d4c_0
  - ipython=7.19.0=py38hb070fc8_1
  - ipython_genutils=0.2.0=pyhd3eb1b0_1
  - isodate=0.6.0=py_1
  - itsdangerous=1.1.0=pyhd3eb1b0_0
  - jedi=0.17.2=py38h06a4308_1
  - jinja2=2.11.2=pyhd3eb1b0_0
  - jmespath=0.10.0=py_0
  - joblib=0.17.0=py_0
  - jpeg=9b=h024ee3a_2
  - jupyter_client=6.1.7=py_0
  - jupyter_core=4.6.3=py38_0
  - kiwisolver=1.3.0=py38h2531618_0
  - krb5=1.17.1=h173b8e3_0
  - lame=3.100=h7b6447c_0
  - lcms2=2.11=h396b838_0
  - ld_impl_linux-64=2.33.1=h53a641e_7
  - libedit=3.1.20191231=h14c3975_1
  - libffi=3.3=he6710b0_2
  - libgcc-ng=9.1.0=hdf63c60_0
  - libgfortran-ng=7.3.0=hdf63c60_0
  - libidn2=2.3.1=h27cfd23_0
  - libopus=1.3.1=h7b6447c_0
  - libpng=1.6.37=hbc83047_0
  - libpq=12.2=h20c2e04_0
  - libprotobuf=3.13.0.1=hd408876_0
  - libsodium=1.0.18=h7b6447c_0
  - libstdcxx-ng=9.1.0=hdf63c60_0
  - libtasn1=4.16.0=h27cfd23_0
  - libtiff=4.1.0=h2733197_1
  - libunistring=0.9.10=h27cfd23_0
  - libuv=1.40.0=h7b6447c_0
  - libvpx=1.7.0=h439df22_0
  - lightgbm=3.1.1=py38h2531618_0
  - lz4-c=1.9.2=heb0550a_3
  - mako=1.1.3=py_0
  - markdown=3.3.3=py38h06a4308_0
  - markupsafe=1.1.1=py38h7b6447c_0
  - matplotlib-base=3.2.2=py38hef1b27d_0
  - mkl=2019.4=243
  - mkl-service=2.3.0=py38he904b0f_0
  - mkl_fft=1.2.0=py38h23d657b_0
  - mkl_random=1.1.0=py38h962f231_0
  - more-itertools=8.6.0=pyhd3eb1b0_0
  - multidict=5.1.0=py38h27cfd23_2
  - ncurses=6.2=he6710b0_1
  - nettle=3.7.3=hbbd107a_1
  - networkx=2.5.1=pyhd3eb1b0_0
  - ninja=1.10.2=hff7bd54_1
  - nltk=3.5=py_0
  - numpy=1.19.2=py38h54aff64_0
  - numpy-base=1.19.2=py38hfa32c7d_0
  - oauthlib=3.1.0=py_0
  - olefile=0.46=py_0
  - openh264=2.1.0=hd408876_0
  - openssl=1.1.1k=h27cfd23_0
  - packaging=20.4=py_0
  - pandas=1.1.5=py38ha9443f7_0
  - paramiko=2.7.2=py_0
  - parso=0.7.0=py_0
  - patsy=0.5.1=py38_0
  - pexpect=4.8.0=pyhd3eb1b0_3
  - pickleshare=0.7.5=pyhd3eb1b0_1003
  - pillow=8.0.1=py38he98fc37_0
  - pip=20.2.4=py38h06a4308_0
  - plotly=4.14.3=pyhd3eb1b0_0
  - prompt-toolkit=3.0.8=py_0
  - prompt_toolkit=3.0.8=0
  - protobuf=3.13.0.1=py38he6710b0_1
  - psutil=5.7.2=py38h7b6447c_0
  - psycopg2=2.8.5=py38h3c74f83_1
  - ptyprocess=0.6.0=pyhd3eb1b0_2
  - pyasn1=0.4.8=py_0
  - pyasn1-modules=0.2.8=py_0
  - pycparser=2.20=py_2
  - pygments=2.7.2=pyhd3eb1b0_0
  - pyjwt=1.7.1=py38_0
  - pynacl=1.4.0=py38h7b6447c_1
  - pyodbc=4.0.30=py38he6710b0_0
  - pyopenssl=19.1.0=pyhd3eb1b0_1
  - pyparsing=2.4.7=pyhd3eb1b0_0
  - pysocks=1.7.1=py38h06a4308_0
  - python=3.8.8=hdb3f193_4
  - python-dateutil=2.8.1=pyhd3eb1b0_0
  - python-editor=1.0.4=py_0
  - pytorch=1.9.0=py3.8_cpu_0
  - pytz=2020.5=pyhd3eb1b0_0
  - pyzmq=19.0.2=py38he6710b0_1
  - readline=8.0=h7b6447c_0
  - regex=2020.10.15=py38h7b6447c_0
  - requests=2.24.0=py_0
  - requests-oauthlib=1.3.0=py_0
  - retrying=1.3.3=py_2
  - rsa=4.7.2=pyhd3eb1b0_1
  - s3transfer=0.3.6=pyhd3eb1b0_0
  - scikit-learn=0.23.2=py38h0573a6f_0
  - scipy=1.5.2=py38h0b6359f_0
  - setuptools=50.3.1=py38h06a4308_1
  - simplejson=3.17.2=py38h27cfd23_2
  - six=1.15.0=py38h06a4308_0
  - smmap=3.0.5=pyhd3eb1b0_0
  - sqlite=3.33.0=h62c20be_0
  - sqlparse=0.4.1=py_0
  - statsmodels=0.12.0=py38h7b6447c_0
  - tabulate=0.8.7=py38h06a4308_0
  - threadpoolctl=2.1.0=pyh5ca1d4c_0
  - tk=8.6.10=hbc83047_0
  - torchvision=0.10.0=py38_cpu
  - tornado=6.0.4=py38h7b6447c_1
  - tqdm=4.50.2=py_0
  - traitlets=5.0.5=pyhd3eb1b0_0
  - typing-extensions=3.7.4.3=hd3eb1b0_0
  - typing_extensions=3.7.4.3=pyh06a4308_0
  - unixodbc=2.3.9=h7b6447c_0
  - urllib3=1.25.11=py_0
  - wcwidth=0.2.5=py_0
  - websocket-client=0.57.0=py38_2
  - werkzeug=1.0.1=pyhd3eb1b0_0
  - wheel=0.35.1=pyhd3eb1b0_0
  - wrapt=1.12.1=py38h7b6447c_1
  - x264=1!157.20191217=h7b6447c_0
  - xz=5.2.5=h7b6447c_0
  - yarl=1.6.3=py38h27cfd23_0
  - zeromq=4.3.3=he6710b0_3
  - zipp=3.4.0=pyhd3eb1b0_0
  - zlib=1.2.11=h7b6447c_3
  - zstd=1.4.5=h9ceee32_0
  - pip:
    - argon2-cffi==20.1.0
    - astunparse==1.6.3
    - async-generator==1.10
    - azure-core==1.11.0
    - azure-storage-blob==12.7.1
    - bleach==3.3.0
    - bottleneck==1.3.2
    - convertdate==2.3.2
    - databricks-cli==0.14.3
    - defusedxml==0.7.1
    - diskcache==5.2.1
    - docker==4.4.4
    - facets-overview==1.0.0
    - flatbuffers==1.12
    - grpcio==1.34.1
    - h5py==3.1.0
    - hijri-converter==2.1.3
    - holidays==0.10.5.2
    - horovod==0.22.1
    - htmlmin==0.1.12
    - imagehash==4.2.0
    - ipywidgets==7.6.3
    - joblibspark==0.3.0
    - jsonschema==3.2.0
    - jupyterlab-pygments==0.1.2
    - jupyterlab-widgets==1.0.0
    - keras-nightly==2.5.0.dev2021032900
    - keras-preprocessing==1.1.2
    - koalas==1.8.1
    - korean-lunar-calendar==0.2.1
    - llvmlite==0.36.0
    - missingno==0.4.2
    - mistune==0.8.4
    - mleap==0.17.0
    - mlflow-skinny==1.18.0
    - msrest==0.6.21
    - multimethod==1.4
    - nbclient==0.5.3
    - nbconvert==6.1.0
    - nbformat==5.1.3
    - nest-asyncio==1.5.1
    - notebook==6.4.0
    - numba==0.53.1
    - opt-einsum==3.3.0
    - pandas-profiling==3.0.0
    - pandocfilters==1.4.3
    - petastorm==0.11.1
    - phik==0.11.2
    - prometheus-client==0.11.0
    - pyarrow==1.0.1
    - pydantic==1.8.2
    - pymeeus==0.5.11
    - pyrsistent==0.18.0
    - pywavelets==1.1.1
    - pyyaml==5.4.1
    - querystring-parser==1.2.4
    - seaborn==0.10.0
    - send2trash==1.7.1
    - shap==0.39.0
    - slicer==0.0.7
    - spark-tensorflow-distributor==0.1.0
    - tangled-up-in-unicode==0.1.0
    - tensorboard==2.5.0
    - tensorboard-data-server==0.6.1
    - tensorboard-plugin-wit==1.8.0
    - tensorflow-cpu==2.5.0
    - tensorflow-estimator==2.5.0
    - termcolor==1.1.0
    - terminado==0.10.1
    - testpath==0.5.0
    - visions==0.7.1
    - webencodings==0.5.1
    - widgetsnbextension==3.5.1
    - xgboost==1.4.2
prefix: /databricks/conda/envs/databricks-ml

GPU 클러스터의 Python 라이브러리

name: databricks-ml-gpu
channels:
  - defaults
dependencies:
  - _libgcc_mutex=0.1=main
  - absl-py=0.11.0=pyhd3eb1b0_1
  - aiohttp=3.7.4=py38h27cfd23_1
  - asn1crypto=1.4.0=py_0
  - astor=0.8.1=py38h06a4308_0
  - async-timeout=3.0.1=py38h06a4308_0
  - attrs=20.3.0=pyhd3eb1b0_0
  - backcall=0.2.0=pyhd3eb1b0_0
  - bcrypt=3.2.0=py38h7b6447c_0
  - blas=1.0=mkl
  - blinker=1.4=py38h06a4308_0
  - boto3=1.16.7=pyhd3eb1b0_0
  - botocore=1.19.7=pyhd3eb1b0_0
  - brotlipy=0.7.0=py38h27cfd23_1003
  - ca-certificates=2021.5.25=h06a4308_1
  - cachetools=4.2.2=pyhd3eb1b0_0
  - certifi=2021.5.30=py38h06a4308_0
  - cffi=1.14.3=py38h261ae71_2
  - chardet=3.0.4=py38h06a4308_1003
  - click=7.1.2=pyhd3eb1b0_0
  - cloudpickle=1.6.0=py_0
  - configparser=5.0.1=py_0
  - cryptography=3.1.1=py38h1ba5d50_0
  - cycler=0.10.0=py38_0
  - cython=0.29.21=py38h2531618_0
  - decorator=4.4.2=pyhd3eb1b0_0
  - dill=0.3.2=py_0
  - docutils=0.15.2=py38h06a4308_1
  - entrypoints=0.3=py38_0
  - flask=1.1.2=pyhd3eb1b0_0
  - freetype=2.10.4=h5ab3b9f_0
  - fsspec=0.8.3=py_0
  - future=0.18.2=py38_1
  - gast=0.4.0=py_0
  - gitdb=4.0.7=pyhd3eb1b0_0
  - gitpython=3.1.12=pyhd3eb1b0_1
  - google-auth=1.22.1=py_0
  - google-auth-oauthlib=0.4.2=pyhd3eb1b0_2
  - google-pasta=0.2.0=py_0
  - gunicorn=20.0.4=py38h06a4308_0
  - hdf5=1.10.4=hb1b8bf9_0
  - icu=58.2=he6710b0_3
  - idna=2.10=pyhd3eb1b0_0
  - importlib-metadata=2.0.0=py_1
  - intel-openmp=2019.4=243
  - ipykernel=5.3.4=py38h5ca1d4c_0
  - ipython=7.19.0=py38hb070fc8_1
  - ipython_genutils=0.2.0=pyhd3eb1b0_1
  - isodate=0.6.0=py_1
  - itsdangerous=1.1.0=pyhd3eb1b0_0
  - jedi=0.17.2=py38h06a4308_1
  - jinja2=2.11.2=pyhd3eb1b0_0
  - jmespath=0.10.0=py_0
  - joblib=0.17.0=py_0
  - jpeg=9b=h024ee3a_2
  - jupyter_client=6.1.7=py_0
  - jupyter_core=4.6.3=py38_0
  - kiwisolver=1.3.0=py38h2531618_0
  - krb5=1.17.1=h173b8e3_0
  - lcms2=2.11=h396b838_0
  - ld_impl_linux-64=2.33.1=h53a641e_7
  - libedit=3.1.20191231=h14c3975_1
  - libffi=3.3=he6710b0_2
  - libgcc-ng=9.1.0=hdf63c60_0
  - libgfortran-ng=7.3.0=hdf63c60_0
  - libpng=1.6.37=hbc83047_0
  - libpq=12.2=h20c2e04_0
  - libprotobuf=3.13.0.1=hd408876_0
  - libsodium=1.0.18=h7b6447c_0
  - libstdcxx-ng=9.1.0=hdf63c60_0
  - libtiff=4.1.0=h2733197_1
  - lightgbm=3.1.1=py38h2531618_0
  - lz4-c=1.9.2=heb0550a_3
  - mako=1.1.3=py_0
  - markdown=3.3.3=py38h06a4308_0
  - markupsafe=1.1.1=py38h7b6447c_0
  - matplotlib-base=3.2.2=py38hef1b27d_0
  - mkl=2019.4=243
  - mkl-service=2.3.0=py38he904b0f_0
  - mkl_fft=1.2.0=py38h23d657b_0
  - mkl_random=1.1.0=py38h962f231_0
  - more-itertools=8.6.0=pyhd3eb1b0_0
  - multidict=5.1.0=py38h27cfd23_2
  - ncurses=6.2=he6710b0_1
  - networkx=2.5.1=pyhd3eb1b0_0
  - nltk=3.5=py_0
  - numpy=1.19.2=py38h54aff64_0
  - numpy-base=1.19.2=py38hfa32c7d_0
  - oauthlib=3.1.0=py_0
  - olefile=0.46=py_0
  - openssl=1.1.1k=h27cfd23_0
  - packaging=20.4=py_0
  - pandas=1.1.5=py38ha9443f7_0
  - paramiko=2.7.2=py_0
  - parso=0.7.0=py_0
  - patsy=0.5.1=py38_0
  - pexpect=4.8.0=pyhd3eb1b0_3
  - pickleshare=0.7.5=pyhd3eb1b0_1003
  - pillow=8.0.1=py38he98fc37_0
  - pip=20.2.4=py38h06a4308_0
  - plotly=4.14.3=pyhd3eb1b0_0
  - prompt-toolkit=3.0.8=py_0
  - prompt_toolkit=3.0.8=0
  - protobuf=3.13.0.1=py38he6710b0_1
  - psutil=5.7.2=py38h7b6447c_0
  - psycopg2=2.8.5=py38h3c74f83_1
  - ptyprocess=0.6.0=pyhd3eb1b0_2
  - pyasn1=0.4.8=py_0
  - pyasn1-modules=0.2.8=py_0
  - pycparser=2.20=py_2
  - pygments=2.7.2=pyhd3eb1b0_0
  - pyjwt=1.7.1=py38_0
  - pynacl=1.4.0=py38h7b6447c_1
  - pyodbc=4.0.30=py38he6710b0_0
  - pyopenssl=19.1.0=pyhd3eb1b0_1
  - pyparsing=2.4.7=pyhd3eb1b0_0
  - pysocks=1.7.1=py38h06a4308_0
  - python=3.8.8=hdb3f193_4
  - python-dateutil=2.8.1=pyhd3eb1b0_0
  - python-editor=1.0.4=py_0
  - pytz=2020.5=pyhd3eb1b0_0
  - pyzmq=19.0.2=py38he6710b0_1
  - readline=8.0=h7b6447c_0
  - regex=2020.10.15=py38h7b6447c_0
  - requests=2.24.0=py_0
  - requests-oauthlib=1.3.0=py_0
  - retrying=1.3.3=py_2
  - rsa=4.7.2=pyhd3eb1b0_1
  - s3transfer=0.3.6=pyhd3eb1b0_0
  - scikit-learn=0.23.2=py38h0573a6f_0
  - scipy=1.5.2=py38h0b6359f_0
  - setuptools=50.3.1=py38h06a4308_1
  - simplejson=3.17.2=py38h27cfd23_2
  - six=1.15.0=py38h06a4308_0
  - smmap=3.0.5=pyhd3eb1b0_0
  - sqlite=3.33.0=h62c20be_0
  - sqlparse=0.4.1=py_0
  - statsmodels=0.12.0=py38h7b6447c_0
  - tabulate=0.8.7=py38h06a4308_0
  - threadpoolctl=2.1.0=pyh5ca1d4c_0
  - tk=8.6.10=hbc83047_0
  - tornado=6.0.4=py38h7b6447c_1
  - tqdm=4.50.2=py_0
  - traitlets=5.0.5=pyhd3eb1b0_0
  - typing-extensions=3.7.4.3=hd3eb1b0_0
  - typing_extensions=3.7.4.3=pyh06a4308_0
  - unixodbc=2.3.9=h7b6447c_0
  - urllib3=1.25.11=py_0
  - wcwidth=0.2.5=py_0
  - websocket-client=0.57.0=py38_2
  - werkzeug=1.0.1=pyhd3eb1b0_0
  - wheel=0.35.1=pyhd3eb1b0_0
  - wrapt=1.12.1=py38h7b6447c_1
  - xz=5.2.5=h7b6447c_0
  - yarl=1.6.3=py38h27cfd23_0
  - zeromq=4.3.3=he6710b0_3
  - zipp=3.4.0=pyhd3eb1b0_0
  - zlib=1.2.11=h7b6447c_3
  - zstd=1.4.5=h9ceee32_0
  - pip:
    - argon2-cffi==20.1.0
    - astunparse==1.6.3
    - async-generator==1.10
    - azure-core==1.11.0
    - azure-storage-blob==12.7.1
    - bleach==3.3.0
    - bottleneck==1.3.2
    - convertdate==2.3.2
    - databricks-cli==0.14.3
    - defusedxml==0.7.1
    - diskcache==5.2.1
    - docker==4.4.4
    - facets-overview==1.0.0
    - flatbuffers==1.12
    - grpcio==1.34.1
    - h5py==3.1.0
    - hijri-converter==2.1.3
    - holidays==0.10.5.2
    - horovod==0.22.1
    - htmlmin==0.1.12
    - imagehash==4.2.0
    - ipywidgets==7.6.3
    - joblibspark==0.3.0
    - jsonschema==3.2.0
    - jupyterlab-pygments==0.1.2
    - jupyterlab-widgets==1.0.0
    - keras-nightly==2.5.0.dev2021032900
    - keras-preprocessing==1.1.2
    - koalas==1.8.1
    - korean-lunar-calendar==0.2.1
    - llvmlite==0.36.0
    - missingno==0.4.2
    - mistune==0.8.4
    - mleap==0.17.0
    - mlflow-skinny==1.18.0
    - msrest==0.6.21
    - multimethod==1.4
    - nbclient==0.5.3
    - nbconvert==6.1.0
    - nbformat==5.1.3
    - nest-asyncio==1.5.1
    - notebook==6.4.0
    - numba==0.53.1
    - opt-einsum==3.3.0
    - pandas-profiling==3.0.0
    - pandocfilters==1.4.3
    - petastorm==0.11.1
    - phik==0.11.2
    - pyarrow==1.0.1
    - pydantic==1.8.2
    - pymeeus==0.5.11
    - pyrsistent==0.17.3
    - pywavelets==1.1.1
    - pyyaml==5.4.1
    - querystring-parser==1.2.4
    - seaborn==0.10.0
    - send2trash==1.7.1
    - shap==0.39.0
    - slicer==0.0.7
    - spark-tensorflow-distributor==0.1.0
    - tangled-up-in-unicode==0.1.0
    - tensorboard==2.5.0
    - tensorboard-data-server==0.6.1
    - tensorboard-plugin-wit==1.8.0
    - tensorflow==2.5.0
    - tensorflow-estimator==2.5.0
    - termcolor==1.1.0
    - terminado==0.10.1
    - testpath==0.5.0
    - torch==1.9.0
    - torchvision==0.10.0
    - visions==0.7.1
    - webencodings==0.5.1
    - widgetsnbextension==3.5.1
    - xgboost==1.4.2
prefix: /databricks/conda/envs/databricks-ml-gpu

Python 모듈이 포함된 Spark 패키지

Spark 패키지 Python 모듈 버전
graphframes graphframes 0.8.1-db3-spark3.1

R 라이브러리

R 라이브러리는 Databricks Runtime 8.4의 R 라이브러리와 동일합니다.

Java 및 Scala 라이브러리(Scala 2.12 클러스터)

Databricks Runtime 8.4의 Java 및 Scala 라이브러리 외에도 Databricks Runtime 8.4 ML에는 다음 JAR이 포함되어 있습니다.

CPU 클러스터

그룹 ID 아티팩트 ID 버전
com.typesafe.akka akka-actor_2.12 2.5.23
ml.combust.mleap mleap-databricks-runtime_2.12 0.17.3-4882dc3
ml.dmlc xgboost4j-spark_2.12 1.4.1
ml.dmlc xgboost4j_2.12 1.4.1
org.mlflow mlflow-client 1.18.0
org.scala-lang.modules scala-java8-compat_2.12 0.8.0
org.tensorflow spark-tensorflow-connector_2.12 1.15.0

GPU 클러스터

그룹 ID 아티팩트 ID 버전
com.typesafe.akka akka-actor_2.12 2.5.23
ml.combust.mleap mleap-databricks-runtime_2.12 0.17.3-4882dc3
ml.dmlc xgboost4j-spark-gpu_2.12 1.4.1
ml.dmlc xgboost4j-gpu_2.12 1.4.1
org.mlflow mlflow-client 1.18.0
org.scala-lang.modules scala-java8-compat_2.12 0.8.0
org.tensorflow spark-tensorflow-connector_2.12 1.15.0