azureml-dataset-runtime accepts old version of numpy only

Question

When creating custom environment, I use following conda file to install additional dependencies:

name: model-env
channels:
  - anaconda
  - conda-forge
dependencies:
  - python=3.11
  - pip
  - pip:
    - azureml-defaults==1.58.0
    - azure-storage-blob==12.11.0
    - azureml-inference-server-http
    - joblib==1.4.2
    - lightgbm==4.5.0
    - matplotlib==3.9.2
    - mlflow-skinny==2.16.0
    - numpy==1.23.5
    - pandas==2.2.3
    - scikit-learn==1.5.2
    - scipy==1.14.1
    - xgboost==2.1.1

to resolve dependencies, azureml is using azureml-dataset-runtime library, which accepts numpy versions of !=1.19.3,<1.24

See full dependences here https://www.wheelodex.org/projects/azureml-dataset-runtime/

This means that the latest numpy library that can be used in azureml environments is 1.23.5

This is a rather old version and is incompatible with newer versions of python like python 3.12. Additionally, some newer processors do not allow older versions of python that would accept the older versions of numpy. This makes packaging and deploying endpoints and pipelines to azureml from local machines very restrictive. Please consider upgrading this library and any related dependency.

Answer 1

Hi @Alexandra Posoldova,

Thank you for reaching out to Microsoft Q&A forum!

We understand that the current constraint of using numpy versions less than 1.24 poses limitations, especially when working with newer Python versions like 3.12. The Azure Machine Learning team is aware of these compatibility issues, and we encourage you to monitor any updates on the azureml-dataset-runtime package, as improvements to support newer versions of dependencies are typically released periodically. In the meantime, we suggest evaluating the potential of using earlier versions of Python or exploring alternative methods like custom Docker environments, which may offer more flexibility in resolving these dependency conflicts. Please also consider raising this issue through Azure support to provide further visibility into the matter, as it may influence future updates to the SDK.

Thank you.