Model evaluation component/environment uses python 3.8 while automl-generated mlflow models require python 3.9

Question

Model evaluation component/environment uses python 3.8 while automl-generated mlflow models require python 3.9

Elie Ayoub 5

Hello,

I am facing an issue when trying to use the Model Prediction component in the designer to test a registered mlflow model that I generated with automl.

Upon further inspection in the logs, there were many errors regarding not finding the right version of certain packages that require python 3.9.

The Model Prediction component uses the model-evaluation environment which specifies the python version to be 3.8 in its conda file.

Is this an issue that needs to be fixed or am I doing something wrong?

Thank you.

YutongTie-MSFT 53,966 Reputation points Moderator

2024-08-02T00:09:53.8333333+00:00

Hello Elie,

Thanks for reaching out to us, could you please share more details about how you build you AutoML? Is there any document you are referring to so that we can try to reproduce your error?

Regards,

Yutong

Elie Ayoub 5

Hello @YutongTie-MSFT ,

Thank you for your response.

I am simply training an instance segmentation model by running:

instance_segmentation_job = automl.image_instance_segmentation(experiment_name="automl-experiment", training_data=training_input, validation_data_size=0.1, target_column_name="label", compute="ml-gpu-cluster")

training_job = ml_client.jobs.create_or_update(instance_segmentation_job)

then registering the output model from the job's output folder.

I then go to Designer and try to use the Model Prediction component to run inference on a test dataset with the registered model.

In the std_log.txt of model_prediction I get multiple errors that look like this:

ERROR: Could not find a version that satisfies the requirement ipython==8.18.1 (from versions: 0.10, 0.10.1, 0.10.2, 0.11, 0.12, 0.12.1, 0.13, 0.13.1, 0.13.2, 1.0.0, 1.1.0, 1.2.0, 1.2.1, 2.0.0, 2.1.0, 2.2.0, 2.3.0, 2.3.1, 2.4.0, 2.4.1, 3.0.0, 3.1.0, 3.2.0, 3.2.1, 3.2.2, 3.2.3, 4.0.0b1, 4.0.0, 4.0.1, 4.0.2, 4.0.3, 4.1.0rc1, 4.1.0rc2, 4.1.0, 4.1.1, 4.1.2, 4.2.0, 4.2.1, 5.0.0b1, 5.0.0b2, 5.0.0b3, 5.0.0b4, 5.0.0rc1, 5.0.0, 5.1.0, 5.2.0, 5.2.1, 5.2.2, 5.3.0, 5.4.0, 5.4.1, 5.5.0, 5.6.0, 5.7.0, 5.8.0, 5.9.0, 5.10.0, 6.0.0rc1, 6.0.0, 6.1.0, 6.2.0, 6.2.1, 6.3.0, 6.3.1, 6.4.0, 6.5.0, 7.0.0b1, 7.0.0rc1, 7.0.0, 7.0.1, 7.1.0, 7.1.1, 7.2.0, 7.3.0, 7.4.0, 7.5.0, 7.6.0, 7.6.1, 7.7.0, 7.8.0, 7.9.0, 7.10.0, 7.10.1, 7.10.2, 7.11.0, 7.11.1, 7.12.0, 7.13.0, 7.14.0, 7.15.0, 7.16.0, 7.16.1, 7.16.2, 7.16.3, 7.17.0, 7.18.0, 7.18.1, 7.19.0, 7.20.0, 7.21.0, 7.22.0, 7.23.0, 7.23.1, 7.24.0, 7.24.1, 7.25.0, 7.26.0, 7.27.0, 7.28.0, 7.29.0, 7.30.0, 7.30.1, 7.31.0, 7.31.1, 7.32.0, 7.33.0, 7.34.0, 8.0.0a1, 8.0.0b1, 8.0.0rc1, 8.0.0, 8.0.1, 8.1.0, 8.1.1, 8.2.0, 8.3.0, 8.4.0, 8.5.0, 8.6.0, 8.7.0, 8.8.0, 8.9.0, 8.10.0, 8.11.0, 8.12.0, 8.12.1, 8.12.2, 8.12.3, 8.13.0)
ERROR: No matching distribution found for ipython==8.18.1

The required python package versions are not available in python 3.8 and require python 3.9. the Model Prediction component uses the following environment azureml://registries/azureml/environments/model-evaluation/versions/32 which specifies the python version to be 3.8.

I also get the following warning:

2024/08/01 15:09:01 WARNING azureml.evaluate.mlflow.aml: The version of Python that the model was saved in, `Python 3.9.7`, differs from the version of Python that is currently running, `Python 3.8.19`, and may be incompatible

and finally, the execution fails with the following returned error:

2024/08/01 15:09:01 ERROR azureml.evaluate.mlflow.aml: Error while loading the models.
2024-08-01 15:09:01,561 [model_evaluation] [39875483-28ff-4f32-8340-d5b7fb7fd215] [logging_utilities] _log_traceback +415: ERROR                [14] Type: SystemError
Code: {'code': 'ClientError'}
Class: ModelLoadingException
Message: Model load failed due to error: [ImportError("cannot import name '_onnx_opset_version' from 'torchvision.ops._register_onnx_ops' (/azureml-envs/model-evaluation/lib/python3.8/site-packages/torchvision/ops/_register_onnx_ops.py)")]
Traceback:   File "model_prediction.py", line 103, in __init__
    self.predictor = predictor_cls(self.model_uri, self.task, self.device)
  File "base.py", line 30, in __init__
    self.model, self.model_flavor = load_model(model_uri=model_uri, device=device, task=task_type)
  File "utils_load.py", line 82, in load_model
    model = aml_mlflow.aml.load_model(model_uri=model_uri,
  File "__init__.py", line 310, in load_model
    raise e1
  File "__init__.py", line 307, in load_model
    model_impl = importlib.import_module(conf[MAIN])._load_pyfunc(data_path)
  File "model.py", line 490, in _load_pyfunc
    context, python_model, signature = _load_context_model_and_signature(model_path, model_config)
  File "model.py", line 472, in _load_context_model_and_signature
    python_model = cloudpickle.load(f)
  File "score.py", line 36, in <module>
    from ..common.object_detection_utils import (_fetch_model_from_artifacts,
  File "object_detection_utils.py", line 19, in <module>
    from azureml.automl.dnn.vision.common.artifacts_utils import _download_model_from_artifacts
  File "artifacts_utils.py", line 20, in <module>
    from azureml.automl.dnn.vision.common.model_export_utils import prepare_model_export
  File "model_export_utils.py", line 34, in <module>
    from azureml.automl.dnn.vision.object_detection.models.base_model_wrapper import \
  File "base_model_wrapper.py", line 14, in <module>
    from torchvision.ops._register_onnx_ops import _onnx_opset_version

ExceptionTarget: None 
ERROR:__main__:Type: SystemError
Class: ModelLoadingException
Message: ModelLoadingException:
	Message: Model load failed due to error: [ImportError("cannot import name '_onnx_opset_version' from 'torchvision.ops._register_onnx_ops' (/azureml-envs/model-evaluation/lib/python3.8/site-packages/torchvision/ops/_register_onnx_ops.py)")]
	InnerException: ImportError
	ErrorResponse 
{
    "error": {
        "code": "SystemError",
        "message": "Model load failed due to error: [ImportError(\"cannot import name '_onnx_opset_version' from 'torchvision.ops._register_onnx_ops' (/azureml-envs/model-evaluation/lib/python3.8/site-packages/torchvision/ops/_register_onnx_ops.py)\")]",
        "inner_error": {
            "code": "ClientError"
        }
    }
}

Thank you for looking into this. I hope this clears it up.

Regards,

Elie

Your answer

YutongTie-MSFT 53,966 Reputation points Moderator

2024-08-02T00:09:53.8333333+00:00

Hello Elie,

Thanks for reaching out to us, could you please share more details about how you build you AutoML? Is there any document you are referring to so that we can try to reproduce your error?

Regards,

Yutong

Share via

Model evaluation component/environment uses python 3.8 while automl-generated mlflow models require python 3.9

Your answer