There is a package conflic when Responsible AI is trying to run. How do I resolve this?

Question

I am using Azure AutoML to train a predictive model that can guess segments. One of the child jobs for Responsible AI fails with the error below. Because of this my trained model does not have an explanation. I need to resolve this, so that I can understand which variables are good predictors in my ML model.

Execution failed. User process 'python' exited with status code 1. Please check log file 'user_logs/std_log.txt' for error details. Error: 
The conflict is caused by:
    The user requested spacy==3.7.4
    en-core-web-sm 3.4.1 depends on spacy<3.5.0 and >=3.4.0

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

Pip subprocess error:
ERROR: Cannot install -r /mnt/azureml/cr/j/599bc9df29c54a5ea77db667ea120275/exe/wd/condaenv.gxdbprd_.requirements.txt (line 75) and spacy==3.7.4 because these package versions have conflicting dependencies.
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts


failed

CondaEnvException: Pip failed

Answer 1

Is there a fix for this issue? We are running into the same issue related to spacy version.

Answer 2

Hello @Tariel Chichua

As per this documentation, i believe your automl job is on regression or classification (binary and multi-class) models trained on tabular structured data.

Next, you register the model from best run.

Got to the model, check in MLmodel file under artifacts tab whether it has sklearn flavor and also download the zip file and extract it to folder.

If you find sklearn flavor, go to notebooks tab , upload the folder downloaded earlier like below.

Execute below code to register new model with sklearn flavor.

import mlflow
from azure.ai.ml.entities import Model
from azure.ai.ml.constants import AssetTypes


ml_client = MLClient(
    credential=DefaultAzureCredential(),
    subscription_id='subscription_id',
    resource_group_name='resource_group',
    workspace_name='workspace_name'
)

tmp_model = mlflow.sklearn.load("3hBestmodel")
mlflow.sklearn.save_model(path="custom_mlflow_sklearn_model",sk_model= tmp_model)

mlflow_model = Model(
    path="custom_mlflow_sklearn_model",
    type=AssetTypes.MLFLOW_MODEL,
    name="custom_mlflow_sklearn_model",
    description="MLflow model created from local path",
)

ml_client.models.create_or_update(mlflow_model)

Above code will register new mlflow model with only sklearn flavor.

Now got to the registered model and you can create dashboard.

Basically, in MLmodel file below content should not be there especially the azureml.engine as automl.

metadata:
  azureml.base_image: mcr.microsoft.com/azureml/curated/ai-ml-automl:12
  azureml.engine: automl

If you don't find sklearn flavor, then you need to follow steps given here to convert the model to sklearn flavor.

So, above are the workaround for automl models, if you have any query, please let us know in comments or in private message.

Note: Make sure you are having all dependencies for model.

Thank you