AZUREML_MODEL_DIR

Question

AZUREML_MODEL_DIR

Eltohami, Hesham (H) 5

I am trying to deploy a model into ACI, the scoring script has the following init() function:

User's image

The deployment fails with the following error:

File "/mnt/batch/tasks/shared/LS_root/mounts/clusters/xyz1/code/Users/xyz/chapter12/score.py", line 8, in init
    model_path = os.path.join(os.getenv("AZUREML_MODEL_DIR"), "model/model.joblib")
  File "/anaconda/envs/azureml_py38/lib/python3.8/posixpath.py", line 76, in join
    a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not NoneType

It seems that the AZUREML_MODEL_DIR variable does not get created during the deployment.

User's image

Any help is appreciated. Thanks a lot.

Hesham

romungi-MSFT 48,911 Reputation points Microsoft Employee Moderator

2023-05-03T05:31:12.4666667+00:00
@Eltohami, Hesham (H) I think your model file is in a folder "model" an example of using the model path in such scenario should be as below:

# Example when the model is a folder containing a file file_path = os.path.join(os.getenv('AZUREML_MODEL_DIR'), 'my_model_folder', 'sklearn_regression_model.pkl')

Could you try something similar?

model_path = os.path.join(os.getenv('AZUREML_MODEL_DIR'), 'model', 'model.joblib') or model_path = os.path.join(os.getenv('AZUREML_MODEL_DIR'), 'model.joblib')

Also, could you check if the path exists if you still see an issue? You could look up the model in ML portal from the models tab under Artifacts.

Finally, if you are printing the variable AZUREML_MODEL_DIR outside of the deployment i.e your notebook the environment variable would not be set. During the deployment the path should already be set to the directory where the models are located. I hope this helps!!
Eltohami, Hesham (H) 5 Reputation points

2023-05-03T15:00:56.49+00:00

@romungi-MSFT Thank you for your reply, I tried both:

model_path = os.path.join(os.getenv('AZUREML_MODEL_DIR'), 'model', 'model.joblib')

or

model_path = os.path.join(os.getenv('AZUREML_MODEL_DIR'), 'model.joblib')

But I am still getting the same error.

Here is the code I used to register the model:

Below is the artifact

romungi-MSFT 48,911 Microsoft Employee Moderator

@Eltohami, Hesham (H) Could you try to get the path instead of the variable and check if this works?

from azureml.core.model import Model

def init():
    global model_1
    # This call will return the path to the model file on the local disk.
    model_1_path = Model.get_model_path(model_name='model.joblib')
    
    # Deserialize the model files back into scikit-learn models.
    model_1 = joblib.load(model_1_path)

Eltohami, Hesham (H) 5

@romungi-MSFT It is still failing with a different error.

When I run the code snippet in from the notebook

User's image

The updated init();

def init():
    global model
    #model_path = os.path.join(os.getenv('AZUREML_MODEL_DIR'),'model.joblib')
    model_path = Model.get_model_path(model_name='model.joblib')
    print(f"Loading model from {model_path}")
    model = joblib.load(model_path)

The error I am getting:

Initializing logger
2023-05-05 16:03:01,019 | root | INFO | Starting up app insights client
logging socket not found. logging not available.
logging socket not found. logging not available.
2023-05-05 16:03:01,023 | root | INFO | Starting up app insight hooks
2023-05-05 16:03:02,754 | root | INFO | Found user script at /mnt/batch/tasks/shared/LS_root/mounts/clusters/xyz1/code/Users/xyz/chapter12/score.py
2023-05-05 16:03:02,754 | root | INFO | run() is not decorated. Server will invoke it with the input in JSON string.
2023-05-05 16:03:02,754 | root | INFO | Invoking user's init function
00000000-0000-0000-0000-000000000000,Loading model from model.joblib
2023-05-05 16:03:03,180 | root | ERROR | User's init function failed
2023-05-05 16:03:03,181 | root | ERROR | Encountered Exception Traceback (most recent call last):
  File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/azureml_inference_server_http/server/user_script.py", line 116, in invoke_init
    self._user_init()
  File "/mnt/batch/tasks/shared/LS_root/mounts/clusters/xyz1/code/Users/xyz/chapter12/score.py", line 12, in init
    model = joblib.load(model_path)
  File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/joblib/numpy_pickle.py", line 605, in load
    obj = _unpickle(fobj, filename, mmap_mode)
  File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/joblib/numpy_pickle.py", line 529, in _unpickle
    obj = unpickler.load()
  File "/anaconda/envs/azureml_py38/lib/python3.8/pickle.py", line 1212, in load
    dispatch[key[0]](self)
KeyError: 0

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/azureml_inference_server_http/server/aml_blueprint.py", line 145, in setup
    self.user_script.invoke_init()
  File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/azureml_inference_server_http/server/user_script.py", line 118, in invoke_init
    raise UserScriptException(ex) from ex
azureml_inference_server_http.server.user_script.UserScriptException: Caught an unhandled exception from the user script

Worker exiting (pid: 59368)
Shutting down: Master
Reason: Worker failed to boot.

romungi-MSFT 48,911 Reputation points Microsoft Employee Moderator

2023-05-08T06:35:30.7433333+00:00

@Eltohami, Hesham (H) Could you add _workspace to download the model. I think the above is trying to load the model from local path but fails since it hasn't been downloaded as workspace parameter is not specified.

model_1_path = Model.get_model_path(model_name='model.joblib', _workspace='<myworkspace>')
Eltohami, Hesham (H) 5 Reputation points

2023-05-08T21:03:21.0533333+00:00

1- When I added the _workspace parameter, the init() fails with AssertionError

2- I noticed when I used .joblib file extention ( i.e. model.joblib) the init() function was able to correctly print model__path, however the joblib.load(model_path) failed with error cannot find model.pkl file.

3-I renamed the file to model.pkl, and now I am getting the following error:

2 answers

Your answer

romungi-MSFT 48,911 Reputation points Microsoft Employee Moderator

2023-05-03T05:31:12.4666667+00:00

@Eltohami, Hesham (H) I think your model file is in a folder "model" an example of using the model path in such scenario should be as below:

# Example when the model is a folder containing a file file_path = os.path.join(os.getenv('AZUREML_MODEL_DIR'), 'my_model_folder', 'sklearn_regression_model.pkl')

Could you try something similar?

model_path = os.path.join(os.getenv('AZUREML_MODEL_DIR'), 'model', 'model.joblib') or model_path = os.path.join(os.getenv('AZUREML_MODEL_DIR'), 'model.joblib')

Also, could you check if the path exists if you still see an issue? You could look up the model in ML portal from the models tab under Artifacts.

Finally, if you are printing the variable AZUREML_MODEL_DIR outside of the deployment i.e your notebook the environment variable would not be set. During the deployment the path should already be set to the directory where the models are located. I hope this helps!!
Eltohami, Hesham (H) 5 Reputation points

2023-05-03T15:00:56.49+00:00

@romungi-MSFT Thank you for your reply, I tried both:

model_path = os.path.join(os.getenv('AZUREML_MODEL_DIR'), 'model', 'model.joblib')

or

model_path = os.path.join(os.getenv('AZUREML_MODEL_DIR'), 'model.joblib')

But I am still getting the same error.

Here is the code I used to register the model:

Below is the artifact
romungi-MSFT 48,911 Reputation points Microsoft Employee Moderator

2023-05-04T04:19:01.3733333+00:00

@Eltohami, Hesham (H) Could you try to get the path instead of the variable and check if this works?

from azureml.core.model import Model def init(): global model_1 # This call will return the path to the model file on the local disk. model_1_path = Model.get_model_path(model_name='model.joblib') # Deserialize the model files back into scikit-learn models. model_1 = joblib.load(model_1_path)
romungi-MSFT 48,911 Reputation points Microsoft Employee Moderator

2023-05-08T06:35:30.7433333+00:00

@Eltohami, Hesham (H) Could you add _workspace to download the model. I think the above is trying to load the model from local path but fails since it hasn't been downloaded as workspace parameter is not specified.

model_1_path = Model.get_model_path(model_name='model.joblib', _workspace='<myworkspace>')
Eltohami, Hesham (H) 5 Reputation points

2023-05-08T21:03:21.0533333+00:00

1- When I added the _workspace parameter, the init() fails with AssertionError

2- I noticed when I used .joblib file extention ( i.e. model.joblib) the init() function was able to correctly print model__path, however the joblib.load(model_path) failed with error cannot find model.pkl file.

3-I renamed the file to model.pkl, and now I am getting the following error:

Answer 1

@romungi-MSFT I finally was able to deploy the model into ACI

I did the following changes to the scoring script:

def init():
    global model
    #model_path = os.path.join(os.getenv('AZUREML_MODEL_DIR'), 'model.joblib')
    model_dir = Model.get_model_path(model_name='chapter12-hesham')
    model_path = os.path.join(model_dir, 'model.joblib')
    print(f"Loading model from {model_path}")
    model = joblib.load(model_path)

The model was successfully loaded:

User's image

What I cannot understand is that when I used the AZUREML__MODEL__DIR, the deployment fails with following error:

User's image

The model_path is the same in both cases but fails to load when using the variable?

Answer 2

Eltohami, Hesham (H) 5

@romungi-MSFT thank you for your reply, I tried both folder name in the model path as you suggested

however I am still the same error.

Below is the code I used to register the model

User's image

Below is a screenshot of the artifact

User's image

Share via

AZUREML_MODEL_DIR

2 answers

Your answer