Investigating AML workspace images crashes due to already deleted model

Question

I am currently working with Azure Machine Learning. Doing this I encountered a problem when trying to access the images associated with the AML workspace I am working on. It seems, that the method is trying to get access to a model that was already deleted from the model registration by me some time ago. I deleted the model in the GUI of AML on the "models" tab.

Calling:

ws.images

Produces:

Received bad response from Model Management Service:
Response Code: 404
Headers: {'Date': 'Thu, 19 Nov 2020 16:56:10 GMT', 'Content-Type': 'application/json', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'Vary': 'Accept-Encoding', 'x-ms-client-request-id': '5b1ae4fc52324429b241197be3f455a3', 'x-ms-client-session-id': '', 'api-deprecated-versions': '1.0, 2018-03-01-preview, 2018-11-19', 'X-Content-Type-Options': 'nosniff', 'x-request-time': '2.912', 'Strict-Transport-Security': 'max-age=15724800; includeSubDomains; preload', 'Content-Encoding': 'gzip'}
Content: b'{"code":"NotFound","statusCode":404,"message":"The specified resource was not found.","details":[{"code":"NoSuchModelRegistered","message":"One or more models are not registered in Account Subscription: da80dbc4-7dd6-4833-b66e-f6dbbbe42ece, ResourceGroup: dat-q-rg, Workspace: dat-q-aml-01. Unregistered modelIds sklearn_regression_model.pkl:13"}],"correlation":{"RequestId":"5b1ae4fc52324429b241197be3f455a3"}}'

Traceback (most recent call last):
  File "C:\Users\d91755\AppData\Local\Programs\Python\Python39\lib\site-packages\azureml\core\image\image.py", line 577, in list
    resp.raise_for_status()
  File "C:\Users\d91755\AppData\Local\Programs\Python\Python39\lib\site-packages\requests\models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://westeurope.modelmanagement.azureml.net/modelmanagement/v1.0/subscriptions/da80dbc4-7dd6-4833-b66e-f6dbbbe42ece/resourceGroups/dat-q-rg/providers/Microsoft.MachineLearningServices/workspaces/dat-q-aml-01/images?expand=true

The model sklearn_regression_model.pkl:13 existed and was registered in AML but was deleted manually by me in AML. How come calling the images method has still knowledge of this model?
How can I fix this to be able to call the images method again?

In case this is a bug, where would I report it?

Answer 1

@Ariel Brandes Thanks for the question. Your registered models are stored at a path pointed to by an environment variable called AZUREML_MODEL_DIR. For more information on the exact directory structure, see Locate model files in your entry script.You can use the Model.get_model_path() method to retrieve the path of the model file or files on the local file system.

If we use Model.deploy to deploy the model, SDK won’t attach the azureml-app folder to the image. Therefore, if we pull the image from ACR and docker run your image locally, you have to place at least your score.py and model.pkl under azureml-app and use the below example command to start the container:

docker run -dit -p 5001:5001 -e AZUREML_MODEL_DIR=azureml-models/<model_name>/<version> -e AZUREML_ENTRY_SCRIPT=score.py --mount src="<local_filepath>",target=/var/azureml-app,type=bind <image_id> runsvdir /var/runit

Please follow the below notebook sample to register and deploy to the cloud.
https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/deployment/deploy-to-cloud/model-register-and-deploy.ipynb