Error in creating new online endpoints and deployments - Internal Server Error

Question

I am trying to create new deployments and enpoints and all operations fail with Error 500 - Internal Server Error. No other details or error codes.

First I have tryed to create a new deployment in an existing endpoint, fails after aprox 25 minutes.

Then I have tried to create a new endpoint together with the deployment, same error but much quicker ~ 1 minute.

The models I am trying to deploy are MFLOW models

Answer 1

Try to access the logs for your deployment to gain insights into the failure using Azure CLI :

az ml online-deployment get-logs --endpoint-name <endpoint-name> --name <deployment-name> --lines 100

Before deploying to the cloud, test your model locally to identify potential issues in your scoring script or environment setup. Azure Machine Learning allows for local deployment, which can be initiated via the CLI:

az ml online-deployment create --endpoint-name <endpoint-name> -n <deployment-name> -f <spec_file.yaml> --local

Try to verify if your user account has the necessary permissions to create and manage deployments. https://learn.microsoft.com/en-us/azure/machine-learning/how-to-troubleshoot-online-endpoints?view=azureml-api-2&utm_source=chatgpt.com

https://learn.microsoft.com/en-us/answers/questions/1812135/azure-open-ai-service-internal-error-500

Answer 2

I finally figured it out. The container registry at workspace level was not properly configured, so image generation during deployment failed.

Still wasn't able to find any reference, error message or log message to indicate this other than the "Internal Server Error".

Lesson Learned - check container registry configuration