Azure Machine Learning deployed serverless models all retuning internal server error

Question

Azure Machine Learning deployed serverless models all retuning internal server error

Taylor, Russell 26

At some point between last Friday (27th Feb) and the morning of Friday (28th Feb) the models we have deployed as serverless endpoints on Azure ML in US East 2 stopped working.

When testing the cohere V3 english embedding model in Azure ML studio the following error is returned:

User's image

If attempting to make a REST request via postman, powershell, python etc a 500 response code and the same error text is returned.

Nothing has changed on the Azure ML service instance or any of its dependencies which leaves me to believe this is a server side issue on the serverless managed endpoints platform.

I have tried deploying new endpoints and various serverless models from the catalog and all result in the same , internal sever error when trying to interact with the model endpoints.

If someone from Azure reads this please advise if there is a general issue with the platform, there is no issues reporting on the Azure Status page, on the Azure Portal or within the AI Studio to enlighten the customer on what is going on.

Taylor, Russell 26 Reputation points

2025-03-05T11:57:06.13+00:00

The service appears to have self healed as of today (March 5th), unfortunately I can find no details as to the cause of the issue, what the resolution was or what mitigations are being put in place to prevent this happening again.

1 answer

Your answer

Taylor, Russell 26 Reputation points

2025-03-05T11:57:06.13+00:00

The service appears to have self healed as of today (March 5th), unfortunately I can find no details as to the cause of the issue, what the resolution was or what mitigations are being put in place to prevent this happening again.

Answer 1

Amira Bedhiafi 42,941 MVP Volunteer Moderator

I think the same problem seems to have started the last few days if I remember correctly.

Despite no changes being made to the Azure ML service instance or its dependencies, the issue persists across various serverless models and newly deployed endpoints. This suggests a potential server-side problem with the serverless managed endpoints platform.

If you are experiencing this issue, I recommend that you reach out to Azure support for further assistance.

Taylor, Russell 26 Reputation points

2025-03-05T08:50:13.34+00:00

We do not have paid support at this time so I cannot raise an Azure support request , but it's poor communications if there is a general issue and its not being reported on the azure status pages.
santoshkc 15,615 Reputation points Microsoft External Staff Moderator

2025-03-05T12:28:30.0566667+00:00

Hi Taylor, Russell,

We have noticed that you rated an answer as not helpful. We appreciate your feedback and are committed to improving your experience with the Q&A.

It appears that there was a temporary issue with Azure Machine Learning's serverless managed endpoints. Serverless API is not available now for this model on any of the Azure regions, please check again later. This could be due to a temporary restriction or a change in regional support for that model. You can verify the latest availability of models in different regions as per your requirements. We will provide an update once we have the information.

If the information is helpful, please click on "Upvote".

I hope you understand! Thank you.
Taylor, Russell 26 Reputation points

2025-03-05T13:09:45.2366667+00:00

Sorry but this statement is also not particularly helpful. The models were available as it was already deployed before the issue happened, the same deployed model is now working again since azure implemented an unknown resolution , no redeployment took place it just started working again.

What is more concerning is a service breaking, no information being published as to why it was broken, what caused the issue or how this will be mitigated in the future.
santoshkc 15,615 Reputation points Microsoft External Staff Moderator

2025-03-06T12:58:08.5333333+00:00

Hi Taylor, Russell,

We understand your concern. The issue was eesolved without action on your end, but the lack of transparency is challenging. We recommend opening a support ticket for a post-incident review to get details on the root cause and future mitigations. If reliability is critical, consider multi-region deployment for redundancy.

Thank you for your understanding.
santoshkc 15,615 Reputation points Microsoft External Staff Moderator

2025-03-07T12:24:07.56+00:00

Hi Taylor, Russell,

Following up to see if the given response was helpful. Could you please retake the survey on the above. Thank you.

Share via

Azure Machine Learning deployed serverless models all retuning internal server error

1 answer

Your answer