Azure ML online endpoint suddenly returns timeout

Albert Garriga Porqueras 11

Hi all,

We have deployed a managed online endpoint in Azure ML and first it works fine. However, after a few days, with the exact same request, the endpoint takes much longer to process the request and gives a timeout (the HTTP code returned is 504). We don't understand this behavior since we did not modify the endpoint and the metrics don't show a huge increase in cpu or memory usage. If we restart it then it works again for a few days until it doesn't work anymore. Has anyone faced the same issue? Could you solve it?

Thanks.

romungi-MSFT 41,961 Reputation points Microsoft Employee

2022-10-18T10:23:42.023+00:00

@Albert Garriga Porqueras You will have to add diagnostic settings for your workspace to get more details on what could have caused the inferencing operation to timeout after some time. Please see this page on the available categories and logs.
romungi-MSFT 41,961 Reputation points Microsoft Employee

2022-10-21T11:37:46.31+00:00

@Albert Garriga Porqueras Did you get a chance to check if are able to add additional logs to your workspace settings?
Albert Garriga Porqueras 11 Reputation points

2022-10-21T14:21:57.533+00:00

We have activated the Insights of the endpoint and deployed it again with more resources to try it, although we did not get an error indicating that we had a problem of resources. After looking more into it, maybe we needed more memory, and we didn't get that error. We are not sure yet, we are trying it.

2 answers

Hung Nguyen Thanh 16 Reputation points

2022-10-29T14:10:52.19+00:00

@Albert Garriga Porqueras
I came across the same error trying to invoke the model from public internet though, I retried invoking it in a VM that is in the same vnet/subnet that Azure Machine Learning Workspace, Storage Account and Container Registry connect to by private endpoints, and it works.
My scenario is public inbound, and public egress network access is disabled.
Please sign in to rate this answer.
Albert Garriga Porqueras 11 Reputation points

2022-11-04T16:05:32.993+00:00

At the end it seems that adding more resources (memory) to the endpoint solved the issue for us. I don't understand why we didn't get an out of memory error, but is looks like we were just over the memory limit.

Hung Nguyen Thanh 16 Reputation points

2022-11-04T16:10:48.357+00:00

Interesting . How can you add more memory to the endpoint ? By expanding instance_count ?

Albert Garriga Porqueras 11 Reputation points

2022-11-04T16:44:13.197+00:00

No, we had to remove it and redeployed it with more resources.
Sign in to comment
Deleted

This answer has been deleted due to a violation of our Code of Conduct. The answer was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.

Comments have been turned off. Learn more