Azure Functions container crashes at runtime

mekker 1 Reputation point
2022-08-31T09:00:49.42+00:00

I have used the package_http function of the azureml-contrib-functions package (https://learn.microsoft.com/en-us/python/api/azureml-contrib-functions/azureml.contrib.functions?view=azure-ml-py) to create an Azure functions package of an Azure ML model for deployment in Azure functions.

I have successfully deployed the model as a LocalWebservice (https://learn.microsoft.com/en-us/python/api/azureml-core/azureml.core.webservice.local.localwebservice?view=azure-ml-py) so the InferenceConfig is correct, possibly except for some custom tuning for azure functions that I am not aware of. The only thing I know of is that azureml-defaults has to be included in the environment.

When I deploy the container, that was created by package_http, from acr I am getting Azure Functions runtime is unreachable. According to the recommended solutions by Microsoft I should check the App Logs, which are reporting "Container exited unexpectedly", which apparently indicates that it is a container runtime crash.

I have also looked at the container logs but I do not seem to find any logs with any error information that I am able to interpret.

default_docker.log:
2022-08-31T04:11:46.426933534Z Hosting environment: Production
2022-08-31T04:11:46.427001235Z Content root path: /
2022-08-31T04:11:46.427009335Z Now listening on: http://[::]:5001
2022-08-31T04:11:46.427014135Z Application started. Press Ctrl+C to shut down.
2022-08-31T04:11:47.677547636Z Application is shutting down... 

2022-08-31T04:15:56.718576350Z   
2022-08-31T04:15:56.802444814Z Unhandled Exception: System.InvalidOperationException: Collection was modified; enumeration operation may not execute.  
2022-08-31T04:15:56.802485914Z    at System.Collections.Generic.Queue`1.Enumerator.MoveNext()  
2022-08-31T04:15:56.802491914Z    at System.Linq.Enumerable.WhereEnumerableIterator`1.MoveNext()  
2022-08-31T04:15:56.802495914Z    at System.String.Join(String separator, IEnumerable`1 values)  
2022-08-31T04:15:56.802499915Z    at Microsoft.Azure.WebJobs.Script.Rpc.WorkerProcess.OnProcessExited(Object sender, EventArgs e) in /azure-functions-host-64e0b54013c35ad7ba867dc8a5b096407b4b4010/src/WebJobs.Script/OutOfProc/ProcessManagement/WorkerProcess.cs:line 124  
2022-08-31T04:15:56.802504215Z    at Microsoft.Azure.WebJobs.Script.Rpc.WorkerProcess.
Azure Functions
Azure Functions
An Azure service that provides an event-driven serverless compute platform.
4,245 questions
0 comments

2 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. MughundhanRaveendran-MSFT 12,421 Reputation points
    2022-09-05T07:48:51.893+00:00

    @mekker

    By default, if your container does not respond after 230 seconds, it will time out. (This time out can be increased by with the WEBSITES_CONTAINER_START_TIME_LIMIT app setting.)
    Suggestion
    Check your Application Settings to make sure that the PORT setting of your container is correct. You can view the Application Logs to see if there was any exception or error message indicating the possible cause.

    1 person found this answer helpful.
    0 comments
  2. mekker 1 Reputation point
    2022-09-20T08:03:31.553+00:00

    Although at first I thought this solved the problem, after lengthy discussions with Microsoft support, it turned out that deployment of models to azure functions, using the azureml-contrib-functions package is not supported. This package is not actively supported by Microsoft.

    It does not seem like Azure supports deployment of Azure ML models to Azure Functions, although this is stated for example here: https://learn.microsoft.com/en-us/training/modules/register-and-deploy-model-with-amls/2-deploy-model

    0 comments