Azure function crashed during cold start after converting to isolated worker

Question

I have an Azure function that had been working fine for the past 2 years. I recently converted to isolated worker and now every time the function is spawned up, it will crash initially during cold start. So if the API has not been called for a while (more than 5'), the first call to the API again will always fail. Then after about 10s, it works fine. Before migrating to the isolated worker, it never had this problem, also cold start was about 1-2s only. I'm using the consumption plan. Diagnostics shows the below error, which is not very helpful. I really want to convert the azure function back to be in-process, but that is being deprecated by Microsoft.

This is really frustrating. The isolated worker seems a huge step back to me.

Exception while executing function
 /Functions.GetSuffixes ---> System.InvalidOperationException 
 /Failed to proxy request with ForwarderError
 /Request ---> System.Net.Http.HttpRequestException 
 /An attempt was made to access a socket in a way forbidden by its access permissions. (localhost:49891) ---> System.Net.Sockets.SocketException 
 /An attempt was made to access a socket in a way forbidden by its access permissions.
   
System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error,CancellationToken cancellationToken)
   
System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   
async System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|277_0(??)
   
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   
async System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host,Int32 port,HttpRequestMessage initialRequest,Boolean async,CancellationToken cancellationToken)
   End of inner exception
   
async System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host,Int32 port,HttpRequestMessage initialRequest,Boolean async,CancellationToken cancellationToken)
   
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   
System.Threading.Tasks.ValueTask`1.get_Result()
   
async System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request,Boolean async,CancellationToken cancellationToken)
   
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   
System.Threading.Tasks.ValueTask`1.get_Result()
   
async System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request,Boolean async,CancellationToken cancellationToken)
   
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   
System.Threading.Tasks.ValueTask`1.get_Result()
   
async System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(HttpRequestMessage request)
   
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   
async System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync[T](??)
   
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   
System.Threading.Tasks.ValueTask`1.get_Result()
   
async System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessage request,Boolean async,CancellationToken cancellationToken)
   
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   
System.Threading.Tasks.ValueTask`1.get_Result()
   
async System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request,Boolean async,Boolean doRequestAuth,CancellationToken cancellationToken)
   
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   
async Microsoft.Azure.WebJobs.Script.Grpc.RetryProxyHandler.SendAsync(HttpRequestMessage request,CancellationToken cancellationToken) 
/_/src/WebJobs.Script.Grpc/Server/RetryProxyHandler.cs 
 /33
   
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   
async Yarp.ReverseProxy.Forwarder.HttpForwarder.SendAsync(??)
   End of inner exception
   
async Microsoft.Azure.WebJobs.Script.Grpc.DefaultHttpProxyService.EnsureSuccessfulForwardingAsync(ScriptInvocationContext context) 
/_/src/WebJobs.Script.Grpc/Server/DefaultHttpProxyService.cs 
 /65
   
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   
async Microsoft.Azure.WebJobs.Script.Grpc.GrpcWorkerChannel.InvokeResponse(InvocationResponse invokeResponse) 
/_/src/WebJobs.Script.Grpc/Channel/GrpcWorkerChannel.cs 
 /1099
   
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   
async Microsoft.Azure.WebJobs.Script.Description.WorkerFunctionInvoker.InvokeCore(Object[] parameters,FunctionInvocationContext context) 
/_/src/WebJobs.Script/Description/Workers/WorkerFunctionInvoker.cs 
 /101
   
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   
async Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase.Invoke(Object[] parameters) 
/_/src/WebJobs.Script/Description/FunctionInvokerBase.cs 
 /82
   
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   
async Microsoft.Azure.WebJobs.Script.Description.FunctionGenerator.Coerce[T](Task`1 src) 
/_/src/WebJobs.Script/Description/FunctionGenerator.cs 
 /225
   
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   
async Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2.InvokeAsync[TReflected,TReturnValue](Object instance,Object[] arguments) 
D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionInvoker.cs 
 /53
   
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   
async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.InvokeWithTimeoutAsync(IFunctionInvoker invoker,ParameterHelper parameterHelper,CancellationTokenSource timeoutTokenSource,CancellationTokenSource functionCancellationTokenSource,Boolean throwOnTimeout,TimeSpan timerInterval,IFunctionInstance instance) 
D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs 
 /581
   
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   
async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithWatchersAsync(IFunctionInstanceEx instance,ParameterHelper parameterHelper,ILogger logger,CancellationTokenSource functionCancellationTokenSource) 
D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs 
 /527
   
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   
async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstanceEx instance,FunctionStartedMessage message,FunctionInstanceLogEntry instanceLogEntry,ParameterHelper parameterHelper,ILogger logger,CancellationToken cancellationToken) 
D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs 
 /306
   End of inner exception
   
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   
async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstanceEx instance,FunctionStartedMessage message,FunctionInstanceLogEntry instanceLogEntry,ParameterHelper parameterHelper,ILogger logger,CancellationToken cancellationToken) 
D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs 
 /352
   
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   
async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.TryExecuteAsync(IFunctionInstance functionInstance,CancellationToken cancellationToken) 
D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs 
 /108

Answer 1

Hi @TD,

It's great to hear that your issue has been resolved, and your willingness to share the solution is appreciated, as it will benefit others facing similar challenges.

As per the Microsoft Q&A community guidelines, the original author of a question cannot accept their own answer but can accept answers provided by others. Therefore, I will repost your solution for you to accept if you wish. The problem arose after transitioning a two-year-old function app to an isolated worker function, which led to crashes on cold starts due to a socket exception. This was confirmed by replicating the issue in a newly created function app.

Your issue became resolved when you updated the following nuget packages:

<PackageReference Include="Microsoft.Azure.Functions.Worker" Version="1.20.1" /> --> 1.22.0 <PackageReference Include="Microsoft.Azure.Functions.Worker.ApplicationInsights" Version="1.1.0" /> --> 1.2.0 
<PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore" Version="1.2.0" /> --> 1.3.0 
<PackageReference Include="Microsoft.Azure.Functions.Worker.Sdk" Version="1.16.4" /> --> 1.17.2

I didn't see any issues under https://github.com/Azure/azure-functions-host calling out your specific issue so it's difficult to know for certain what the underlying root cause was.