Azure Functions v3 deadlock issue

Roberto Ramón 1

Hi all, we have experienced some days ago that 2 environments where we are playing with functions v3 (3.3.1) are not working properly.

Some async actions that took 15-20 mins, they are not finishing now or taking hours. We have checked inside kudu in the eventlog.xml and we have found errors related with this

One error:*"Microsoft.AspNetCore.Server.Kestrel EventId: 22 Heartbeat took longer than "00:00:01" at "03/15/2022 10:10:56 +00:00". This could be caused by thread pool starvation."

Second error: "Application: w3wp.exe CoreCLR Version: 4.700.21.35901 .NET Core Version: 3.1.18 Description: The application requested process termination through System.Environment.FailFast(string message). Message: The operation 'ListBlobs' with id 'xxxxxxxxxx' did not complete in '00:02:00'. Hit maximum number (5) of timeouts. Terminating the process to mitigate potential deadlock. Stack: at System.Environment.FailFast(System.String) at DurableTask.AzureStorage.TimeoutHandler+<>c.<.cctor>b__5_0(System.String) at DurableTask.AzureStorage.TimeoutHandler+\

MughundhanRaveendran-MSFT 12,456 Reputation points

2022-03-16T09:17:20.47+00:00
@Roberto Ramón , please help me in answering the below queries

Is this durable function?

what does the async actions do?

Have you enabled any logging at your end? What does the invocation logs show?
Roberto Ramón 1 Reputation point

2022-03-17T09:34:00.48+00:00
Thanks for coming back.

Yes, it is a durable function.

Lot of things, updating database status, provisioning site collection with customizations to SharePoint, etc...

We have found just the error described into eventlog.xml because our code is not breaking, at least we cannot see anything in app insights. Basically, the experience is it starts to work and at some point it stops and is restarted, strarting from the beginning again. When it stops is not always in the same place, it's random.
MughundhanRaveendran-MSFT 12,456 Reputation points

2022-03-22T13:57:56.417+00:00

@Roberto Ramón , Is the storage account associated with the Function app shared amoung other function apps as well? We recommend to use a single storage account per function app
MughundhanRaveendran-MSFT 12,456 Reputation points

2022-03-24T13:51:33.88+00:00

@Roberto Ramón , Please share your inputs for the above mentioned comment so that I can proceed with the troubleshooting
Roberto Ramón 1 Reputation point

2022-03-25T08:35:09.1+00:00

Hi @MughundhanRaveendran-MSFT thanks for your message. Yes, it is shared by 2 function apps. We didn't think that this was a problem as in the documentation https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-task-hubs?tabs=csharp says that a storage account can contain varios Task hubs.
MughundhanRaveendran-MSFT 12,456 Reputation points

2022-03-25T08:42:56.327+00:00

@Roberto Ramón , Function app entirely depends on Storage account for logging and data storage purposes. We dont recommend multiple apps sharing the same accont for production environments

https://learn.microsoft.com/en-us/azure/azure-functions/storage-considerations#shared-storage-accounts

Storage account can contain various task hubs however it is possible that one function app can have mutiple task hubs. This isnt a problem.
Peter Asamoah 0 Reputation points

2023-11-01T20:50:53.11+00:00

Currently experiencing this issue. When I read a file from Azure Blob storage within an Activity Functions and the function execution takes longer than 2mins. The Function host restarts and the Activity function starts again. This fails over and over till I stop the Function App. Is there any way to solve this ?

1 answer

Yevhen Tsybulskyi 0 Reputation points

2023-02-13T15:33:33.69+00:00

Hi @Roberto Ramón !

Did you manage to fix the problem?
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

Share via

Azure Functions v3 deadlock issue

1 answer