Share via

Azure Durable Function Issue

Gupta, Vivek 0 Reputation points
2024-01-24T20:54:55.9333333+00:00

Hi, For the last couple of weeks we are observing issues with one of our Azure Function. This is a Durable Function and some activities take time to complete. Now, for a couple of weeks we have observed, the activities either do not complete and keep running again & again. When we checked KUDU logs and found these errors in eventlog.xml. Error 1: Application: w3wp.exe CoreCLR Version: 6.0.2523.51912 .NET Version: 6.0.25 Description: The application requested process termination through System.Environment.FailFast(string message). Message: The operation 'Queue GetMessage' with id 'xxxxxxxxx' did not complete in '00:02:00'. Hit maximum number (5) of timeouts. Terminating the process to mitigate potential deadlock. Stack: at System.Environment.FailFast(System.String) at DurableTask.AzureStorage.TimeoutHandler+<ExecuteWithTimeout>d__31 **Error 2:** Application: w3wp.exe CoreCLR Version: 6.0.2523.51912 .NET Version: 6.0.25 Description: The application requested process termination through System.Environment.FailFast(string message). Message: The operation '*Blob RenewLease*' with id 'yyyyyyyyy' did not complete in '00:02:00'. Hit maximum number (5) of timeouts. Terminating the process to mitigate potential deadlock. Stack: at System.Environment.FailFast(System.String) at DurableTask.AzureStorage.TimeoutHandler+<ExecuteWithTimeout>d__31

Could you please point us in the right direction, where we're going wrong. Thank-you.

Azure Functions
Azure Functions
An Azure service that provides an event-driven serverless compute platform.
4,679 questions
0 comments

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Mike Urnun 9,786 Reputation points Microsoft Employee
    2024-01-29T03:34:44.68+00:00

    Hello @Gupta, Vivek - Thanks for reaching out, and posting on the MS Q&A.

    This doesn't seem like something caused by you. The deadlock issue with the DurableTask.AzureStorage.TimeoutHandler exception and a retry of 5 times sounds like something that stems from the underlying version of the Storage SDK. Here's a comment by the Functions team on why/how this may be happening:

    This is unfortunately occasionally expected behavior currently.v9.3.1 of the Storage SDK that we use internally occasionally deadlocks in a way that causes the storage request stalls out. We wrap all of our storage requests in DurableTask.AzureStorage.TimeoutHandler.ExeucuteWithTimeout to catch this. The first few times we hit this, we will just retry the operation, but once we hit this 5 times, we kill the process, as we are afraid of leaking too many threads.

    Here's an open item on the DurableTask framework tracking this issue: Upgrade to a later version of the Storage SDK for DurableTask.AzureStorage

    I hope the findings above help bring clarity to the issue that you're observing with your Durable Functions. If you have any follow-up questions, feel free to comment below or directly engage the Functions team members on the open issue referenced above.

    Please "Accept Answer" if the answer is helpful so that others in the community may benefit from your experience.