Azure Durable Function Issue

Question

Hi, For the last couple of weeks we are observing issues with one of our Azure Function. This is a Durable Function and some activities take time to complete. Now, for a couple of weeks we have observed, the activities either do not complete and keep running again & again. When we checked KUDU logs and found these errors in eventlog.xml. Error 1: Application: w3wp.exe CoreCLR Version: 6.0.2523.51912 .NET Version: 6.0.25 Description: The application requested process termination through System.Environment.FailFast(string message). Message: The operation 'Queue GetMessage' with id 'xxxxxxxxx' did not complete in '00:02:00'. Hit maximum number (5) of timeouts. Terminating the process to mitigate potential deadlock. Stack: at System.Environment.FailFast(System.String) at DurableTask.AzureStorage.TimeoutHandler+<ExecuteWithTimeout>d__31 **Error 2:** Application: w3wp.exe CoreCLR Version: 6.0.2523.51912 .NET Version: 6.0.25 Description: The application requested process termination through System.Environment.FailFast(string message). Message: The operation '*Blob RenewLease*' with id 'yyyyyyyyy' did not complete in '00:02:00'. Hit maximum number (5) of timeouts. Terminating the process to mitigate potential deadlock. Stack: at System.Environment.FailFast(System.String) at DurableTask.AzureStorage.TimeoutHandler+<ExecuteWithTimeout>d__31

Could you please point us in the right direction, where we're going wrong. Thank-you.

Answer 1

Hello @Gupta, Vivek - Thanks for reaching out, and posting on the MS Q&A.

This doesn't seem like something caused by you. The deadlock issue with the DurableTask.AzureStorage.TimeoutHandler exception and a retry of 5 times sounds like something that stems from the underlying version of the Storage SDK. Here's a comment by the Functions team on why/how this may be happening:

This is unfortunately occasionally expected behavior currently.v9.3.1 of the Storage SDK that we use internally occasionally deadlocks in a way that causes the storage request stalls out. We wrap all of our storage requests in DurableTask.AzureStorage.TimeoutHandler.ExeucuteWithTimeout to catch this. The first few times we hit this, we will just retry the operation, but once we hit this 5 times, we kill the process, as we are afraid of leaking too many threads.

Here's an open item on the DurableTask framework tracking this issue: Upgrade to a later version of the Storage SDK for DurableTask.AzureStorage

I hope the findings above help bring clarity to the issue that you're observing with your Durable Functions. If you have any follow-up questions, feel free to comment below or directly engage the Functions team members on the open issue referenced above.

---

Please "Accept Answer" if the answer is helpful so that others in the community may benefit from your experience.