Some Scheduled Durable Orchestration Functions Not Always Executing

Asiful Nobel 196 Reputation points
2020-07-06T05:32:55.653+00:00

I have a Timer triggered function that runs every 30 minutes and schedules a Durable Orchestration function to run. The Timer triggered function passes the current time to the Orchestration function. The Orchestration function then schedules a Sub-Orchestration function for each tenant with retries on exception atleast 3 times with backoff. Each Sub-Orchestration function then sends an email to specific email addresses according to schedule and passed down time is used to determine current time.

But I am not always getting this behavior. I am specifically having problems with the first Durable Orchestration function which starts other Sub-Orchestration functions. In two weeks of last month (6/11/2020-6/17/2020 and 6/24/2020-6/27/2020), I have seen logs in specific periods where the Timer is triggered, the Durable Orchestration Functions are scheduled. But not run at all until a deployment is done. Even restarting the Function app does not solve it. However, all other Functions like HTTP triggered functions and Durable Entity Functions were running fine.

On 6/29/2020, while actively monitoring the logs and Azure Table Storage, I found that I was having the same issue. One other thing that I found was that I have 4 Control Queues for the Function App in Storage Account and all four of those were growing. I checked back periodically every hour, but messages in those Control Queues were not all dequeuing. Message count in each Control Queues were almost 1000-2000.

So, I have increased the instance count for my Function App by one on 6/30/2020. After that I have not had this issue. But I want to know what is the cause of this issue. Because there are no errors in the log. I have been checking the Application Map and Metrics for bottleneck, but there were no significant delays except some calls to Table Storage. There were some 4xx errors, but they were expected too such as not finding an entity in Table Storage or CreateIfNotExistsAsync Table call to Azure Table Storage. I am not getting any 503 or 500 errors either from any of the dependency services which may indicate that I am exhausting my resources.

Some more relevant information are I am using Functions v2, codebase is .NET Core. On a typical day, the Timer Triggered Orchestration Function starts around 20-30 Sub-Orchestration functions which generates some types of reports and sends about 400 emails. Of those 400 emails, there are two times during day when almost 70% are sent which is 9 AM and 11 AM UTC. After sending the emails, events are raised and tracked using Durable Entity Functions and Azure Table Storage.

Azure Functions
Azure Functions
An Azure service that provides an event-driven serverless compute platform.
4,635 questions
{count} votes

Accepted answer
  1. MayankBargali-MSFT 69,991 Reputation points
    2020-07-27T05:54:24.95+00:00

    Upgraded to the latest Microsoft.Azure.WebJobs.Extensions.DurableTask and monitored for some time but haven't observed the issue.
    We suspect Microsoft.Azure.WebJobs.Extensions.DurableTask might be related to the issue and recommend upgrading to the latest stable version to avoid potential risk in previous releases.

    0 comments No comments

0 additional answers

Sort by: Most helpful