Timer triggered functions not firing or firing past due

Tran, Nguyen 20

Hello all,

We currently have two timer triggered functions that either don't get triggered at all or get triggered past due. Of course, we want these functions to trigger and trigger on time.

For example, here's one of our functions that's scheduled to run daily at 12am CST time zone in Texas (we now changed it to trigger daily at 11am):

User's image

We can see that, when the function first got deployed (6/15) until 6/18, it got triggered daily and rather on time. Then from 6/19 to 6/25, it got triggered daily but was past due. Then from 6/28 onwards, it no longer gets triggered daily, and when it gets triggered it's due to "Trigger Details: UnscheduledInvocationReason: IsPastDue"

Here's the info about our function app for further context:

Our function app uses In-process C# and is on Premium Plan
We deploy our functions to the function app via GITHUB_ZIP_DEPLOY_FUNCTIONS_V1
WEBSITE_TIME_ZONE = Central Standard Time
I set this function's CRON expression as "0 0 11 * * *" and expect it to run daily at 11am CST.
Sample invocation id to find our function app: "Executing 'LoadPeopleEmployeeAllAA' (Reason='Timer fired at 2024-07-11T15:07:09.7613018-05:00', Id=f51a0277-9233-4105-81be-7cd5ee463af5)"

Please provide us guidance on how to resolve this. I'm also wondering it has something to do with some logging issue, or the timer triggers are not 'synced up' properly.

Luke Murray 11,091 Reputation points MVP

2024-07-13T02:25:13.6133333+00:00
Try adding a catch to it, to grab the Date/Time to the Timertrigger: https://stackoverflow.com/questions/78220331/azure-function-issue-trigger-details-unscheduledinvocationreason-ispastdue-o

And check the time/date on the Function console manually to confirm whether Central Standard is actually correct.

On the left-hand side Blade, underneath Development Tools, click on Console

Type the following into the console:

Time

Date

On the left-hand side Blade, underneath Development Tools, click on Console

Type the following into the console:

Time

Date

2 answers

Sander van de Velde | MVP 32,726 Reputation points MVP

2024-07-14T11:45:24.28+00:00

Hello @Tran, Nguyen,

welcome to this moderated Azure community forum.

Timer triggered Azure Functions are great to execute code at certain points in time.

The underlying schedular act like a singleton and cannot run two timer triggered functions at once by default.

Check out this blog post for the background story.

This seems to be fixable by adding the ‘UseMonitor = false’ parameter.

If the response helped, do "Accept Answer". If it doesn't work, please let us know the progress. All community members with similar issues will benefit by doing so. Your contribution is highly appreciated.
Please sign in to rate this answer.
Tran, Nguyen 20 Reputation points

2024-07-15T18:06:14.77+00:00

Thanks for the input.

Upon looking at the traces table in Application Insights today, I noticed that the function LoadPeopleEmployeeAllAA gets triggered on time:

But for some reason the first one time execution run ‘got stuck’ and the function gets triggered again multiple times (due to Trigger Details: UnscheduledInvocationReason: IsPastDue, OriginalSchedule: 2024-07-15T11:00:00.0000000-05:00) afterwards

The function seems to get stuck at “Populating the DataTable with the data from the searchResponse”. So I believe I should looking into my function code. And this probably causes a function run to 'get stuck'.

And it seems like there's a default setting of retrying whenever the function run 'gets stuck', as indicated by "Total Occurrences". Could you please confirm if this is correct? And where do I go to disable or reconfigure this setting?

Tran, Nguyen 20 Reputation points

2024-07-15T20:07:45.6066667+00:00

More info:

It looks like functions firing Past due is usually caused by that one execution didn't complete successfully, as indicated by the "Timer Trigger Issue Analysis" in our function app.

Tran, Nguyen 20 Reputation points

2024-07-15T20:12:14.9+00:00

It looks like functions being fired Past due is usually caused by that executions that didn't complete successfully, as indicated by the "Timer Trigger Issue Analysis" in our function app.

Ryan Hill 28,191 Reputation points Microsoft Employee

2024-07-15T20:17:14.6733333+00:00

@Tran, Nguyen not unless you have the retry (attribute) set. By default, the timer trigger won't run again until the scheduled time even if it fails. See https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-timer?tabs=python-v2%2Cisolated-process%2Cnodejs-v4&pivots=programming-language-csharp

Tran, Nguyen 20 Reputation points

2024-07-15T20:29:02.97+00:00

We don't have the retry attribute set. We only have these two set:

The function seems to 'get stuck' at “Populating the DataTable with the data from the searchResponse” line, which is not the end of the function code, so I believe the function did not even get failed out at all, maybe it freezes and eventually got ended by the function app?

Sander van de Velde | MVP 32,726 Reputation points MVP

2024-07-15T22:58:36.28+00:00

Hello @Tran, Nguyen,

the easiest solution is just to fix that 'getting stuck' issue in the first place.

Your function runs for a maximum of 5 minutes in most cases if not configured otherwise (or running in another tiers).

Tran, Nguyen 20 Reputation points

2024-07-15T23:15:24.4733333+00:00

Ok I understand, I've been looking into the code, and it's just that sometimes it gets passed “Populating the DataTable with the data from the searchResponse” line but gets stopped somewhere else, sometimes it gets stuck even before that line.

Looks like the code probably has some problem or is not efficient which could lead to using too much computing resources. As I also noticed there's logs regarding high cpu usage...

Thanks for the input!
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.
Tran, Nguyen 20 Reputation points

2024-07-16T17:03:00.8633333+00:00

SOLUTION:

Seemingly, our function code is not efficient enough for EP1 plan and thus mid run, the JobHost restarts which then causes the function to never complete its on-time run due to “Thread Pool Starvation and high memory usage”

Our fix for now is to upgrade the plan from EP1 to EP2 while optimizing the code to later switch back to EP1 to save on costs.
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Timer triggered functions not firing or firing past due

2 answers

Your answer