My Python Function App ran 9 times unexpectedly and on wrong time.

Question

Hey There,

I have my function app written in Python which runs at 4:00 PM, Today it did not ran at scheduled time and throws error and I fix that error and deployed it by 9:00 PM, and my function app ran 9 times and the FUNNY PART is that none of the run is mentioned in Monitor section of Function App, it shows only 1 ran that was scheduled at 4:00 PM but it was errored.

Any help would be highly appreciated.

Answer 1

Hello, @Yasir Khan - Thanks for reaching out!

Was this a one-off behavior that was observed only on that day? Or are you still continuing to see this behavior where the trigger invocations are off & runs are inconsistent against the logs? Determining whether it was a transient vs an ongoing/permanent issue is an important piece of detail that can be used to (a) explain the behavior & (b) establish mitigation steps if needed. Based on the information you provided, I would verify the following:

• Were the 9 runs possibly "catch-up" runs on missed executions during the downtime? TimeTriggers don't have Retry behavior but it does try to maintain the schedule throughout restarts/crashes by leveraging the history of past execution, please see: Scheduling
  • You can try leveraging the decorators such as run_on_startup, use_monitor and the IsPastDue property to distinguish if the 9 runs were indeed missed invocations that can be ignored. The doc states the following: "The isPastDue property is true when the current function invocation is later than scheduled. For example, a function app restart might cause an invocation to be missed."
• Regarding the missing logs, is it possible that sampling is enabled in App Insights?
  • Please see: Be aware of sampling
  • Is the WEBSITE_TIME_ZONE setting absent?
  • If you're using the NCRONTAB (instead of TimeSpan) for defining the schedule, which it sounds like you are, it's important to set the WEBSITE_TIME_ZONE app setting provided that you're on Linux Premium/Dedicated plans. The hotfix that you deployed at 9pm has likely caused a restart of the Function App and if the WEBSITE_TIME_ZONE setting was not present at the time, TimeTrigger defaults to using the UTC timezone which could have thrown off the timer calculations. Note that this setting isn't supported on the Consumption plan.
  • Lastly, what was the reason the TimeTrigger failed in the first place? Did the ensuing deployment of the fix succeed without any errors/issues?

I hope this answer gives a starting point. If you're still blocked with the same issue and this answer doesn't help much, let me know in the comments below and we'd be happy to dig deeper for troubleshooting the root cause.