question

AhsanHabib-1895 avatar image
0 Votes"
AhsanHabib-1895 asked MayankBargali-MSFT edited

Azure function time trigger ran late

I have a schedule function trigger which supposes to run every 10 minutes. The CRON expression I am using is this: 0 */10 * * * *

The function runs every 10 minutes. However, sometimes it run late. For instance, the function suppose to run on 24 MAR, 07:30:00. But it ran on 24 MAR 08:44:30 instead. Another time it supposes to run at 24 MAR, 06:30:00. But ran at 24 MAR, 06:54:56
Usually, it runs correctly. Sometimes it does late for few seconds or a minute. But yesterday the gap was huge which caused many others error. From both the App Insights and Support section, I have discovered that it past due execution. Here is the screenshot from the support request section:
81591-image.png


81608-image.png
81633-image.png

The last correct InvocationId: fbfb719c-42d8-4134-85aa-407d43163262(Run at 07:20:00 GMT, this is correct)
The next InvocationId: c5d78293-c1e8-42b2-9d4d-81612d4c8247(which ran at 08:44:31 GMT, this suppose to be run at 07:30:00 GMT)

correct InvocationId: d03fb73c-3a8d-410a-b6d6-bcde2662dcb4(Run at 06:20:00 GMT, this is correct)
The next InvocationId: d3d45712-811b-4102-8ee5-7704965b622e(which ran at 06:54:56 GMT, this suppose to be run at 06:30:00 GMT)

I also checked the Web App Restart detection section. Nothing was restarted. So I was wondering what could cause such an issue?
I am using Consumption Plan. I always deploy through VSCode. Have not set any WEBSITE_TIME_ZONE. Therefore using the UTC time zone I believe. Also, I am using Node.JS SDK.


azure-functions
image.png (40.4 KiB)
image.png (41.1 KiB)
image.png (12.0 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

BhupenderSingh-8302 avatar image
0 Votes"
BhupenderSingh-8302 answered

this looks bizarre , Did you tried to have logs running inside your node function itself.
though it won't help much but still you will get surety that it has nothing to do with your node code.
Let me get back to you with my PoC on the same issue.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

AhsanHabib-1895 avatar image
0 Votes"
AhsanHabib-1895 answered AhsanHabib-1895 edited

@BhupenderSingh-8302 Yes !!! I do have logs running inside of my function. Here is the Log I collected from App Insights

81847-image.png

The selected text is my console.log. As you can see it also mentions the past due issue. The function ran perfectly but late.



image.png (81.9 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

MayankBargali-MSFT avatar image
0 Votes"
MayankBargali-MSFT answered MayankBargali-MSFT edited

Hi @AhsanHabib-1895

There could be different reasons for this issue and I will suggest you to review the below document to troubleshoot the issue and see if you are able to find the root cause.

Timer triggered function app uses TimerTriggerAttribute. This attribute consists of the Singleton Lock feature which ensures that only a single instance of the function is running at any given time. If any process runs longer than the scheduled timer, the new incoming process waits for the older process to finish and then uses the same instance. If you are using the same storage account across different timer trigger functions then this could be one of the reasons as mentioned here.

If not then I will suggest you to open a support ticket for in depth analysis to find the root cause of the issue. If you still couldn't find the root cause and don't have support plan please refer to my private comment.

https://github.com/Azure/azure-functions-host/wiki/Investigating-and-reporting-issues-with-timer-triggered-functions-not-firing
https://github.com/Azure/azure-webjobs-sdk-extensions/wiki/TimerTrigger#troubleshooting

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.