Watcher Tasks stopping randomly

Question

Watcher Tasks stopping randomly

Tom Joyce 61

Is there a way to monitor when a watcher tasks stops running and be notified about it? I tried looking through the log analytics data but not able to find anything related to this. Some of our watcher tasks are highly important for monitoring systems so knowing when they are not running is important.

Accepted answer

2 additional answers

Your answer

Answer 1

Stanislav Zhelyazkov 28,596 MVP Volunteer Moderator

Hi,
As a workaround I can recommend to add code on your watcher runbook to log a record in Log Analytics by using HTTP Data Collector API. After that you can create alert that notifies you when such record is not present for the last 5 minutes. That would mean that the watcher tasks have stopped.

Please "Accept the answer" if the information helped you. This will help us and others in the community as well.

Answer 2

Hi,
Watcher tasks do not run all the time but rather run on frequency. This means that watcher task starts and stop depending on the frequency. If you want to monitor if watcher tasks fails, you can just use the job logs. These are the same logs that are for runbook jobs. Watcher task overall is a runbook that has frequency and a few more specific features compared to runbooks. That said it may be good to also check the statement here:

Starting in May 2020, using Azure Logic Apps is the recommended and supported way to monitor for events, schedule recurring tasks, and trigger actions. See Schedule and run recurring automated tasks, processes, and workflows with Azure Logic Apps.

Please "Accept the answer" if the information helped you. This will help us and others in the community as well.

Answer 3

Tom Joyce 61

Using another product to schedule runbooks seems a bit silly in my opinion and will only switch as a last resort. There has been no set date for watcher task to be decommissioned and I like having all my runbooks under one management system. Having to manage Azure Logic Apps on top of my Automation environment is not something I wish to do at this time.

As for the issue at hand I understand that watcher tasks are running on a schedule but I guess I worded it wrong in my original question. The watcher scheduler for each of my runbooks goes into a stop randomly so the watcher runbook does not run when it is supposed to. It would be good to know how to monitor when a watcher schedule is stopped instead of running.

Stanislav Zhelyazkov 28,596 Reputation points MVP Volunteer Moderator

2021-10-07T13:37:36.44+00:00

ok. Now I understand what you mean. That action seems like management plane action which means it should be logged in Azure Activity log. Unfortunately I only see logs for watcher task being stopped only when it is done by an actual user. At least in my test environment I did not see logs for watcher task being stopped when it is not stopped by an actual user. If in your setup you can see those logs you can use activity log alert to alert upon this. You can also send activity log alerts to Log Analytics and use alert based on Kusto query to alert. If you see the same behavior like me I would suggest to log an official support case. For me this is a bug that these operations are not in activity log when it is not an user stopping the watcher task.

Share via

Watcher Tasks stopping randomly

2 additional answers

Your answer