Health service stops on 2012R2 servers on Log end to end workflow with access denied

Question

For a couple of weeks we now see the following new behavior:

On our Windows Server 2012R2 systems the Health Services stops with the following error:

The System Center Management Health Service 75BEBE6D-7C3B-362D-3AC7-2613679FB06F running on host JTA23007Pxxxxt and serving management group with id {A9D908C8-532E-C695-796F-F5EAF0453908} is not healthy. Some system rules failed to load.

On the affected system the follwoing entry is in the event log:

Failed to create process due to error '0x80070005 : Access is denied.
', this workflow will be unloaded. 

Command executed: "C:\windows\system32\windowspowershell\v1.0\powershell.exe" -ExecutionPolicy Unrestricted -Command "& '"C:\Program Files\Microsoft Monitoring Agent\Agent\Health Service State\Monitoring Host Temporary Files 8112\2021\LogEndToEndEvent.ps1"'"
Working Directory: C:\Program Files\Microsoft Monitoring Agent\Agent\Health Service State\Monitoring Host Temporary Files 8112\2021\ 

One or more workflows were affected by this.  

Workflow name: Microsoft.SystemCenter.AgentManagement.LogEndToEndEvent 

Out of our 850 2012R2 systems this happens every day on around 5 of them. Restarting the SCOM agent solves the issue. We have all flavours of Windows Servers (2008R2, 2016, 2019) but it only happens on the 2012R2 systems.

The issue occurs on both of our SCOM environments running 2019UR1 and 2019UR3. So most likely not related to the SCOM agent.
I suspect that it might be caused by a recent Windows patch as I seem to remember having seen it happen first in our dev environment.

So far I cannot recognize a pattern in the errors. And it does not happen regularly on the same system so that I could start to investigate any further.

Has anybody seen something similar ?

Thanks
Thorsten

Answer 1

@T_Schneider , For the error message we get, it shows access is denied. Please go to "C:\windows\system32\windowspowershell\v1.0" and find Powershell.exe. Check the permission for the account that runs the Operation Manager agent like local system and see if it has read & execute rights. If not, grant the permission to see if it is working.

However, if the issue still persists, please clear cache to see if it can be fixed.
https://learn.microsoft.com/en-us/system-center/scom/manage-clear-healthservice-cache?view=sc-om-2019

In addition, I notice you doubt it may be related with windows updates. To clarify this, we can uninstall the patch one by one to see if we can find the one that may be related.

Hope it can help.

---

If the response is helpful, please click "Accept Answer" and upvote it.
Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.

Answer 2

LogEndToEndEvent.ps1 is a very basic script that uses a momapi (scom agent) function to log an event to OperationsManager event log :
https://systemcenter.wiki/?GetElement=Microsoft.SystemCenter.AgentManagement.LogEvent&Type=WriteActionModuleType&ManagementPack=Microsoft.SystemCenter.2007&Version=10.19.10505.0

However, it looks like the script is not even starting ("failed to create process"), which is quite weird; and not even all the time which is even weirder.
Under what account is your agent running?
Do you somehow restrict the modification of the ExecutionPolicy to prevent it from being set to Unrestricted?
Do you see corresponding events in the Security event log?

Answer 3

If it always happens around the same time, you could take a workflow trace : https://monitoringguys.com/2020/12/15/tracing-scom-workflows-with-powershell/
They usually provide a lot of useful information!

Answer 4

@T_Schneider Hello-I am dealing with the same exact issues. Same setup as far as AV- were you able to find a resolution to this?
Any help would be greatly appreciated!