Runbooks remain queued when using custom hybrid worker credentials, work fine as LocalSystem

Question

I'm working with an Arc enabled server/extension based V2 HRW. For some reason runbooks do not execute when the HRW is configured to use an Azure credential. The portal shows them as never leaving the queued state. The Microsoft-SMA event log on the HRW indicates the sandbox was created, then follows with Win32 Process Exited with code [2148734720]. The Azure credential being used is a domain user that is a member of the local administrators group on the HRW.

When I change the HRW to use the default credential (LocalSystem), runbooks execute.

I know the credential is good, as I can use it with cmdlets in runbooks that support -Credential. Leaving it to use LocalSystem would be fine if all cmdlets used -Credential, but the Exchange cmdlets like New-RemoteMailbox don't--which is why we need the HRW to run runbooks/jobs under a credential.

Any insight is appreciated, and thanks!

AVO

Answer 1

@Andrew VO , @Christian Hiebler , thank you for bringing this issue to our attention. After a bit of test and research, here are my findings about this issue. I will summarize it below, so that it can help others also looking for answers to similar questions.

Issue:
Based on the behavior mentioned above, when you use a different user (apart from the default Local System) as Run as credential for Hybrid worker, the runbook jobs targeted to run on this hybrid worker remains in Queued state for approximately 15 minutes and then goes in Suspended state. This happens despite having this user added to Administrators group on the Worker Machine and explicit permissions have to be added to some directories to get this to work.

Cause
This happens because of Filtered Token feature of User Access Control (UAC) on Windows Server - When a user with administrative or other powerful privileges or group memberships logs on, Windows creates two access tokens to represent the user account. The unfiltered token has all the user's group memberships and privileges. The filtered token represents the user with the equivalent of standard user rights. By default, this filtered token is used to run the user's programs. The unfiltered token is associated only with elevated programs. To better protect those users who are members of the local Administrators group, we implement UAC restrictions on the network. This mechanism helps prevent loopback attacks. This mechanism also helps prevent local malicious software from running remotely with administrative rights.
For more details, please refer to the following article:
How to disable User Account Control (UAC) on Windows Server

Resolution and workaround
The resolution of this issue would be - to add the following permissions to the user used in run as credential for Hybrid Runbook Worker. These permissions will have to be added explicitly instead of adding the user to Administrators because by default, the Filtered Token would disallow the user from accessing protected resources/resources requiring elevated permissions.

Registry path

HKLM\SYSTEM\CurrentControlSet\Services\EventLog (read)
HKLM\SYSTEM\CurrentControlSet\Services\WinSock2\Parameters (full access)
HKLM\SOFTWARE\Microsoft\Wbem\CIMOM (full access)
HKLM\Software\Policies\Microsoft\SystemCertificates\Root (full access)
HKLM\Software\Microsoft\SystemCertificates (full access)
HKLM\Software\Microsoft\EnterpriseCertificates (full access)
HKLM\software\Microsoft\HybridRunbookWorker (full access)
HKLM\software\Microsoft\HybridRunbookWorkerV2 (full access)
HKEY_CURRENT_USER\SOFTWARE\Policies\Microsoft\SystemCertificates\Disallowed (full access)
HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Setup\PnpLockdownFiles (full access)

Folders

C:\ProgramData\AzureConnectedMachineAgent\Tokens (read)
C:\Packages\Plugins\Microsoft.Azure.Automation.HybridWorker.HybridWorkerForWindows\**<version>**\HybridWorkerPackage\HybridWorkerAgent (full access)

Please refer to this link for details - Use a credential asset for a Hybrid Runbook Worker group

---

The workaround to adding explicit permission as mentioned above to the user used in "run as credential" for Hybrid Runbook Worker will involve the following steps. Please note this would be a not recommended workaround, as it would disable the default UAC security check in place.

1. Add the user to local "Administrators" group
2. Disable the local group policy - User Account Control: Run all administrators in Admin Approval Mode, available in Group Policy editor here: Local Security Policy --> Security Settings --> Local Policies --> Security Options.

3. Reboot the machine

Please let me know if you have any questions.

---

Please 'Accept as answer' and ‘Upvote’ if it helped so that it can help others in the community looking for help on similar topics.

Answer 2

RunAs account on the local HRW was member of local administrator, yet missing permissions to various subdirectories within C:\Program Files\AzureConnectedMachineAgent and C:\ProgramData\AzureConnectedMachineAgent directory structures.

Answer 3

More than a year and we still need to apply workarounds... MS is pushing customers to migrate to V2, but that's far from being mature.

Answer 4

Hi

I'm experiencing the same issue. After manually assigning the following folder permissions the issues gets resolved:

C:\ProgramData\AzureConnectedMachineAgent\Tokens (read)
C:\Packages\Plugins\Microsoft.Azure.Automation.HybridWorker.HybridWorkerForWindows**<version>**\HybridWorkerPackage\HybridWorkerAgent (full access)

However, the first permission is automatically removed after a few days and the original issue re-surfaces.

Is there a fix to this issue yet?

Thanks

Answer 5

I have experienced that these permissions get reset and removed from the mentioned folders when we run windows updates and the hybrid worker is updated.

Azure support acknowledged this issue but did not solve it, and i have colleagues who confirm it in other environments too.