Runbooks remain queued when using custom hybrid worker credentials, work fine as LocalSystem

Andrew VO 96 Reputation points
2022-06-07T12:58:29.07+00:00

I'm working with an Arc enabled server/extension based V2 HRW. For some reason runbooks do not execute when the HRW is configured to use an Azure credential. The portal shows them as never leaving the queued state. The Microsoft-SMA event log on the HRW indicates the sandbox was created, then follows with Win32 Process Exited with code [2148734720]. The Azure credential being used is a domain user that is a member of the local administrators group on the HRW.

When I change the HRW to use the default credential (LocalSystem), runbooks execute.

I know the credential is good, as I can use it with cmdlets in runbooks that support -Credential. Leaving it to use LocalSystem would be fine if all cmdlets used -Credential, but the Exchange cmdlets like New-RemoteMailbox don't--which is why we need the HRW to run runbooks/jobs under a credential.

Any insight is appreciated, and thanks!

AVO

Azure Automation
Azure Automation
An Azure service that is used to automate, configure, and install updates across hybrid environments.
684 questions
{count} votes

Accepted answer
  1. AnuragSingh-MSFT 9,976 Reputation points Microsoft Employee
    2022-06-11T07:35:37.663+00:00

    @Andrew VO , @Christian Hiebler , thank you for bringing this issue to our attention. After a bit of test and research, here are my findings about this issue. I will summarize it below, so that it can help others also looking for answers to similar questions.

    Issue:
    Based on the behavior mentioned above, when you use a different user (apart from the default Local System) as Run as credential for Hybrid worker, the runbook jobs targeted to run on this hybrid worker remains in Queued state for approximately 15 minutes and then goes in Suspended state. This happens despite having this user added to Administrators group on the Worker Machine and explicit permissions have to be added to some directories to get this to work.

    Cause
    This happens because of Filtered Token feature of User Access Control (UAC) on Windows Server - When a user with administrative or other powerful privileges or group memberships logs on, Windows creates two access tokens to represent the user account. The unfiltered token has all the user's group memberships and privileges. The filtered token represents the user with the equivalent of standard user rights. By default, this filtered token is used to run the user's programs. The unfiltered token is associated only with elevated programs. To better protect those users who are members of the local Administrators group, we implement UAC restrictions on the network. This mechanism helps prevent loopback attacks. This mechanism also helps prevent local malicious software from running remotely with administrative rights.
    For more details, please refer to the following article:
    How to disable User Account Control (UAC) on Windows Server

    Resolution and workaround
    The resolution of this issue would be - to add the following permissions to the user used in run as credential for Hybrid Runbook Worker. These permissions will have to be added explicitly instead of adding the user to Administrators because by default, the Filtered Token would disallow the user from accessing protected resources/resources requiring elevated permissions.

    Registry path

    HKLM\SYSTEM\CurrentControlSet\Services\EventLog (read)
    HKLM\SYSTEM\CurrentControlSet\Services\WinSock2\Parameters (full access)
    HKLM\SOFTWARE\Microsoft\Wbem\CIMOM (full access)
    HKLM\Software\Policies\Microsoft\SystemCertificates\Root (full access)
    HKLM\Software\Microsoft\SystemCertificates (full access)
    HKLM\Software\Microsoft\EnterpriseCertificates (full access)
    HKLM\software\Microsoft\HybridRunbookWorker (full access)
    HKLM\software\Microsoft\HybridRunbookWorkerV2 (full access)
    HKEY_CURRENT_USER\SOFTWARE\Policies\Microsoft\SystemCertificates\Disallowed (full access)
    HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Setup\PnpLockdownFiles (full access)

    Folders

    C:\ProgramData\AzureConnectedMachineAgent\Tokens (read)
    C:\Packages\Plugins\Microsoft.Azure.Automation.HybridWorker.HybridWorkerForWindows\**<version>**\HybridWorkerPackage\HybridWorkerAgent (full access)

    Please refer to this link for details - Use a credential asset for a Hybrid Runbook Worker group


    The workaround to adding explicit permission as mentioned above to the user used in "run as credential" for Hybrid Runbook Worker will involve the following steps. Please note this would be a not recommended workaround, as it would disable the default UAC security check in place.

    1. Add the user to local "Administrators" group
    2. Disable the local group policy - User Account Control: Run all administrators in Admin Approval Mode, available in Group Policy editor here: Local Security Policy --> Security Settings --> Local Policies --> Security Options.
    210409-image.png
    3. Reboot the machine

    Please let me know if you have any questions.


    Please 'Accept as answer' and ‘Upvote’ if it helped so that it can help others in the community looking for help on similar topics.


1 additional answer

Sort by: Most helpful
  1. Andrew VO 96 Reputation points
    2022-06-07T23:43:50.773+00:00

    RunAs account on the local HRW was member of local administrator, yet missing permissions to various subdirectories within C:\Program Files\AzureConnectedMachineAgent and C:\ProgramData\AzureConnectedMachineAgent directory structures.