Moving to Extension based HW (v2) will be a problem for us

Question

Hi all,

We have an Azure Automation environment we use to run standard changes, monthly scripts, reports, etc. on customer environments.

For when we need to do something on a server, we use Hybrid Workers we install @ the customer sides. Usually this is a server that's in the Azure tenant of the customer. With Hybrid Worker Agent (V1) this worked fine, but with the new Extension based Hybrid Worker (V2) we are in trouble.

The servers are already connected with the customers azure environment; thus, we cannot install Azure Arc on them without a firewall blocking outbound IMDS, disable the Azure Guest Agent, and Remove any extension from the server (in the customer environment).

https://learn.microsoft.com/en-us/azure/azure-arc/servers/plan-evaluate-on-azure-virtual-machine#reconfigure-azure-vm

By doing this we CAN convert the server to Azure Arc, but as stated on the learn page, this is for testing and development only.
And worse, we cannot install the extension from the Azure Arc environment, because it has to use IMDS, but enabling IMDS will cause the server to connect to it's on azure tenant and it won't be available for us to use in our Azure Automation environment.

So, I successfully added it as an Azure Arc server, I eventually enabled a Managed Identity (on the Customers Tenant env..., because there is no option in Azure Arc?), I reenabled IMDS, but unfortunately it tries to search for the Azure Automation environment in the wrong tenant and this is the error I received:

Extension Message: [Internal Error] The Hybrid Worker Extension failed to install: {"Message":"Specified machineId is not associated with automation account. AccountId AZAUTOMATIONGUIDINOURTENANT, machineId /subscriptions/CUSTOMERSUBSCRIPTIONID/resourcegroups/SERVERRGINCUSTOMERTENANT/providers/Microsoft.Compute/virtualMachines/SERVERNAME."} .
More information about the failure can be found in the logs located under 'C:\ProgramData\GuestConfig\extension_logs\Microsoft.Azure.Automation.HybridWorker.HybridWorkerForWindows' on the VM. For more troubleshooting, please follow the steps mentioned here: 'https://aka.ms/troubleshoot-hybrid-runbook-worker-v2', HybridWorkerExtensionLog: [2023-10-06 07:06:31Z] Extension request for sequence number 0 attempting to acquire lock.0 file
[2023-10-06 07:06:31Z] Extension request for sequence number 0 attempting to acquire lock file
[2023-10-06 07:06:32Z] File lock does not exist: begin processing
[2023-10-06 07:06:32Z] Starting HybridWorker Extension ...
[2023-10-06 07:06:32Z] Getting handler execution status HKLM:\SOFTWARE\Microsoft\Azure\HybridWorker\1.1.12\Status ...
[2023-10-06 07:06:33Z] Error while handling extension configuration...
[2023-10-06 07:06:34Z] Error Writing Events file The property 'eventsFolder' cannot be found on this object. Verify that the property exists.
[2023-10-06 07:06:35Z] Error encountered handling extension configuration...
[2023-10-06 07:06:35Z] [ERROR] {"Message":"Specified machineId is not associated with automation account. AccountId AZAUTOMATIONGUIDINOURTENANT, machineId /subscriptions/CUSTOMERSUBSCRIPTIONID/resourcegroups/SERVERRGINCUSTOMERTENANT/providers/Microsoft.Compute/virtualMachines/SERVERNAME."} 
[2023-10-06 07:06:35Z] {
    "Exception":  {
                      "Message":  "{\"Message\":\"Specified machineId is not associated with automation account. AccountId AZAUTOMATIONGUIDINOURTENANT, machineId /subscriptions/CUSTOMERSUBSCRIPTIONID/resourcegroups/SERVERRGINCUSTOMERTENANT/providers/Microsoft.Compute/virtualMachines/SERVERNAME.\"} ",
                      "Data":  {
                                   "Code":  52
                               },
                      "InnerException":  null,
                      "TargetSite":  null,
                      "StackTrace":  null,
                      "HelpLink":  null,
                      "Source":  null,
                      "HResult":  -2146233088
                  },
    "TargetObject":  null,
    "CategoryInfo":  {
                         "Category":  0,
                         "Activity":  "",
                         "Reason":  "Exception",
                         "TargetName":  "",
                         "TargetType":  ""
                     },
    "FullyQualifiedErrorId":  "HybridWorkerHandlerTerminatingError",
    "ErrorDetails":  null,
    "InvocationInfo":  {
                           "MyCommand":  null,
                           "BoundParameters":  {

                                               },
                           "UnboundArguments":  [

                                                ],
                           "ScriptLineNumber":  289,
                           "OffsetInLine":  25,
                           "HistoryId":  -1,
                           "ScriptName":  "C:\\Packages\\Plugins\\Microsoft.Azure.Automation.HybridWorker.HybridWorkerForWindows\\1.1.12\\bin\\HybridWorkerExtensionHandler.psm1",
                           "Line":  "                        throw (New-HandlerTerminatingError -Code $HybridWorker_Status.InstallError -Message $executionStatus.ErrorMessage)    \r\n",
                           "PositionMessage":  "At C:\\Packages\\Plugins\\Microsoft.Azure.Automation.HybridWorker.HybridWorkerForWindows\\1.1.12\\bin\\HybridWorkerExtensionHandler.psm1:289 char:25\r\n+ ...             throw (New-HandlerTerminatingError -Code $HybridWorker_St ...\r\n+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~",
                           "PSScriptRoot":  "C:\\Packages\\Plugins\\Microsoft.Azure.Automation.HybridWorker.HybridWorkerForWindows\\1.1.12\\bin",
                           "PSCommandPath":  "C:\\Packages\\Plugins\\Microsoft.Azure.Automation.HybridWorker.HybridWorkerForWindows\\1.1.12\\bin\\HybridWorkerExtensionHandler.psm1",
                           "InvocationName":  "",
                           "PipelineLength":  0,
                           "PipelinePosition":  0,
                           "ExpectingInput":  false,
                           "CommandOrigin":  1,
                           "DisplayScriptPosition":  null
                       },
    "ScriptStackTrace":  "at \u003cScriptBlock\u003e, C:\\Packages\\Plugins\\Microsoft.Azure.Automation.HybridWorker.HybridWorkerForWindows\\1.1.12\\bin\\HybridWorkerExtensionHandler.psm1: line 289\r\nat Invoke-HybridWorkerExtensionSingleInstance, C:\\Packages\\Plugins\\Microsoft.Azure.Automation.HybridWorker.HybridWorkerForWindows\\1.1.12\\bin\\HybridWorkerExtensionHandler.psm1: line 592\r\nat Invoke-HybridWorkerExtension, C:\\Packages\\Plugins\\Microsoft.Azure.Automation.HybridWorker.HybridWorkerForWindows\\1.1.12\\bin\\HybridWorkerExtensionHandler.psm1: line 154\r\nat \u003cScriptBlock\u003e, C:\\Packages\\Plugins\\Microsoft.Azure.Automation.HybridWorker.HybridWorkerForWindows\\1.1.12\\bin\\enable.ps1: line 38\r\nat \u003cScriptBlock\u003e, \u003cNo file\u003e: line 1",
    "PipelineIterationInfo":  [

                              ]
}
[2023-10-06 07:06:37Z] Setting install status to 'Error' (HKLM:\SOFTWARE\Microsoft\Azure\HybridWorker\1.1.12\InstallStatus)
[2023-10-06 07:06:38Z] Settings handler status to 'error' (C:\Packages\Plugins\Microsoft.Azure.Automation.HybridWorker.HybridWorkerForWindows\1.1.12\status\0.status)
Extension Error: 
C:\Packages\Plugins\Microsoft.Azure.Automation.HybridWorker.HybridWorkerForWindows\1.1.12>powershell.exe -NoProfile -NonInteractive -ExecutionPolicy RemoteSigned C:\Packages\Plugins\Microsoft.Azure.Automation.HybridWorker.HybridWorkerForWindows\1.1.12\bin\install.ps1  0<nul 
VERBOSE: Starting installation of the hybrid worker extension...
VERBOSE: Setting up Hybrid worker source paths...
VERBOSE: Deleting the old binaries...
VERBOSE: Determining method to expand zip file...
VERBOSE: Expanding zip file with built-in cmdlet...
Copyting Hybrid worker agent helper files...
VERBOSE: Hybrid Worker installation complete.

C:\Packages\Plugins\Microsoft.Azure.Automation.HybridWorker.HybridWorkerForWindows\1.1.12>
C:\Packages\Plugins\Microsoft.Azure.Automation.HybridWorker.HybridWorkerForWindows\1.1.12>exit 0

C:\Packages\Plugins\Microsoft.Azure.Automation.HybridWorker.HybridWorkerForWindows\1.1.12>exit 0 

C:\Packages\Plugins\Microsoft.Azure.Automation.HybridWorker.HybridWorkerForWindows\1.1.12>rem

C:\Packages\Plugins\Microsoft.Azure.Automation.HybridWorker.HybridWorkerForWindows\1.1.12>rem A wrapper around enable.ps1 (executes private binaries and waits for WMF install to complete) 

C:\Packages\Plugins\Microsoft.Azure.Automation.HybridWorker.HybridWorkerForWindows\1.1.12>rem  

C:\Packages\Plugins\Microsoft.Azure.Automation.HybridWorker.HybridWorkerForWindows\1.1.12>if exist C:\Packages\Plugins\Microsoft.Azure.Automation.HybridWorker.HybridWorkerForWindows\1.1.12\bin_dev (set bin_root=C:\Packages\Plugins\Microsoft.Azure.Automation.HybridWorker.HybridWorkerForWindows\1.1.12\bin_dev )  else (set bin_root=C:\Packages\Plugins\Microsoft.Azure.Automation.HybridWorker.HybridWorkerForWindows\1.1.12\bin ) 

C:\Packages\Plugins\Microsoft.Azure.Automation.HybridWorker.HybridWorkerForWindows\1.1.12>if exist C:\Packages\Plugins\Microsoft.Azure.Automation.HybridWorker.HybridWorkerForWindows\1.1.12\HybridWorkerPackage (set hybridworker_root=C:\Packages\Plugins\Microsoft.Azure.Automation.HybridWorker.HybridWorkerForWindows\1.1.12\HybridWorkerPackage )  else (
echo hybrid worker installation directory not found. Exiting extension installation... Hybrid worker root.. ...  
 exit 1 
) 

C:\Packages\Plugins\Microsoft.Azure.Automation.HybridWorker.HybridWorkerForWindows\1.1.12>if not exist C:\Packages\Plugins\Microsoft.Azure.Automation.HybridWorker.HybridWorkerForWindows\1.1.12\HybridWorkerPackage\HybridWorkerAgent (
echo Hybrid worker folder not found. Installing Hybrid Worker Agent... Hybrid Worker root : C:\Packages\Plugins\Microsoft.Azure.Automation.HybridWorker.HybridWorkerForWindows\1.1.12\HybridWorkerPackage...  
  
 C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive -ExecutionPolicy RemoteSigned -File C:\Packages\Plugins\Microsoft.Azure.Automation.HybridWorker.HybridWorkerForWindows\1.1.12\bin\install.ps1  0<nul 
) 

C:\Packages\Plugins\Microsoft.Azure.Automation.HybridWorker.HybridWorkerForWindows\1.1.12>echo Starting Hybrid Worker Extension ... 
Starting Hybrid Worker Extension ...

C:\Packages\Plugins\Microsoft.Azure.Automation.HybridWorker.HybridWorkerForWindows\1.1.12>C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive -ExecutionPolicy RemoteSigned -Command "& {C:\Packages\Plugins\Microsoft.Azure.Automation.HybridWorker.HybridWorkerForWindows\1.1.12\bin\enable.ps1 -StartAsyncProcess}"   0<nul 
VERBOSE: [2023-10-06 07:06:05Z] Retrieving status of current request ...
VERBOSE: [2023-10-06 07:06:07Z] Getting handler execution status 
HKLM:\SOFTWARE\Microsoft\Azure\HybridWorker\1.1.12\Status ...
VERBOSE: [2023-10-06 07:06:08Z]     Sequence Number     : 0
VERBOSE: [2023-10-06 07:06:08Z]     Previous Sequence   : 0
VERBOSE: [2023-10-06 07:06:08Z]     UseExisting         : False
VERBOSE: [2023-10-06 07:06:08Z]     State               : Error
VERBOSE: [2023-10-06 07:06:08Z]     PS Version          : 5.1.17763.4840
VERBOSE: [2023-10-06 07:06:08Z] Resuming request with sequence number 0 ...
VERBOSE: [2023-10-06 07:06:08Z] Log: 
C:\ProgramData\GuestConfig\extension_logs\Microsoft.Azure.Automation.HybridWorker.HybridWorkerForWindows\HybridWorkerEx
tensionHandler.0.20231006-070608.log
VERBOSE: [2023-10-06 07:06:09Z] Starting asynchronous enable process...

Handles  NPM(K)    PM(K)      WS(K)     CPU(s)     Id  SI ProcessName                                                  
-------  ------    -----      -----     ------     --  -- -----------                                                  
      0       2     1532         88       0.03   4760   0 cmd                                                          



C:\Packages\Plugins\Microsoft.Azure.Automation.HybridWorker.HybridWorkerForWindows\1.1.12>
C:\Packages\Plugins\Microsoft.Azure.Automation.HybridWorker.HybridWorkerForWindows\1.1.12>exit 0

C:\Packages\Plugins\Microsoft.Azure.Automation.HybridWorker.HybridWorkerForWindows\1.1.12>exit 0

Answer 1

Bas Wijdenes, thank you for posting this question.

Based on my understanding, the scripts that you run in Hybrid runbook workers are to configure these machines (Hybrid workers) and collect information. This is triggered from your subscription which I assume is in a different tenant than that of your customer's subscription.

In this regard, note that managed identity cannot be used in a cross-tenant scenario - Can I use a managed identity to access a resource in a different directory/tenant?

Also, as you mentioned IMDS tries to connect to the home tenant only as specified.

In this current scenario, I would suggest using Azure Lighthouse to delegate yourself as a service provider. With Lighthouse enabled, you would be able to perform control plane level operations which also includes being able to add servers in your customer's tenant as Hybrid worker of your own tenant's Automation Account. When you add hybrid workers from other tenant using portal, they get added as extension based HW and much of the same functionality remains as you have in current scenario. With this solution, Azure Arc would not be required as well.

For more details, see Azure Lighthouse: The managed service provider perspective

Hope this helps.

If the answer did not help, please add more context/follow-up question for it, and we will help you out. Else, if the answer helped, please click Accept answer so that it can help others in the community looking for help on similar topics.