Troubleshoot Azure Arc VM management

Applies to: Azure Stack HCI, version 23H2

This article provides guidance on how to collect logs and troubleshoot issues with Azure Arc virtual machines (VMs) in your Azure Stack HCI cluster. It also lists the limitations and known issues that currently exist with Azure Arc VM management.

Collect logs

You can collect logs to identify and troubleshoot issues with Arc VMs in your Azure Stack HCI system. Use these logs to gather key information before you contact Microsoft support for additional help.

Make sure you have the latest PowerShell module for log collection. To update the PowerShell module, run the following command:

#Update the PowerShell module
Install-Module -Name ArcHci -Force -Confirm:$false -SkipPublisherCheck -AcceptLicense

To collect logs for Arc VMs in your Azure Stack HCI cluster, run the following command:

$csv_path="<input-from-admin>"
$VMIP_1="<input-from-admin>"
az login --use-device-code
Get-ArcHCILogs -workDirectory $csv_path\ResourceBridge -kvaTokenPath $csv_path\ResourceBridge\kvatoken.tok -ip $VMIP_1

Where:

• $csv_path is the full path of the cluster shared volume provided for creating Arc Resource Bridge.

• $VMIP_1 is the IP address of the Arc Resource Bridge VM.

• Optionally, set the -logDir parameter to specify the path to the directory where the generated logs are stored. If you don't specify the path or the parameter, by default the logs are stored in your current working directory.

Troubleshoot Azure Arc VMs

This section describes the errors related to Azure Arc VM management and their recommended resolutions.

Failure when trying to enable guest management

When trying to run the command to enable guest management, you see the following error:

Error: Deployment failed. Correlation ID: 5d0c4921-78e0-4493-af16-dffee5cbf9d8. VM Spec validation failed for guest agent provisioning: Invalid managed identity. A system-assigned managed identity must be enabled in parent resource: Invalid Configuration

The above failure is because the managed identity wasn't created for this VM. System-assigned Managed Identity is required to enable guest management.

Resolution:

Follow these steps to verify that the Managed Identity isn't created for this VM and then enable System-assigned Managed Identity.

1. In the Azure portal, go to the VM. Browse to the Overview page. On the Properties tab, under Configuration, the Guest management should show as Disabled. Select the JSON View from the top right corner.

   

2. Under Identity parameter, the type should show as None.

   

3. To create managed identity, connect to the Azure Stack HCI server via RDP. Run the following command:

   az extension add --name connectedmachine
   

4. Verify that the connected machine CLI extension is installed on the cluster. Here's a sample output with the extension successfully installed. The connectedmachine indicates that version 0.7.0 is installed.

   [v-hostl]: PS C:\Clusterstorage\lnfrastructure_l\ArcHci> az version
   {
   "azure-cli": "2.53.0",
   "azure-cli-core": "2.53.0",
   "azure-cli-telemetry": "1.1.0",
   "extensions": {
       "akshybrid": "0.1.1",
       "arcappliance"^ "1.0.2",
       "connectedk8s": "1.5.4",
       "connectedmachine": "0.7.0",
       "customlocation": "0.1.3",
       "hybridaks": "0.2.4",
       "k8s-extension": "1.4.5",
       "stack-hci-vm": "0.1.8"
       }
   }
   [v-hostl]: PS C:\ClusterStorage\Infrastructure_l\ArcHci>
       ```
   

5. Run the following command to assign a system managed identity to the VM.

   az connectedmachine update --ids "<Resource Manager ID for the VM>" --set identity.type="SystemAssigned"
   

6. Go to the Azure portal and browse to the Overview page. The JSON View should indicate that the system managed identity is now assigned to the VM.

   

Failure deploying an Arc VM

You see the following error when trying to deploy an Arc VM on your Azure Stack HCI cluster:

Error: {"code":"ConflictingOperation","message":"Unable to process request 'Microsoft.AzureStackHCI/virtualMachineInstances'. There is already a previous running operation for resource '/subscriptions/<subscription ID>/resourceGroups/<Resource group name>/providers/Microsoft.HybridCompute/machines/<VM name>/providers/Microsoft.AzureStackHCI/virtualMachineInstances/default'. Please wait for the previous operation to complete."}

The above failure is because the SystemAssigned managed identity object isn't under the Microsoft.HybridCompute/machines resource type.

Resolution:

Verify in your deployment template that:

The SystemAssigned managed identity object is under Microsoft.HybridCompute/machines resource type and not under Microsoft.AzureStackHCI/VirtualMachineInstances resource type.

The deployment template should match the provided sample template. For more information, see the sample template in Create Arc virtual machines on Azure Stack HCI.

Failure deleting storage path

When trying to delete a storage path on your Azure Stack HCI cluster, you might see an error similar to the following message. Resource numbers and versions may vary in your scenario.

Error: "errorMessage" serviceClient returned an error during deletion: The storage container service returned an error during deletion: rpc error: code = Unknown desc = Container is in ACTIVE use by Resources [6:
- linux-cblmariner-0.2.0.10503
- windows-windows2019-0.2.0.10503
- windows-windows2022-0.2.0.10503
].
Remove all the Resources from this container, before trying to delete: In Use: Failed,

Resolution:

The images listed in the error message differ from typical workloads, which are represented as Azure Resource Manager (ARM) objects on the Azure portal and CLI. This error occurs because these images are directly downloaded onto the file system, which Azure couldn't recognize.

Follow these steps before trying to remove a storage path:

1. Remove the associated workloads and the images present on the storage path you want to delete. Look for the following prefixes on the image names: linux-cblmariner, windows-windows2019, windows-windows2022, windows_k8s, aks-image-merged, linux-K8s.
2. File a support ticket in the Azure portal.

Azure CLI installation isn't recognized

If your environment fails to recognize Azure CLI after installing it, run the following code block to add the Azure CLI installation path to the environment path.

        if ( -not( $env:PATH -like '*C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\wbin*') ) {
            $env:PATH += "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\wbin;"
            Write-Host "Updated path $env:PATH"
        }

Next steps

• Azure Arc VM management FAQs