This article identifies critical known issues and their workarounds in Azure Local.
These release notes are continuously updated, and as critical issues requiring a workaround are discovered, they're added. Before you deploy your Azure Local instance, carefully review the information contained here.
Importante
For information about supported update paths for this release, see Release information.
This software release maps to software version number 2411.3.2.
Importante
The new deployments of this software use the 2411.3.2 build. You can also update from 2411.2.
Release notes for this version include the issues fixed in this release, known issues in this release, and release note issues carried over from previous versions.
Microsoft is not aware of any known issues in this release.
Known issues from previous releases
The following table lists the known issues from previous releases:
Feature
Issue
Workaround
Update
When monitoring update progress in the Azure Update Management portal, the progress might appear to not have updated for several hours.
Run Get-SolutionUpdate on one of the cluster nodes. If an update object is returned, the update might be taking longer than expected but it is progressing. If an update object is not returned, the update may be stalled. For detailed steps on how to resolve this issue, see the Troubleshooting guide.
Deployment
Validation times out due to timestamp deserialization.
When deploying the operating system, select English (United States) as the installation language, as well as the time and currency format. For detailed remediation steps, see the troubleshooting guide in the Azure Local Supportability GitHub repository.
Update
When updating from version 2408.2.7 to 2411.0.24, the update process could fail with the following error message: Type 'CauPreRequisites' of Role 'CAU' raised an exception: Could not finish cau prerequisites due to error 'Cannot remove item C:\UpdateDistribution\<any_file_name>: Access to the path is denied.'
With the 2411 release, solution and Solution Builder Extension update aren't combined in a single update run.
To apply a Solution Builder Extension package, you need a separate update run.
Update
When applying solution update in this release, the update can fail. This will occur only if the update was started prior to November 26. The issue that causes the failure can result in one of the following error messages:
Error 1 - The step "update ARB and extension" error "Clear-AzContext failed with 0 and Exception calling "Initialize" with "1" argument(s): "Object reference not set to an instance of an object." at "Clear-AzPowerShellCache".
Error 2 - The step "EvalTVMFlow" error "CloudEngine.Actions.InterfaceInvocationFailedException: Type 'EvalTVMFlow' of Role 'ArcIntegration' raised an exception: This module requires Az.Accounts version 3.0.5. An earlier version of Az.Accounts is imported in the current PowerShell session. Please open a new session before importing this module. This error could indicate that multiple incompatible versions of the Azure PowerShell cmdlets are installed on your system. Please see https://aka.ms/azps-version-error for troubleshooting information."
Depending on the version of PowerShell modules, the above error could be reported for both versions 3.0.4 and 3.0.5.
After you repair a node and run the command Set-AzureStackLCMUserPassword, you may encounter the following error:
CloudEngine.Actions.InterfaceInvocationFailedException: Type 'ValidateCredentials' of Role 'SecretRotation' raised an exception: Cannot load encryption certificate. The certificate setting 'CN=DscEncryptionCert' does not represent a valid base-64 encoded certificate, nor does it represent a valid certificate by file, directory, thumbprint, or subject name. at Validate-Credentials
Follow these steps to mitigate the issue:
$NewPassword = <Provide new password as secure string>
$OldPassword = <Provide the old/current password as secure string>
if ($eceClusterGroup.State -ne "Online") {Write-AzsSecurityError -Message "ECE cluster group is not in an Online state. Cannot continue with password rotation." -ErrRecord $_}
3. Update the ECE with the new password:
Write-AzsSecurityVerbose -Message "Updating password in ECE" -Verbose
Using an exported Azure VM OS disk as a VHD to create a gallery image for provisioning an Arc VM is unsupported.
Run the command restart-service mochostagent to restart the mochostagent service.
Networking
When a node is configured with a proxy server that has capital letters in its address, such as HTTPS://10.100.000.00:8080, Arc extensions fail to install or update on the node in existing builds, including version 2408.1. However, the node remains Arc connected.
Follow these steps to mitigate the issue:
1. Set the environment values in lowercase. [System.Environment]::SetEnvironmentVariable("HTTPS_PROXY", "https://10.100.000.00:8080", "Machine").
2. Validate that the values were set. [System.Environment]::GetEnvironmentVariable("HTTPS_PROXY", "Machine").
3. Restart Arc services.
Restart-Service himds
Restart-Service ExtensionService
Restart-Service GCArcService
4. Signal the AzcmaAgent with the lowercase proxy information.
& 'C:\Program Files\AzureConnectedMachineAgent\azcmagent.exe' config set proxy.url https://10.100.000.00:8080
& 'C:\Program Files\AzureConnectedMachineAgent\azcmagent.exe' config list
Networking
When Arc machines go down, the "All Clusters" page, in the new portal experience shows a "PartiallyConnected" or "Not Connected Recently status. Even when the Arc machines become healthy, they may not show a "Connected" status.
There's no known workaround for this issue. To check the connectivity status, use the old experience to see if it shows as "Connected".
Arc VM management
The Mochostagent service might appear to be running but can get stuck without updating logs for over a month. You can identify this issue by checking the service logs in C:\programdata\mochostagent\logs to see if logs are being updated.
Run the following command to restart the mochostagent service: restart-service mochostagent.
Upgrade
When upgrading the stamp from 2311 or prior builds to 2408 or later, add node and repair node operations may fail. For example, you could see an error: Type 'AddAsZHostToDomain' of Role 'BareMetal' raised an exception.
There's no workaround in this release. If you encounter this issue, contact Microsoft Support to determine next steps.
Update
When viewing the readiness check results for an Azure Local instance via the Azure Update Manager, there might be multiple readiness checks with the same name.
There's no known workaround in this release. Select View details to view specific information about the readiness check.
Update
There's an intermittent issue in this release when the Azure portal incorrectly reports the update status as Failed to update or In progress though the update is complete.
$Update = get-solutionupdate| ? version -eq "<version string>"
Replace the version string with the version you're running. For example, "10.2405.0.23".
$Update.state
If the update status is Installed, no further action is required on your part. Azure portal refreshes the status correctly within 24 hours. To refresh the status sooner, follow these steps on one of the nodes. Restart the Cloud Management cluster group. Stop-ClusterGroup "Cloud Management" Start-ClusterGroup "Cloud Management"
Update
During an initial MOC update, a failure occurs due to the target MOC version not being found in the catalog cache. The follow-up updates and retries show MOC in the target version, without the update succeeding, and as a result the Arc Resource Bridge update fails.
To validate this issue, collect the update logs using Troubleshoot solution updates for Azure Local. The log files should show a similar error message (current version might differ in the error message):
[ERROR: { "errorCode": "InvalidEntityError", "errorResponse": "{\n\"message\": \"the cloud fabric (MOC) is currently at version v0.13.1. A minimum version of 0.15.0 is required for compatibility\"\n}" }]
Follow these steps to mitigate the issue:
1. To find the MOC agent version, run the following command: 'C:\Program Files\AksHci\wssdcloudagent.exe' version.
2. Use the output of the command to find the MOC version from the table below that matches the agent version, and set $initialMocVersion to that MOC version. Set the $targetMocVersion by finding the Azure Local build you're updating to and get the matching MOC version from the following table. Use these values in the mitigation script provided below:
Build
MOC version
Agent version
2311.2
1.0.24.10106
v0.13.0-6-gf13a73f7, v0.11.0-alpha.38,01/06/2024
2402
1.0.25.10203
v0.14.0, v0.13.1, 02/02/2024
2402.1
1.0.25.10302
v0.14.0, v0.13.1, 03/02/2024
2402.2
1.1.1.10314
v0.16.0-1-g04bf0dec, v0.15.1, 03/14/2024
2405/2402.3
1.3.0.10418
v0.17.1, v0.16.5, 04/18/2024
For example, if the agent version is v0.13.0-6-gf13a73f7, v0.11.0-alpha.38,01/06/2024, then $initialMocVersion = "1.0.24.10106" and if you're updating to 2405.0.23, then $targetMocVersion = "1.3.0.10418".
3. Run the following PowerShell commands on the first node:
$initialMocVersion = "<initial version determined from step 2>" $targetMocVersion = "<target version determined from step 2>"
# Import MOC module twice import-module moc import-module moc $verbosePreference = "Continue"
# Clear the SFS catalog cache Remove-Item (Get-MocConfig).manifestCache
# Set version to the current MOC version prior to update, and set state as update failed Set-MocConfigValue -name "version" -value $initialMocVersion Set-MocConfigValue -name "installState" -value ([InstallState]::UpdateFailed)
# Rerun the MOC update to desired version Update-Moc -version $targetMocVersion
4. Resume the update.
Deployment
In some instances, during the registration of Azure Local machines, this error might be seen in the debug logs: Encountered internal server error. One of the mandatory extensions for device deployment might not be installed.
AKS cluster creation fails with the Error: Invalid AKS network resource id. This issue can occur when the associated logical network name has an underscore.
Underscores aren't supported in logical network names. Make sure to not use underscore in the names for logical networks deployed on your Azure Local.
Repair server
In rare instances, the Repair-Server operation fails with the HealthServiceWaitForDriveFW error. In these cases, the old drives from the repaired node aren't removed and new disks are stuck in the maintenance mode.
To prevent this issue, make sure that you DO NOT drain the node either via the Windows Admin Center or using the Suspend-ClusterNode -Drain PowerShell cmdlet before you start Repair-Server. If the issue occurs, contact Microsoft Support for next steps.
Repair server
This issue is seen when the single node Azure Local instance is updated from 2311 to 2402 and then the Repair-Server is performed. The repair operation fails.
Before you repair the single node, follow these steps: 1. Run version 2402 for the ADPrepTool. Follow the steps in Prepare Active Directory. This action is quick and adds the required permissions to the Organizational Unit (OU). 2. Move the computer object from Computers segment to the root OU. Run the following command: Get-ADComputer <HOSTNAME> | Move-ADObject -TargetPath "<OU path>"
Deployment
If you prepare the Active Directory on your own (not using the script and procedure provided by Microsoft), your Active Directory validation could fail with missing Generic All permission. This is due to an issue in the validation check that checks for a dedicated permission entry for msFVE-RecoverInformationobjects – General – Permissions Full control, which is required for BitLocker recovery.
Use the Prepare AD script method or if using your own method, make sure to assign the specific permission msFVE-RecoverInformationobjects – General – Permissions Full control.
Deployment
There's a rare issue in this release where the DNS record is deleted during the Azure Local deployment. When that occurs, the following exception is seen: Type 'PropagatePublicRootCertificate' of Role 'ASCA' raised an exception:<br>The operation on computer 'ASB88RQ22U09' failed: WinRM cannot process the request. The following error occurred while using Kerberos authentication: Cannot find the computer ASB88RQ22U09.local. Verify that the computer exists on the network and that the name provided is spelled correctly at PropagatePublicRootCertificate, C:\NugetStore\Microsoft.AzureStack, at Orchestration.Roles.CertificateAuthority.10.2402.0.14\content\Classes\ASCA\ASCA.psm1: line 38, at C:\CloudDeployment\ECEngine\InvokeInterfaceInternal.psm1: line 127,at Invoke-EceInterfaceInternal, C:\CloudDeployment\ECEngine\InvokeInterfaceInternal.psm1: line 123.
Check the DNS server to see if any DNS records of the nodes are missing. Apply the following mitigation on the nodes where its DNS record is missing.
Restart the DNS client service. Open a PowerShell session and run the following cmdlet on the affected node: Taskkill /f /fi "SERVICES eq dnscache"
Deployment
In this release, there's a remote task failure on a multi-node deployment that results in the following exception: ECE RemoteTask orchestration failure with ASRR1N42R01U31 (node pingable - True): A WebException occurred while sending a RestRequest. WebException.Status: ConnectFailure on [https://<URL>](https://<URL>).
The mitigation is to restart the ECE agent on the affected node. On your machine, open a PowerShell session and run the following command: Restart-Service ECEAgent.
Add server
In this release and previous releases, when adding a machine to the system, isn't possible to update the proxy bypass list string to include the new machine. Updating environment variables proxy bypass list on the hosts won't update the proxy bypass list on Azure Resource Bridge or AKS.
There's no workaround in this release. If you encounter this issue, contact Microsoft Support to determine next steps.
Add/Repair server
In this release, when adding or repairing a machine, a failure is seen when the software load balancer or network controller VM certificates are being copied from the existing nodes. The failure is because these certificates weren't generated during the deployment/update.
There's no workaround in this release. If you encounter this issue, contact Microsoft Support to determine next steps.
Deployment
In this release, there's a transient issue resulting in the deployment failure with the following exception: Type 'SyncDiagnosticLevel' of Role 'ObservabilityConfig' raised an exception:*<br>*Syncing Diagnostic Level failed with error: The Diagnostic Level does not match. Portal was not set to Enhanced, instead is Basic.
As this is a transient issue, retrying the deployment should fix this. For more information, see how to Rerun the deployment.
Deployment
In this release, there's an issue with the Secrets URI/location field. This is a required field that is marked Not mandatory and results in Azure Resource Manager template deployment failures.
Use the sample parameters file in the Deploy Azure Local via Azure Resource Manager template to ensure that all the inputs are provided in the required format and then try the deployment. If there's a failed deployment, you must also clean up the following resources before you Rerun the deployment: 1. Delete C:\EceStore. 2. Delete C:\CloudDeployment. 3. Delete C:\nugetstore. 4. Remove-Item HKLM:\Software\Microsoft\LCMAzureStackStampInformation.
Security
For new deployments, Secured-core capable devices won't have Dynamic Root of Measurement (DRTM) enabled by default. If you try to enable (DRTM) using the Enable-AzSSecurity cmdlet, you see an error that DRTM setting isn't supported in the current release. Microsoft recommends defense in depth, and UEFI Secure Boot still protects the components in the Static Root of Trust (SRT) boot chain by ensuring that they're loaded only when they're signed and verified.
DRTM isn't supported in this release.
Networking
An environment check fails when a proxy server is used. By design, the bypass list is different for winhttp and wininet, which causes the validation check to fail.
Follow these workaround steps:
1. Clear the proxy bypass list prior to the health check and before starting the deployment or the update.
2. After passing the check, wait for the deployment or update to fail.
3. Set your proxy bypass list again.
Arc VM management
Deployment or update of Arc Resource Bridge could fail when the automatically generated temporary SPN secret during this operation, starts with a hyphen.
Retry the deployment/update. The retry should regenerate the SPN secret and the operation will likely succeed.
Arc VM management
Arc Extensions on Arc VMs stay in "Creating" state indefinitely.
Sign in to the VM, open a command prompt, and type the following: Windows: notepad C:\ProgramData\AzureConnectedMachineAgent\Config\agentconfig.json Linux: sudo vi /var/opt/azcmagent/agentconfig.json Next, find the resourcename property. Delete the GUID that is appended to the end of the resource name, so this property matches the name of the VM. Then restart the VM.
Arc VM management
When a new machine is added to an Azure Local instance, storage path isn't created automatically for the newly created volume.
You can manually create a storage path for any new volumes. For more information, see Create a storage path.
Arc VM management
Restart of Arc VM operation completes after approximately 20 minutes although the VM itself restarts in about a minute.
There's no known workaround in this release.
Arc VM management
In some instances, the status of the logical network shows as Failed in Azure portal. This occurs when you try to delete the logical network without first deleting any resources such as network interfaces associated with that logical network. You should still be able to create resources on this logical network. The status is misleading in this instance.
If the status of this logical network was Succeeded at the time when this network was provisioned, then you can continue to create resources on this network.
Arc VM management
In this release, when you update a VM with a data disk attached to it using the Azure CLI, the operation fails with the following error message: Couldn't find a virtual hard disk with the name.
In rare instances, you may encounter this error while updating your Azure Local instance: Type 'UpdateArbAndExtensions' of Role 'MocArb' raised an exception: Exception Upgrading ARB and Extension in step [UpgradeArbAndExtensions :Get-ArcHciConfig] UpgradeArb: Invalid applianceyaml = [C:\AksHci\hci-appliance.yaml].
If you see this issue, contact Microsoft Support to assist you with the next steps.
Networking
There's an infrequent DNS client issue in this release that causes the deployment to fail on a two-node system with a DNS resolution error: A WebException occurred while sending a RestRequest. WebException.Status: NameResolutionFailure. As a result of the bug, the DNS record of the second node is deleted soon after it's created resulting in a DNS error.
Restart the machine. This operation registers the DNS record, which prevents it from getting deleted.
Azure portal
In some instances, the Azure portal might take a while to update and the view might not be current.
You might need to wait for 30 minutes or more to see the updated view.
Arc VM management
Deleting a network interface on an Arc VM from Azure portal doesn't work in this release.
Providing the OU name in an incorrect syntax isn't detected in the Azure portal. The incorrect syntax includes unsupported characters such as &,",',<,>. The incorrect syntax is detected at a later step during system validation.
Make sure that the OU path syntax is correct and doesn't include unsupported characters.
Deployment
Deployments via Azure Resource Manager time out after 2 hours. Deployments that exceed 2 hours show up as failed in the resource group though the system is successfully created.
To monitor the deployment in the Azure portal, go to the Azure Local instance resource and then go to new Deployments entry.
Azure Site Recovery
Azure Site Recovery can't be installed on an Azure Local instance in this release.
There's no known workaround in this release.
Update
When updating the Azure Local instance via the Azure Update Manager, the update progress and results may not be visible in the Azure portal.
To work around this issue, on each node, add the following registry key (no value needed):
Then on one of the nodes, restart the Cloud Management cluster group.
Stop-ClusterGroup "Cloud Management"
Start-ClusterGroup "Cloud Management"
This won't fully remediate the issue as the progress details may still not be displayed for a duration of the update process. To get the latest update details, you can Retrieve the update progress with PowerShell.
Update
In rare instances, if a failed update is stuck in an In progress state in Azure Update Manager, the Try again button is disabled.
To resume the update, run the following PowerShell command: Get-SolutionUpdate|Start-SolutionUpdate.
Update
In some cases, SolutionUpdate commands could fail if run after the Send-DiagnosticData command.
Make sure to close the PowerShell session used for Send-DiagnosticData. Open a new PowerShell session and use it for SolutionUpdate commands.
Update
In rare instances, when applying an update from 2311.0.24 to 2311.2.4, system status reports In Progress instead of expected Failed to update.
Retry the update. If the issue persists, contact Microsoft Support.
Update
Attempts to install solution updates can fail at the end of the CAU steps with: There was a failure in a Common Information Model (CIM) operation, that is, an operation performed by software that Cluster-Aware Updating depends on. This rare issue occurs if the Cluster Name or Cluster IP Address resources fail to start after a node reboot and is most typical in small deployments.
If you encounter this issue, contact Microsoft Support for next steps. They can work with you to manually restart the Azure Local resources and resume the update as needed.
Update
When applying a system update to 10.2402.3.11, the Get-SolutionUpdate cmdlet may not respond and eventually fails with a RequestTimeoutException after approximately 10 minutes. This is likely to occur following an add or repair server scenario.
Use the Start-ClusterGroup and Stop-ClusterGroup cmdlets to restart the update service.
A successful run of these cmdlets should bring the update service online.
Cluster aware updating
Resume node operation failed to resume node.
This is a transient issue and could resolve on its own. Wait for a few minutes and retry the operation. If the issue persists, contact Microsoft Support.
Cluster aware updating
Suspend node operation was stuck for greater than 90 minutes.
This is a transient issue and could resolve on its own. Wait for a few minutes and retry the operation. If the issue persists, contact Microsoft Support.
Known issues for version 2411.2
This software release maps to software version number 2411.2.12.
Importante
The new deployments of this software use the 2411.2.12 build. You can also update from 2411.0 and 2411.1.
Release notes for this version include the issues fixed in this release, known issues in this release, and release note issues carried over from previous versions.
The storage path couldn't be deleted with a pre-downloaded AKS required image.
Arc VM Management
Image deletion retry fails after the node restarts.
When the node goes down and if you try deleting an image, the deletion times out. When the node restarts and retries deletion, the deletion fails again.
Update
A solution extension package was unintentionally applied into a solution update.
Known issues in this release
The following table lists the known issues in this release:
Feature
Issue
Workaround
Update
When monitoring update progress in the Azure Update Management portal, the progress might appear to not have updated for several hours.
Run Get-SolutionUpdate on one of the cluster nodes. If an update object is returned, the update might be taking longer than expected but it is progressing. If an update object is not returned, the update may be stalled. For detailed steps on how to resolve this issue, see the Troubleshooting guide.
Known issues from previous releases
The following table lists the known issues from previous releases:
Feature
Issue
Workaround
Deployment
Validation times out due to timestamp deserialization.
When deploying the operating system, select English (United States) as the installation language, as well as the time and currency format. For detailed remediation steps, see the troubleshooting guide in the Azure Local Supportability GitHub repository.
Update
When updating from version 2408.2.7 to 2411.0.24, the update process could fail with the following error message: Type 'CauPreRequisites' of Role 'CAU' raised an exception: Could not finish cau prerequisites due to error 'Cannot remove item C:\UpdateDistribution\<any_file_name>: Access to the path is denied.'
With the 2411 release, solution and Solution Builder Extension update aren't combined in a single update run.
To apply a Solution Builder Extension package, you need a separate update run.
Update
When applying solution update in this release, the update can fail. This will occur only if the update was started prior to November 26. The issue that causes the failure can result in one of the following error messages:
Error 1 - The step "update ARB and extension" error "Clear-AzContext failed with 0 and Exception calling "Initialize" with "1" argument(s): "Object reference not set to an instance of an object." at "Clear-AzPowerShellCache".
Error 2 - The step "EvalTVMFlow" error "CloudEngine.Actions.InterfaceInvocationFailedException: Type 'EvalTVMFlow' of Role 'ArcIntegration' raised an exception: This module requires Az.Accounts version 3.0.5. An earlier version of Az.Accounts is imported in the current PowerShell session. Please open a new session before importing this module. This error could indicate that multiple incompatible versions of the Azure PowerShell cmdlets are installed on your system. Please see https://aka.ms/azps-version-error for troubleshooting information."
Depending on the version of PowerShell modules, the above error could be reported for both versions 3.0.4 and 3.0.5.
After you repair a node and run the command Set-AzureStackLCMUserPassword, you may encounter the following error:
CloudEngine.Actions.InterfaceInvocationFailedException: Type 'ValidateCredentials' of Role 'SecretRotation' raised an exception: Cannot load encryption certificate. The certificate setting 'CN=DscEncryptionCert' does not represent a valid base-64 encoded certificate, nor does it represent a valid certificate by file, directory, thumbprint, or subject name. at Validate-Credentials
Follow these steps to mitigate the issue:
$NewPassword = <Provide new password as secure string>
$OldPassword = <Provide the old/current password as secure string>
if ($eceClusterGroup.State -ne "Online") {Write-AzsSecurityError -Message "ECE cluster group is not in an Online state. Cannot continue with password rotation." -ErrRecord $_}
3. Update the ECE with the new password:
Write-AzsSecurityVerbose -Message "Updating password in ECE" -Verbose
Using an exported Azure VM OS disk as a VHD to create a gallery image for provisioning an Arc VM is unsupported.
Run the command restart-service mochostagent to restart the mochostagent service.
Networking
When a node is configured with a proxy server that has capital letters in its address, such as HTTPS://10.100.000.00:8080, Arc extensions fail to install or update on the node in existing builds, including version 2408.1. However, the node remains Arc connected.
Follow these steps to mitigate the issue:
1. Set the environment values in lowercase. [System.Environment]::SetEnvironmentVariable("HTTPS_PROXY", "https://10.100.000.00:8080", "Machine").
2. Validate that the values were set. [System.Environment]::GetEnvironmentVariable("HTTPS_PROXY", "Machine").
3. Restart Arc services.
Restart-Service himds
Restart-Service ExtensionService
Restart-Service GCArcService
4. Signal the AzcmaAgent with the lowercase proxy information.
& 'C:\Program Files\AzureConnectedMachineAgent\azcmagent.exe' config set proxy.url https://10.100.000.00:8080
& 'C:\Program Files\AzureConnectedMachineAgent\azcmagent.exe' config list
Networking
When Arc machines go down, the "All Clusters" page, in the new portal experience shows a "PartiallyConnected" or "Not Connected Recently status. Even when the Arc machines become healthy, they may not show a "Connected" status.
There's no known workaround for this issue. To check the connectivity status, use the old experience to see if it shows as "Connected".
Arc VM management
The Mochostagent service might appear to be running but can get stuck without updating logs for over a month. You can identify this issue by checking the service logs in C:\programdata\mochostagent\logs to see if logs are being updated.
Run the following command to restart the mochostagent service: restart-service mochostagent.
Upgrade
When upgrading the stamp from 2311 or prior builds to 2408 or later, add node and repair node operations may fail. For example, you could see an error: Type 'AddAsZHostToDomain' of Role 'BareMetal' raised an exception.
There's no workaround in this release. If you encounter this issue, contact Microsoft Support to determine next steps.
Update
When viewing the readiness check results for an Azure Local instance via the Azure Update Manager, there might be multiple readiness checks with the same name.
There's no known workaround in this release. Select View details to view specific information about the readiness check.
Update
There's an intermittent issue in this release when the Azure portal incorrectly reports the update status as Failed to update or In progress though the update is complete.
$Update = get-solutionupdate| ? version -eq "<version string>"
Replace the version string with the version you're running. For example, "10.2405.0.23".
$Update.state
If the update status is Installed, no further action is required on your part. Azure portal refreshes the status correctly within 24 hours. To refresh the status sooner, follow these steps on one of the nodes. Restart the Cloud Management cluster group. Stop-ClusterGroup "Cloud Management" Start-ClusterGroup "Cloud Management"
Update
During an initial MOC update, a failure occurs due to the target MOC version not being found in the catalog cache. The follow-up updates and retries show MOC in the target version, without the update succeeding, and as a result the Arc Resource Bridge update fails.
To validate this issue, collect the update logs using Troubleshoot solution updates for Azure Local. The log files should show a similar error message (current version might differ in the error message):
[ERROR: { "errorCode": "InvalidEntityError", "errorResponse": "{\n\"message\": \"the cloud fabric (MOC) is currently at version v0.13.1. A minimum version of 0.15.0 is required for compatibility\"\n}" }]
Follow these steps to mitigate the issue:
1. To find the MOC agent version, run the following command: 'C:\Program Files\AksHci\wssdcloudagent.exe' version.
2. Use the output of the command to find the MOC version from the table below that matches the agent version, and set $initialMocVersion to that MOC version. Set the $targetMocVersion by finding the Azure Local build you're updating to and get the matching MOC version from the following table. Use these values in the mitigation script provided below:
Build
MOC version
Agent version
2311.2
1.0.24.10106
v0.13.0-6-gf13a73f7, v0.11.0-alpha.38,01/06/2024
2402
1.0.25.10203
v0.14.0, v0.13.1, 02/02/2024
2402.1
1.0.25.10302
v0.14.0, v0.13.1, 03/02/2024
2402.2
1.1.1.10314
v0.16.0-1-g04bf0dec, v0.15.1, 03/14/2024
2405/2402.3
1.3.0.10418
v0.17.1, v0.16.5, 04/18/2024
For example, if the agent version is v0.13.0-6-gf13a73f7, v0.11.0-alpha.38,01/06/2024, then $initialMocVersion = "1.0.24.10106" and if you're updating to 2405.0.23, then $targetMocVersion = "1.3.0.10418".
3. Run the following PowerShell commands on the first node:
$initialMocVersion = "<initial version determined from step 2>" $targetMocVersion = "<target version determined from step 2>"
# Import MOC module twice import-module moc import-module moc $verbosePreference = "Continue"
# Clear the SFS catalog cache Remove-Item (Get-MocConfig).manifestCache
# Set version to the current MOC version prior to update, and set state as update failed Set-MocConfigValue -name "version" -value $initialMocVersion Set-MocConfigValue -name "installState" -value ([InstallState]::UpdateFailed)
# Rerun the MOC update to desired version Update-Moc -version $targetMocVersion
4. Resume the update.
Deployment
In some instances, during the registration of Azure Local machines, this error might be seen in the debug logs: Encountered internal server error. One of the mandatory extensions for device deployment might not be installed.
AKS cluster creation fails with the Error: Invalid AKS network resource id. This issue can occur when the associated logical network name has an underscore.
Underscores aren't supported in logical network names. Make sure to not use underscore in the names for logical networks deployed on your Azure Local.
Repair server
In rare instances, the Repair-Server operation fails with the HealthServiceWaitForDriveFW error. In these cases, the old drives from the repaired node aren't removed and new disks are stuck in the maintenance mode.
To prevent this issue, make sure that you DO NOT drain the node either via the Windows Admin Center or using the Suspend-ClusterNode -Drain PowerShell cmdlet before you start Repair-Server. If the issue occurs, contact Microsoft Support for next steps.
Repair server
This issue is seen when the single node Azure Local instance is updated from 2311 to 2402 and then the Repair-Server is performed. The repair operation fails.
Before you repair the single node, follow these steps: 1. Run version 2402 for the ADPrepTool. Follow the steps in Prepare Active Directory. This action is quick and adds the required permissions to the Organizational Unit (OU). 2. Move the computer object from Computers segment to the root OU. Run the following command: Get-ADComputer <HOSTNAME> | Move-ADObject -TargetPath "<OU path>"
Deployment
If you prepare the Active Directory on your own (not using the script and procedure provided by Microsoft), your Active Directory validation could fail with missing Generic All permission. This is due to an issue in the validation check that checks for a dedicated permission entry for msFVE-RecoverInformationobjects – General – Permissions Full control, which is required for BitLocker recovery.
Use the Prepare AD script method or if using your own method, make sure to assign the specific permission msFVE-RecoverInformationobjects – General – Permissions Full control.
Deployment
There's a rare issue in this release where the DNS record is deleted during the Azure Local deployment. When that occurs, the following exception is seen: Type 'PropagatePublicRootCertificate' of Role 'ASCA' raised an exception:<br>The operation on computer 'ASB88RQ22U09' failed: WinRM cannot process the request. The following error occurred while using Kerberos authentication: Cannot find the computer ASB88RQ22U09.local. Verify that the computer exists on the network and that the name provided is spelled correctly at PropagatePublicRootCertificate, C:\NugetStore\Microsoft.AzureStack, at Orchestration.Roles.CertificateAuthority.10.2402.0.14\content\Classes\ASCA\ASCA.psm1: line 38, at C:\CloudDeployment\ECEngine\InvokeInterfaceInternal.psm1: line 127,at Invoke-EceInterfaceInternal, C:\CloudDeployment\ECEngine\InvokeInterfaceInternal.psm1: line 123.
Check the DNS server to see if any DNS records of the nodes are missing. Apply the following mitigation on the nodes where its DNS record is missing.
Restart the DNS client service. Open a PowerShell session and run the following cmdlet on the affected node: Taskkill /f /fi "SERVICES eq dnscache"
Deployment
In this release, there's a remote task failure on a multi-node deployment that results in the following exception: ECE RemoteTask orchestration failure with ASRR1N42R01U31 (node pingable - True): A WebException occurred while sending a RestRequest. WebException.Status: ConnectFailure on [https://<URL>](https://<URL>).
The mitigation is to restart the ECE agent on the affected node. On your machine, open a PowerShell session and run the following command: Restart-Service ECEAgent.
Add server
In this release and previous releases, when adding a machine to the system, isn't possible to update the proxy bypass list string to include the new machine. Updating environment variables proxy bypass list on the hosts won't update the proxy bypass list on Azure Resource Bridge or AKS.
There's no workaround in this release. If you encounter this issue, contact Microsoft Support to determine next steps.
Add/Repair server
In this release, when adding or repairing a machine, a failure is seen when the software load balancer or network controller VM certificates are being copied from the existing nodes. The failure is because these certificates weren't generated during the deployment/update.
There's no workaround in this release. If you encounter this issue, contact Microsoft Support to determine next steps.
Deployment
In this release, there's a transient issue resulting in the deployment failure with the following exception: Type 'SyncDiagnosticLevel' of Role 'ObservabilityConfig' raised an exception:*<br>*Syncing Diagnostic Level failed with error: The Diagnostic Level does not match. Portal was not set to Enhanced, instead is Basic.
As this is a transient issue, retrying the deployment should fix this. For more information, see how to Rerun the deployment.
Deployment
In this release, there's an issue with the Secrets URI/location field. This is a required field that is marked Not mandatory and results in Azure Resource Manager template deployment failures.
Use the sample parameters file in the Deploy Azure Local via Azure Resource Manager template to ensure that all the inputs are provided in the required format and then try the deployment. If there's a failed deployment, you must also clean up the following resources before you Rerun the deployment: 1. Delete C:\EceStore. 2. Delete C:\CloudDeployment. 3. Delete C:\nugetstore. 4. Remove-Item HKLM:\Software\Microsoft\LCMAzureStackStampInformation.
Security
For new deployments, Secured-core capable devices won't have Dynamic Root of Measurement (DRTM) enabled by default. If you try to enable (DRTM) using the Enable-AzSSecurity cmdlet, you see an error that DRTM setting isn't supported in the current release. Microsoft recommends defense in depth, and UEFI Secure Boot still protects the components in the Static Root of Trust (SRT) boot chain by ensuring that they're loaded only when they're signed and verified.
DRTM isn't supported in this release.
Networking
An environment check fails when a proxy server is used. By design, the bypass list is different for winhttp and wininet, which causes the validation check to fail.
Follow these workaround steps:
1. Clear the proxy bypass list prior to the health check and before starting the deployment or the update.
2. After passing the check, wait for the deployment or update to fail.
3. Set your proxy bypass list again.
Arc VM management
Deployment or update of Arc Resource Bridge could fail when the automatically generated temporary SPN secret during this operation, starts with a hyphen.
Retry the deployment/update. The retry should regenerate the SPN secret and the operation will likely succeed.
Arc VM management
Arc Extensions on Arc VMs stay in "Creating" state indefinitely.
Sign in to the VM, open a command prompt, and type the following: Windows: notepad C:\ProgramData\AzureConnectedMachineAgent\Config\agentconfig.json Linux: sudo vi /var/opt/azcmagent/agentconfig.json Next, find the resourcename property. Delete the GUID that is appended to the end of the resource name, so this property matches the name of the VM. Then restart the VM.
Arc VM management
When a new machine is added to an Azure Local instance, storage path isn't created automatically for the newly created volume.
You can manually create a storage path for any new volumes. For more information, see Create a storage path.
Arc VM management
Restart of Arc VM operation completes after approximately 20 minutes although the VM itself restarts in about a minute.
There's no known workaround in this release.
Arc VM management
In some instances, the status of the logical network shows as Failed in Azure portal. This occurs when you try to delete the logical network without first deleting any resources such as network interfaces associated with that logical network. You should still be able to create resources on this logical network. The status is misleading in this instance.
If the status of this logical network was Succeeded at the time when this network was provisioned, then you can continue to create resources on this network.
Arc VM management
In this release, when you update a VM with a data disk attached to it using the Azure CLI, the operation fails with the following error message: Couldn't find a virtual hard disk with the name.
In rare instances, you may encounter this error while updating your Azure Local instance: Type 'UpdateArbAndExtensions' of Role 'MocArb' raised an exception: Exception Upgrading ARB and Extension in step [UpgradeArbAndExtensions :Get-ArcHciConfig] UpgradeArb: Invalid applianceyaml = [C:\AksHci\hci-appliance.yaml].
If you see this issue, contact Microsoft Support to assist you with the next steps.
Networking
There's an infrequent DNS client issue in this release that causes the deployment to fail on a two-node system with a DNS resolution error: A WebException occurred while sending a RestRequest. WebException.Status: NameResolutionFailure. As a result of the bug, the DNS record of the second node is deleted soon after it's created resulting in a DNS error.
Restart the machine. This operation registers the DNS record, which prevents it from getting deleted.
Azure portal
In some instances, the Azure portal might take a while to update and the view might not be current.
You might need to wait for 30 minutes or more to see the updated view.
Arc VM management
Deleting a network interface on an Arc VM from Azure portal doesn't work in this release.
Providing the OU name in an incorrect syntax isn't detected in the Azure portal. The incorrect syntax includes unsupported characters such as &,",',<,>. The incorrect syntax is detected at a later step during system validation.
Make sure that the OU path syntax is correct and doesn't include unsupported characters.
Deployment
Deployments via Azure Resource Manager time out after 2 hours. Deployments that exceed 2 hours show up as failed in the resource group though the system is successfully created.
To monitor the deployment in the Azure portal, go to the Azure Local instance resource and then go to new Deployments entry.
Azure Site Recovery
Azure Site Recovery can't be installed on an Azure Local instance in this release.
There's no known workaround in this release.
Update
When updating the Azure Local instance via the Azure Update Manager, the update progress and results may not be visible in the Azure portal.
To work around this issue, on each node, add the following registry key (no value needed):
Then on one of the nodes, restart the Cloud Management cluster group.
Stop-ClusterGroup "Cloud Management"
Start-ClusterGroup "Cloud Management"
This won't fully remediate the issue as the progress details may still not be displayed for a duration of the update process. To get the latest update details, you can Retrieve the update progress with PowerShell.
Update
In rare instances, if a failed update is stuck in an In progress state in Azure Update Manager, the Try again button is disabled.
To resume the update, run the following PowerShell command: Get-SolutionUpdate|Start-SolutionUpdate.
Update
In some cases, SolutionUpdate commands could fail if run after the Send-DiagnosticData command.
Make sure to close the PowerShell session used for Send-DiagnosticData. Open a new PowerShell session and use it for SolutionUpdate commands.
Update
In rare instances, when applying an update from 2311.0.24 to 2311.2.4, system status reports In Progress instead of expected Failed to update.
Retry the update. If the issue persists, contact Microsoft Support.
Update
Attempts to install solution updates can fail at the end of the CAU steps with: There was a failure in a Common Information Model (CIM) operation, that is, an operation performed by software that Cluster-Aware Updating depends on. This rare issue occurs if the Cluster Name or Cluster IP Address resources fail to start after a node reboot and is most typical in small deployments.
If you encounter this issue, contact Microsoft Support for next steps. They can work with you to manually restart the Azure Local resources and resume the update as needed.
Update
When applying a system update to 10.2402.3.11, the Get-SolutionUpdate cmdlet may not respond and eventually fails with a RequestTimeoutException after approximately 10 minutes. This is likely to occur following an add or repair server scenario.
Use the Start-ClusterGroup and Stop-ClusterGroup cmdlets to restart the update service.
A successful run of these cmdlets should bring the update service online.
Cluster aware updating
Resume node operation failed to resume node.
This is a transient issue and could resolve on its own. Wait for a few minutes and retry the operation. If the issue persists, contact Microsoft Support.
Cluster aware updating
Suspend node operation was stuck for greater than 90 minutes.
This is a transient issue and could resolve on its own. Wait for a few minutes and retry the operation. If the issue persists, contact Microsoft Support.
Known issues for version 2411.1
This software release maps to software version number 2411.1.10.
Importante
The new deployments of this software use the 2411.1.10 build. If you updated from 2408.2, you’ve received either the 2411.0.22 or 2411.0.24 build. Both builds can be updated to 2411.1.10.
Release notes for this version include the issues fixed in this release, known issues in this release, and release note issues carried over from previous versions.
Redeploying an Arc VM causes connection issues with that Arc VM and the agent disconnects.
Upgrade
Resolved conflict with third party PowerShell modules.
Upgrade
Stopped indefinite logging of negligible error events.
Upgrade
Added validation to check for free memory.
Update
Added check to ensure that solution extension content has been copied correctly.
Deployment Upgrade
If the timezone isn't set to UTC before you deploy Azure Local, an ArcOperationTimeOut error occurs during validation. The following error message is displayed: *OperationTimeOut, No updates received from device for operation.
Security vulnerability
Microsoft identified a security vulnerability that could expose the local admin credentials used during the creation of Arc VMs on Azure Local to non-admin users on the VM and on the hosts. Arc VMs running on releases prior to Azure Local 2411 release are vulnerable.
Known issues in this release
The following table lists the known issues in this release:
Feature
Issue
Workaround
Deployment
Validation times out due to timestamp deserialization.
When deploying the operating system, select English (United States) as the installation language, as well as the time and currency format. For detailed remediation steps, see the troubleshooting guide in the Azure Local Supportability GitHub repository.
Known issues from previous releases
The following table lists the known issues from previous releases:
Feature
Issue
Workaround
Update
When updating from version 2408.2.7 to 2411.0.24, the update process could fail with the following error message: Type 'CauPreRequisites' of Role 'CAU' raised an exception: Could not finish cau prerequisites due to error 'Cannot remove item C:\UpdateDistribution\<any_file_name>: Access to the path is denied.'
With the 2411 release, solution and Solution Builder Extension update aren't combined in a single update run.
To apply a Solution Builder Extension package, you need a separate update run.
Update
When applying solution update in this release, the update can fail. This will occur only if the update was started prior to November 26. The issue that causes the failure can result in one of the following error messages:
Error 1 - The step "update ARB and extension" error "Clear-AzContext failed with 0 and Exception calling "Initialize" with "1" argument(s): "Object reference not set to an instance of an object." at "Clear-AzPowerShellCache".
Error 2 - The step "EvalTVMFlow" error "CloudEngine.Actions.InterfaceInvocationFailedException: Type 'EvalTVMFlow' of Role 'ArcIntegration' raised an exception: This module requires Az.Accounts version 3.0.5. An earlier version of Az.Accounts is imported in the current PowerShell session. Please open a new session before importing this module. This error could indicate that multiple incompatible versions of the Azure PowerShell cmdlets are installed on your system. Please see https://aka.ms/azps-version-error for troubleshooting information."
Depending on the version of PowerShell modules, the above error could be reported for both versions 3.0.4 and 3.0.5.
After you repair a node and run the command Set-AzureStackLCMUserPassword, you may encounter the following error:
CloudEngine.Actions.InterfaceInvocationFailedException: Type 'ValidateCredentials' of Role 'SecretRotation' raised an exception: Cannot load encryption certificate. The certificate setting 'CN=DscEncryptionCert' does not represent a valid base-64 encoded certificate, nor does it represent a valid certificate by file, directory, thumbprint, or subject name. at Validate-Credentials
Follow these steps to mitigate the issue:
$NewPassword = <Provide new password as secure string>
$OldPassword = <Provide the old/current password as secure string>
if ($eceClusterGroup.State -ne "Online") {Write-AzsSecurityError -Message "ECE cluster group is not in an Online state. Cannot continue with password rotation." -ErrRecord $_}
3. Update the ECE with the new password:
Write-AzsSecurityVerbose -Message "Updating password in ECE" -Verbose
Using an exported Azure VM OS disk as a VHD to create a gallery image for provisioning an Arc VM is unsupported.
Run the command restart-service mochostagent to restart the mochostagent service.
Networking
When a node is configured with a proxy server that has capital letters in its address, such as HTTPS://10.100.000.00:8080, Arc extensions fail to install or update on the node in existing builds, including version 2408.1. However, the node remains Arc connected.
Follow these steps to mitigate the issue:
1. Set the environment values in lowercase. [System.Environment]::SetEnvironmentVariable("HTTPS_PROXY", "https://10.100.000.00:8080", "Machine").
2. Validate that the values were set. [System.Environment]::GetEnvironmentVariable("HTTPS_PROXY", "Machine").
3. Restart Arc services.
Restart-Service himds
Restart-Service ExtensionService
Restart-Service GCArcService
4. Signal the AzcmaAgent with the lowercase proxy information.
& 'C:\Program Files\AzureConnectedMachineAgent\azcmagent.exe' config set proxy.url https://10.100.000.00:8080
& 'C:\Program Files\AzureConnectedMachineAgent\azcmagent.exe' config list
Networking
When Arc machines go down, the "All Clusters" page, in the new portal experience shows a "PartiallyConnected" or "Not Connected Recently status. Even when the Arc machines become healthy, they may not show a "Connected" status.
There's no known workaround for this issue. To check the connectivity status, use the old experience to see if it shows as "Connected".
Arc VM management
The Mochostagent service might appear to be running but can get stuck without updating logs for over a month. You can identify this issue by checking the service logs in C:\programdata\mochostagent\logs to see if logs are being updated.
Run the following command to restart the mochostagent service: restart-service mochostagent.
Upgrade
When upgrading the stamp from 2311 or prior builds to 2408 or later, add node and repair node operations may fail. For example, you could see an error: Type 'AddAsZHostToDomain' of Role 'BareMetal' raised an exception.
There's no workaround in this release. If you encounter this issue, contact Microsoft Support to determine next steps.
Update
When viewing the readiness check results for an Azure Local instance via the Azure Update Manager, there might be multiple readiness checks with the same name.
There's no known workaround in this release. Select View details to view specific information about the readiness check.
Update
There's an intermittent issue in this release when the Azure portal incorrectly reports the update status as Failed to update or In progress though the update is complete.
$Update = get-solutionupdate| ? version -eq "<version string>"
Replace the version string with the version you're running. For example, "10.2405.0.23".
$Update.state
If the update status is Installed, no further action is required on your part. Azure portal refreshes the status correctly within 24 hours. To refresh the status sooner, follow these steps on one of the nodes. Restart the Cloud Management cluster group. Stop-ClusterGroup "Cloud Management" Start-ClusterGroup "Cloud Management"
Update
During an initial MOC update, a failure occurs due to the target MOC version not being found in the catalog cache. The follow-up updates and retries show MOC in the target version, without the update succeeding, and as a result the Arc Resource Bridge update fails.
To validate this issue, collect the update logs using Troubleshoot solution updates for Azure Local. The log files should show a similar error message (current version might differ in the error message):
[ERROR: { "errorCode": "InvalidEntityError", "errorResponse": "{\n\"message\": \"the cloud fabric (MOC) is currently at version v0.13.1. A minimum version of 0.15.0 is required for compatibility\"\n}" }]
Follow these steps to mitigate the issue:
1. To find the MOC agent version, run the following command: 'C:\Program Files\AksHci\wssdcloudagent.exe' version.
2. Use the output of the command to find the MOC version from the table below that matches the agent version, and set $initialMocVersion to that MOC version. Set the $targetMocVersion by finding the Azure Local build you're updating to and get the matching MOC version from the following table. Use these values in the mitigation script provided below:
Build
MOC version
Agent version
2311.2
1.0.24.10106
v0.13.0-6-gf13a73f7, v0.11.0-alpha.38,01/06/2024
2402
1.0.25.10203
v0.14.0, v0.13.1, 02/02/2024
2402.1
1.0.25.10302
v0.14.0, v0.13.1, 03/02/2024
2402.2
1.1.1.10314
v0.16.0-1-g04bf0dec, v0.15.1, 03/14/2024
2405/2402.3
1.3.0.10418
v0.17.1, v0.16.5, 04/18/2024
For example, if the agent version is v0.13.0-6-gf13a73f7, v0.11.0-alpha.38,01/06/2024, then $initialMocVersion = "1.0.24.10106" and if you're updating to 2405.0.23, then $targetMocVersion = "1.3.0.10418".
3. Run the following PowerShell commands on the first node:
$initialMocVersion = "<initial version determined from step 2>" $targetMocVersion = "<target version determined from step 2>"
# Import MOC module twice import-module moc import-module moc $verbosePreference = "Continue"
# Clear the SFS catalog cache Remove-Item (Get-MocConfig).manifestCache
# Set version to the current MOC version prior to update, and set state as update failed Set-MocConfigValue -name "version" -value $initialMocVersion Set-MocConfigValue -name "installState" -value ([InstallState]::UpdateFailed)
# Rerun the MOC update to desired version Update-Moc -version $targetMocVersion
4. Resume the update.
Deployment
In some instances, during the registration of Azure Local machines, this error might be seen in the debug logs: Encountered internal server error. One of the mandatory extensions for device deployment might not be installed.
AKS cluster creation fails with the Error: Invalid AKS network resource id. This issue can occur when the associated logical network name has an underscore.
Underscores aren't supported in logical network names. Make sure to not use underscore in the names for logical networks deployed on your Azure Local.
Repair server
In rare instances, the Repair-Server operation fails with the HealthServiceWaitForDriveFW error. In these cases, the old drives from the repaired node aren't removed and new disks are stuck in the maintenance mode.
To prevent this issue, make sure that you DO NOT drain the node either via the Windows Admin Center or using the Suspend-ClusterNode -Drain PowerShell cmdlet before you start Repair-Server. If the issue occurs, contact Microsoft Support for next steps.
Repair server
This issue is seen when the single node Azure Local instance is updated from 2311 to 2402 and then the Repair-Server is performed. The repair operation fails.
Before you repair the single node, follow these steps: 1. Run version 2402 for the ADPrepTool. Follow the steps in Prepare Active Directory. This action is quick and adds the required permissions to the Organizational Unit (OU). 2. Move the computer object from Computers segment to the root OU. Run the following command: Get-ADComputer <HOSTNAME> | Move-ADObject -TargetPath "<OU path>"
Deployment
If you prepare the Active Directory on your own (not using the script and procedure provided by Microsoft), your Active Directory validation could fail with missing Generic All permission. This is due to an issue in the validation check that checks for a dedicated permission entry for msFVE-RecoverInformationobjects – General – Permissions Full control, which is required for BitLocker recovery.
Use the Prepare AD script method or if using your own method, make sure to assign the specific permission msFVE-RecoverInformationobjects – General – Permissions Full control.
Deployment
There's a rare issue in this release where the DNS record is deleted during the Azure Local deployment. When that occurs, the following exception is seen: Type 'PropagatePublicRootCertificate' of Role 'ASCA' raised an exception:<br>The operation on computer 'ASB88RQ22U09' failed: WinRM cannot process the request. The following error occurred while using Kerberos authentication: Cannot find the computer ASB88RQ22U09.local. Verify that the computer exists on the network and that the name provided is spelled correctly at PropagatePublicRootCertificate, C:\NugetStore\Microsoft.AzureStack, at Orchestration.Roles.CertificateAuthority.10.2402.0.14\content\Classes\ASCA\ASCA.psm1: line 38, at C:\CloudDeployment\ECEngine\InvokeInterfaceInternal.psm1: line 127,at Invoke-EceInterfaceInternal, C:\CloudDeployment\ECEngine\InvokeInterfaceInternal.psm1: line 123.
Check the DNS server to see if any DNS records of the nodes are missing. Apply the following mitigation on the nodes where its DNS record is missing.
Restart the DNS client service. Open a PowerShell session and run the following cmdlet on the affected node: Taskkill /f /fi "SERVICES eq dnscache"
Deployment
In this release, there's a remote task failure on a multi-node deployment that results in the following exception: ECE RemoteTask orchestration failure with ASRR1N42R01U31 (node pingable - True): A WebException occurred while sending a RestRequest. WebException.Status: ConnectFailure on [https://<URL>](https://<URL>).
The mitigation is to restart the ECE agent on the affected node. On your machine, open a PowerShell session and run the following command: Restart-Service ECEAgent.
Add server
In this release and previous releases, when adding a machine to the system, isn't possible to update the proxy bypass list string to include the new machine. Updating environment variables proxy bypass list on the hosts won't update the proxy bypass list on Azure Resource Bridge or AKS.
There's no workaround in this release. If you encounter this issue, contact Microsoft Support to determine next steps.
Add/Repair server
In this release, when adding or repairing a machine, a failure is seen when the software load balancer or network controller VM certificates are being copied from the existing nodes. The failure is because these certificates weren't generated during the deployment/update.
There's no workaround in this release. If you encounter this issue, contact Microsoft Support to determine next steps.
Deployment
In this release, there's a transient issue resulting in the deployment failure with the following exception: Type 'SyncDiagnosticLevel' of Role 'ObservabilityConfig' raised an exception:*<br>*Syncing Diagnostic Level failed with error: The Diagnostic Level does not match. Portal was not set to Enhanced, instead is Basic.
As this is a transient issue, retrying the deployment should fix this. For more information, see how to Rerun the deployment.
Deployment
In this release, there's an issue with the Secrets URI/location field. This is a required field that is marked Not mandatory and results in Azure Resource Manager template deployment failures.
Use the sample parameters file in the Deploy Azure Local via Azure Resource Manager template to ensure that all the inputs are provided in the required format and then try the deployment. If there's a failed deployment, you must also clean up the following resources before you Rerun the deployment: 1. Delete C:\EceStore. 2. Delete C:\CloudDeployment. 3. Delete C:\nugetstore. 4. Remove-Item HKLM:\Software\Microsoft\LCMAzureStackStampInformation.
Security
For new deployments, Secured-core capable devices won't have Dynamic Root of Measurement (DRTM) enabled by default. If you try to enable (DRTM) using the Enable-AzSSecurity cmdlet, you see an error that DRTM setting isn't supported in the current release. Microsoft recommends defense in depth, and UEFI Secure Boot still protects the components in the Static Root of Trust (SRT) boot chain by ensuring that they're loaded only when they're signed and verified.
DRTM isn't supported in this release.
Networking
An environment check fails when a proxy server is used. By design, the bypass list is different for winhttp and wininet, which causes the validation check to fail.
Follow these workaround steps:
1. Clear the proxy bypass list prior to the health check and before starting the deployment or the update.
2. After passing the check, wait for the deployment or update to fail.
3. Set your proxy bypass list again.
Arc VM management
Deployment or update of Arc Resource Bridge could fail when the automatically generated temporary SPN secret during this operation, starts with a hyphen.
Retry the deployment/update. The retry should regenerate the SPN secret and the operation will likely succeed.
Arc VM management
Arc Extensions on Arc VMs stay in "Creating" state indefinitely.
Sign in to the VM, open a command prompt, and type the following: Windows: notepad C:\ProgramData\AzureConnectedMachineAgent\Config\agentconfig.json Linux: sudo vi /var/opt/azcmagent/agentconfig.json Next, find the resourcename property. Delete the GUID that is appended to the end of the resource name, so this property matches the name of the VM. Then restart the VM.
Arc VM management
When a new machine is added to an Azure Local instance, storage path isn't created automatically for the newly created volume.
You can manually create a storage path for any new volumes. For more information, see Create a storage path.
Arc VM management
Restart of Arc VM operation completes after approximately 20 minutes although the VM itself restarts in about a minute.
There's no known workaround in this release.
Arc VM management
In some instances, the status of the logical network shows as Failed in Azure portal. This occurs when you try to delete the logical network without first deleting any resources such as network interfaces associated with that logical network. You should still be able to create resources on this logical network. The status is misleading in this instance.
If the status of this logical network was Succeeded at the time when this network was provisioned, then you can continue to create resources on this network.
Arc VM management
In this release, when you update a VM with a data disk attached to it using the Azure CLI, the operation fails with the following error message: Couldn't find a virtual hard disk with the name.
In rare instances, you may encounter this error while updating your Azure Local instance: Type 'UpdateArbAndExtensions' of Role 'MocArb' raised an exception: Exception Upgrading ARB and Extension in step [UpgradeArbAndExtensions :Get-ArcHciConfig] UpgradeArb: Invalid applianceyaml = [C:\AksHci\hci-appliance.yaml].
If you see this issue, contact Microsoft Support to assist you with the next steps.
Networking
There's an infrequent DNS client issue in this release that causes the deployment to fail on a two-node system with a DNS resolution error: A WebException occurred while sending a RestRequest. WebException.Status: NameResolutionFailure. As a result of the bug, the DNS record of the second node is deleted soon after it's created resulting in a DNS error.
Restart the machine. This operation registers the DNS record, which prevents it from getting deleted.
Azure portal
In some instances, the Azure portal might take a while to update and the view might not be current.
You might need to wait for 30 minutes or more to see the updated view.
Arc VM management
Deleting a network interface on an Arc VM from Azure portal doesn't work in this release.
Providing the OU name in an incorrect syntax isn't detected in the Azure portal. The incorrect syntax includes unsupported characters such as &,",',<,>. The incorrect syntax is detected at a later step during system validation.
Make sure that the OU path syntax is correct and doesn't include unsupported characters.
Deployment
Deployments via Azure Resource Manager time out after 2 hours. Deployments that exceed 2 hours show up as failed in the resource group though the system is successfully created.
To monitor the deployment in the Azure portal, go to the Azure Local instance resource and then go to new Deployments entry.
Azure Site Recovery
Azure Site Recovery can't be installed on an Azure Local instance in this release.
There's no known workaround in this release.
Update
When updating the Azure Local instance via the Azure Update Manager, the update progress and results may not be visible in the Azure portal.
To work around this issue, on each node, add the following registry key (no value needed):
Then on one of the nodes, restart the Cloud Management cluster group.
Stop-ClusterGroup "Cloud Management"
Start-ClusterGroup "Cloud Management"
This won't fully remediate the issue as the progress details may still not be displayed for a duration of the update process. To get the latest update details, you can Retrieve the update progress with PowerShell.
Update
In rare instances, if a failed update is stuck in an In progress state in Azure Update Manager, the Try again button is disabled.
To resume the update, run the following PowerShell command: Get-SolutionUpdate|Start-SolutionUpdate.
Update
In some cases, SolutionUpdate commands could fail if run after the Send-DiagnosticData command.
Make sure to close the PowerShell session used for Send-DiagnosticData. Open a new PowerShell session and use it for SolutionUpdate commands.
Update
In rare instances, when applying an update from 2311.0.24 to 2311.2.4, system status reports In Progress instead of expected Failed to update.
Retry the update. If the issue persists, contact Microsoft Support.
Update
Attempts to install solution updates can fail at the end of the CAU steps with: There was a failure in a Common Information Model (CIM) operation, that is, an operation performed by software that Cluster-Aware Updating depends on. This rare issue occurs if the Cluster Name or Cluster IP Address resources fail to start after a node reboot and is most typical in small deployments.
If you encounter this issue, contact Microsoft Support for next steps. They can work with you to manually restart the Azure Local resources and resume the update as needed.
Update
When applying a system update to 10.2402.3.11, the Get-SolutionUpdate cmdlet may not respond and eventually fails with a RequestTimeoutException after approximately 10 minutes. This is likely to occur following an add or repair server scenario.
Use the Start-ClusterGroup and Stop-ClusterGroup cmdlets to restart the update service.
A successful run of these cmdlets should bring the update service online.
Cluster aware updating
Resume node operation failed to resume node.
This is a transient issue and could resolve on its own. Wait for a few minutes and retry the operation. If the issue persists, contact Microsoft Support.
Cluster aware updating
Suspend node operation was stuck for greater than 90 minutes.
This is a transient issue and could resolve on its own. Wait for a few minutes and retry the operation. If the issue persists, contact Microsoft Support.
Known issues for version 2411
This software release maps to software version number 2411.0.24.
Importante
The new deployments of this software will use the 2411.0.22 build whereas if you update from 2408.2, you'll get the 2411.0.24 build. No action is required if you have already updated from 2408.2 to 2411.0.22.
Release notes for this version include the issues fixed in this release, known issues in this release, and release note issues carried over from previous versions.
If you try to enable guest management on a migrated VM, the operation fails with the following error: (InternalError) admission webhook "createupdatevalidationwebhook.infrastructure.azstackhci.microsoft.com" denied the request: OsProfile cannot be changed after resource creation
Known issues in this release
The following table lists the known issues in this release:
Feature
Issue
Workaround
Security vulnerability
Microsoft has identified a security vulnerability that could expose the local admin credentials used during the creation of Arc VMs on Azure Local to non-admin users on the VM and on the hosts. Arc VMs running on releases prior to Azure Local 2411 release are vulnerable.
If the timezone is not set to UTC before you deploy Azure Local, an ArcOperationTimeOut error occurs during validation. The following error message is displayed: OperationTimeOut, No updates received from device for operation.
Depending on your scenario, choose one of the following workarounds for this issue:
Scenario 1. Before you start the deployment, make sure that the timezone is set to UTC.
Connect to each of the Azure Local nodes and change the timezone to UTC.
Run the following command: Set-TimeZone -Id "UTC".
Scenario 2. If you started the deployment without setting the UTC timezone and received the error mentioned in the validation phase, follow these steps:
1. Connect to each Azure Local node. Change the time zone to UTC with Set-TimeZone -Id "UTC". Reboot the nodes.
2. After the nodes have restarted, go to the Azure Local resource in Azure portal. Start the validation again to resolve the issue and continue with the deployment or upgrade.
For detailed remediation steps, see the troubleshooting guide in the Azure Local Supportability GitHub repository.
Update
When updating from version 2408.2.7 to 2411.0.24, the update process could fail with the following error message: Type 'CauPreRequisites' of Role 'CAU' raised an exception: Could not finish cau prerequisites due to error 'Cannot remove item C:\UpdateDistribution\<any_file_name>: Access to the path is denied.'
With the 2411 release, solution and Solution Builder Extension update are not combined in a single update run.
To apply a Solution Builder Extension package, you need a separate update run.
Update
When applying solution update in this release, the update can fail. This will occur only if the update was started prior to November 26. The issue that causes the failure can result in one of the following error messages:
Error 1 - The step "update ARB and extension" error "Clear-AzContext failed with 0 and Exception calling "Initialize" with "1" argument(s): "Object reference not set to an instance of an object." at "Clear-AzPowerShellCache".
Error 2 - The step "EvalTVMFlow" error "CloudEngine.Actions.InterfaceInvocationFailedException: Type 'EvalTVMFlow' of Role 'ArcIntegration' raised an exception: This module requires Az.Accounts version 3.0.5. An earlier version of Az.Accounts is imported in the current PowerShell session. Please open a new session before importing this module. This error could indicate that multiple incompatible versions of the Azure PowerShell cmdlets are installed on your system. Please see https://aka.ms/azps-version-error for troubleshooting information."
Depending on the version of PowerShell modules, the above error could be reported for both versions 3.0.4 and 3.0.5.
The following table lists the known issues from previous releases:
Feature
Issue
Workaround
Repair server
After you repair a node and run the command Set-AzureStackLCMUserPassword, you may encounter the following error:
CloudEngine.Actions.InterfaceInvocationFailedException: Type 'ValidateCredentials' of Role 'SecretRotation' raised an exception: Cannot load encryption certificate. The certificate setting 'CN=DscEncryptionCert' does not represent a valid base-64 encoded certificate, nor does it represent a valid certificate by file, directory, thumbprint, or subject name. at Validate-Credentials
Follow these steps to mitigate the issue:
$NewPassword = <Provide new password as secure string>
$OldPassword = <Provide the old/current password as secure string>
if ($eceClusterGroup.State -ne "Online") {Write-AzsSecurityError -Message "ECE cluster group is not in an Online state. Cannot continue with password rotation." -ErrRecord $_}
3. Update the ECE with the new password:
Write-AzsSecurityVerbose -Message "Updating password in ECE" -Verbose
Using an exported Azure VM OS disk as a VHD to create a gallery image for provisioning an Arc VM is unsupported.
Run the command restart-service mochostagent to restart the mochostagent service.
Networking
When a node is configured with a proxy server that has capital letters in its address, such as HTTPS://10.100.000.00:8080, Arc extensions fail to install or update on the node in existing builds, including version 2408.1. However, the node remains Arc connected.
Follow these steps to mitigate the issue:
1. Set the environment values in lowercase. [System.Environment]::SetEnvironmentVariable("HTTPS_PROXY", "https://10.100.000.00:8080", "Machine").
2. Validate that the values were set. [System.Environment]::GetEnvironmentVariable("HTTPS_PROXY", "Machine").
3. Restart Arc services.
Restart-Service himds
Restart-Service ExtensionService
Restart-Service GCArcService
4. Signal the AzcmaAgent with the lowercase proxy information.
& 'C:\Program Files\AzureConnectedMachineAgent\azcmagent.exe' config set proxy.url https://10.100.000.00:8080
& 'C:\Program Files\AzureConnectedMachineAgent\azcmagent.exe' config list
Networking
When Arc machines go down, the "All Clusters" page, in the new portal experience shows a "PartiallyConnected" or "Not Connected Recently status. Even when the Arc machines become healthy, they may not show a "Connected" status.
There's no known workaround for this issue. To check the connectivity status, use the old experience to see if it shows as "Connected".
Arc VM management
The Mochostagent service might appear to be running but can get stuck without updating logs for over a month. You can identify this issue by checking the service logs in C:\programdata\mochostagent\logs to see if logs are being updated.
Run the following command to restart the mochostagent service: restart-service mochostagent.
Upgrade
When upgrading the stamp from 2311 or prior builds to 2408 or later, add node and repair node operations may fail. For example, you could see an error: Type 'AddAsZHostToDomain' of Role 'BareMetal' raised an exception.
There's no workaround in this release. If you encounter this issue, contact Microsoft Support to determine next steps.
Update
When viewing the readiness check results for an Azure Local instance via the Azure Update Manager, there might be multiple readiness checks with the same name.
There's no known workaround in this release. Select View details to view specific information about the readiness check.
Deployment
In some instances, during the registration of Azure Local machines, this error might be seen in the debug logs: Encountered internal server error. One of the mandatory extensions for device deployment might not be installed.
There's an intermittent issue in this release when the Azure portal incorrectly reports the update status as Failed to update or In progress though the update is complete.
$Update = get-solutionupdate| ? version -eq "<version string>"
Replace the version string with the version you're running. For example, "10.2405.0.23".
$Update.state
If the update status is Installed, no further action is required on your part. Azure portal refreshes the status correctly within 24 hours. To refresh the status sooner, follow these steps on one of the nodes. Restart the Cloud Management cluster group. Stop-ClusterGroup "Cloud Management" Start-ClusterGroup "Cloud Management"
Update
During an initial MOC update, a failure occurs due to the target MOC version not being found in the catalog cache. The follow-up updates and retries show MOC in the target version, without the update succeeding, and as a result the Arc Resource Bridge update fails.
To validate this issue, collect the update logs using Troubleshoot solution updates for Azure Local. The log files should show a similar error message (current version might differ in the error message):
[ERROR: { "errorCode": "InvalidEntityError", "errorResponse": "{\n\"message\": \"the cloud fabric (MOC) is currently at version v0.13.1. A minimum version of 0.15.0 is required for compatibility\"\n}" }]
Follow these steps to mitigate the issue:
1. To find the MOC agent version, run the following command: 'C:\Program Files\AksHci\wssdcloudagent.exe' version.
2. Use the output of the command to find the MOC version from the table below that matches the agent version, and set $initialMocVersion to that MOC version. Set the $targetMocVersion by finding the Azure Local build you're updating to and get the matching MOC version from the following table. Use these values in the mitigation script provided below:
Build
MOC version
Agent version
2311.2
1.0.24.10106
v0.13.0-6-gf13a73f7, v0.11.0-alpha.38,01/06/2024
2402
1.0.25.10203
v0.14.0, v0.13.1, 02/02/2024
2402.1
1.0.25.10302
v0.14.0, v0.13.1, 03/02/2024
2402.2
1.1.1.10314
v0.16.0-1-g04bf0dec, v0.15.1, 03/14/2024
2405/2402.3
1.3.0.10418
v0.17.1, v0.16.5, 04/18/2024
For example, if the agent version is v0.13.0-6-gf13a73f7, v0.11.0-alpha.38,01/06/2024, then $initialMocVersion = "1.0.24.10106" and if you are updating to 2405.0.23, then $targetMocVersion = "1.3.0.10418".
3. Run the following PowerShell commands on the first node:
$initialMocVersion = "<initial version determined from step 2>" $targetMocVersion = "<target version determined from step 2>"
# Import MOC module twice import-module moc import-module moc $verbosePreference = "Continue"
# Clear the SFS catalog cache Remove-Item (Get-MocConfig).manifestCache
# Set version to the current MOC version prior to update, and set state as update failed Set-MocConfigValue -name "version" -value $initialMocVersion Set-MocConfigValue -name "installState" -value ([InstallState]::UpdateFailed)
# Rerun the MOC update to desired version Update-Moc -version $targetMocVersion
4. Resume the update.
AKS on Azure Local
AKS cluster creation fails with the Error: Invalid AKS network resource id. This issue can occur when the associated logical network name has an underscore.
Underscores aren't supported in logical network names. Make sure to not use underscore in the names for logical networks deployed on your Azure Local.
Repair server
In rare instances, the Repair-Server operation fails with the HealthServiceWaitForDriveFW error. In these cases, the old drives from the repaired node aren't removed and new disks are stuck in the maintenance mode.
To prevent this issue, make sure that you DO NOT drain the node either via the Windows Admin Center or using the Suspend-ClusterNode -Drain PowerShell cmdlet before you start Repair-Server. If the issue occurs, contact Microsoft Support for next steps.
Repair server
This issue is seen when the single node Azure Local instance is updated from 2311 to 2402 and then the Repair-Server is performed. The repair operation fails.
Before you repair the single node, follow these steps: 1. Run version 2402 for the ADPrepTool. Follow the steps in Prepare Active Directory. This action is quick and adds the required permissions to the Organizational Unit (OU). 2. Move the computer object from Computers segment to the root OU. Run the following command: Get-ADComputer <HOSTNAME> | Move-ADObject -TargetPath "<OU path>"
Deployment
If you prepare the Active Directory on your own (not using the script and procedure provided by Microsoft), your Active Directory validation could fail with missing Generic All permission. This is due to an issue in the validation check that checks for a dedicated permission entry for msFVE-RecoverInformationobjects – General – Permissions Full control, which is required for BitLocker recovery.
Use the Prepare AD script method or if using your own method, make sure to assign the specific permission msFVE-RecoverInformationobjects – General – Permissions Full control.
Deployment
There's a rare issue in this release where the DNS record is deleted during the Azure Local deployment. When that occurs, the following exception is seen: Type 'PropagatePublicRootCertificate' of Role 'ASCA' raised an exception:<br>The operation on computer 'ASB88RQ22U09' failed: WinRM cannot process the request. The following error occurred while using Kerberos authentication: Cannot find the computer ASB88RQ22U09.local. Verify that the computer exists on the network and that the name provided is spelled correctly at PropagatePublicRootCertificate, C:\NugetStore\Microsoft.AzureStack, at Orchestration.Roles.CertificateAuthority.10.2402.0.14\content\Classes\ASCA\ASCA.psm1: line 38, at C:\CloudDeployment\ECEngine\InvokeInterfaceInternal.psm1: line 127,at Invoke-EceInterfaceInternal, C:\CloudDeployment\ECEngine\InvokeInterfaceInternal.psm1: line 123.
Check the DNS server to see if any DNS records of the nodes are missing. Apply the following mitigation on the nodes where its DNS record is missing.
Restart the DNS client service. Open a PowerShell session and run the following cmdlet on the affected node: Taskkill /f /fi "SERVICES eq dnscache"
Deployment
In this release, there's a remote task failure on a multi-node deployment that results in the following exception: ECE RemoteTask orchestration failure with ASRR1N42R01U31 (node pingable - True): A WebException occurred while sending a RestRequest. WebException.Status: ConnectFailure on [https://<URL>](https://<URL>).
The mitigation is to restart the ECE agent on the affected node. On your machine, open a PowerShell session and run the following command: Restart-Service ECEAgent.
Add server
In this release and previous releases, when adding a machine to the system, is not possible to update the proxy bypass list string to include the new machine. Updating environment variables proxy bypass list on the hosts will not update the proxy bypass list on Azure Resource Bridge or AKS.
There's no workaround in this release. If you encounter this issue, contact Microsoft Support to determine next steps.
Add/Repair server
In this release, when adding or repairing a machine, a failure is seen when the software load balancer or network controller VM certificates are being copied from the existing nodes. The failure is because these certificates weren't generated during the deployment/update.
There's no workaround in this release. If you encounter this issue, contact Microsoft Support to determine next steps.
Deployment
In this release, there's a transient issue resulting in the deployment failure with the following exception: Type 'SyncDiagnosticLevel' of Role 'ObservabilityConfig' raised an exception:*<br>*Syncing Diagnostic Level failed with error: The Diagnostic Level does not match. Portal was not set to Enhanced, instead is Basic.
As this is a transient issue, retrying the deployment should fix this. For more information, see how to Rerun the deployment.
Deployment
In this release, there's an issue with the Secrets URI/location field. This is a required field that is marked Not mandatory and results in Azure Resource Manager template deployment failures.
Use the sample parameters file in the Deploy Azure Local via Azure Resource Manager template to ensure that all the inputs are provided in the required format and then try the deployment. If there's a failed deployment, you must also clean up the following resources before you Rerun the deployment: 1. Delete C:\EceStore. 2. Delete C:\CloudDeployment. 3. Delete C:\nugetstore. 4. Remove-Item HKLM:\Software\Microsoft\LCMAzureStackStampInformation.
Security
For new deployments, Secured-core capable devices won't have Dynamic Root of Measurement (DRTM) enabled by default. If you try to enable (DRTM) using the Enable-AzSSecurity cmdlet, you see an error that DRTM setting isn't supported in the current release. Microsoft recommends defense in depth, and UEFI Secure Boot still protects the components in the Static Root of Trust (SRT) boot chain by ensuring that they're loaded only when they're signed and verified.
DRTM isn't supported in this release.
Networking
An environment check fails when a proxy server is used. By design, the bypass list is different for winhttp and wininet, which causes the validation check to fail.
Follow these workaround steps:
1. Clear the proxy bypass list prior to the health check and before starting the deployment or the update.
2. After passing the check, wait for the deployment or update to fail.
3. Set your proxy bypass list again.
Arc VM management
Deployment or update of Arc Resource Bridge could fail when the automatically generated temporary SPN secret during this operation, starts with a hyphen.
Retry the deployment/update. The retry should regenerate the SPN secret and the operation will likely succeed.
Arc VM management
Arc Extensions on Arc VMs stay in "Creating" state indefinitely.
Sign in to the VM, open a command prompt, and type the following: Windows: notepad C:\ProgramData\AzureConnectedMachineAgent\Config\agentconfig.json Linux: sudo vi /var/opt/azcmagent/agentconfig.json Next, find the resourcename property. Delete the GUID that is appended to the end of the resource name, so this property matches the name of the VM. Then restart the VM.
Arc VM management
When a new machine is added to an Azure Local instance, storage path isn't created automatically for the newly created volume.
You can manually create a storage path for any new volumes. For more information, see Create a storage path.
Arc VM management
Restart of Arc VM operation completes after approximately 20 minutes although the VM itself restarts in about a minute.
There's no known workaround in this release.
Arc VM management
In some instances, the status of the logical network shows as Failed in Azure portal. This occurs when you try to delete the logical network without first deleting any resources such as network interfaces associated with that logical network. You should still be able to create resources on this logical network. The status is misleading in this instance.
If the status of this logical network was Succeeded at the time when this network was provisioned, then you can continue to create resources on this network.
Arc VM management
In this release, when you update a VM with a data disk attached to it using the Azure CLI, the operation fails with the following error message: Couldn't find a virtual hard disk with the name.
In rare instances, you may encounter this error while updating your Azure Local instance: Type 'UpdateArbAndExtensions' of Role 'MocArb' raised an exception: Exception Upgrading ARB and Extension in step [UpgradeArbAndExtensions :Get-ArcHciConfig] UpgradeArb: Invalid applianceyaml = [C:\AksHci\hci-appliance.yaml].
If you see this issue, contact Microsoft Support to assist you with the next steps.
Networking
There's an infrequent DNS client issue in this release that causes the deployment to fail on a two-node system with a DNS resolution error: A WebException occurred while sending a RestRequest. WebException.Status: NameResolutionFailure. As a result of the bug, the DNS record of the second node is deleted soon after it's created resulting in a DNS error.
Restart the machine. This operation registers the DNS record, which prevents it from getting deleted.
Azure portal
In some instances, the Azure portal might take a while to update and the view might not be current.
You might need to wait for 30 minutes or more to see the updated view.
Arc VM management
Deleting a network interface on an Arc VM from Azure portal doesn't work in this release.
Providing the OU name in an incorrect syntax isn't detected in the Azure portal. The incorrect syntax includes unsupported characters such as &,",',<,>. The incorrect syntax is detected at a later step during system validation.
Make sure that the OU path syntax is correct and doesn't include unsupported characters.
Deployment
Deployments via Azure Resource Manager time out after 2 hours. Deployments that exceed 2 hours show up as failed in the resource group though the system is successfully created.
To monitor the deployment in the Azure portal, go to the Azure Local instance resource and then go to new Deployments entry.
Azure Site Recovery
Azure Site Recovery can't be installed on an Azure Local instance in this release.
There's no known workaround in this release.
Update
When updating the Azure Local instance via the Azure Update Manager, the update progress and results may not be visible in the Azure portal.
To work around this issue, on each node, add the following registry key (no value needed):
Then on one of the nodes, restart the Cloud Management cluster group.
Stop-ClusterGroup "Cloud Management"
Start-ClusterGroup "Cloud Management"
This won't fully remediate the issue as the progress details may still not be displayed for a duration of the update process. To get the latest update details, you can Retrieve the update progress with PowerShell.
Update
In rare instances, if a failed update is stuck in an In progress state in Azure Update Manager, the Try again button is disabled.
To resume the update, run the following PowerShell command: Get-SolutionUpdate|Start-SolutionUpdate.
Update
In some cases, SolutionUpdate commands could fail if run after the Send-DiagnosticData command.
Make sure to close the PowerShell session used for Send-DiagnosticData. Open a new PowerShell session and use it for SolutionUpdate commands.
Update
In rare instances, when applying an update from 2311.0.24 to 2311.2.4, system status reports In Progress instead of expected Failed to update.
Retry the update. If the issue persists, contact Microsoft Support.
Update
Attempts to install solution updates can fail at the end of the CAU steps with: There was a failure in a Common Information Model (CIM) operation, that is, an operation performed by software that Cluster-Aware Updating depends on. This rare issue occurs if the Cluster Name or Cluster IP Address resources fail to start after a node reboot and is most typical in small deployments.
If you encounter this issue, contact Microsoft Support for next steps. They can work with you to manually restart the Azure Local resources and resume the update as needed.
Update
When applying a system update to 10.2402.3.11, the Get-SolutionUpdate cmdlet may not respond and eventually fails with a RequestTimeoutException after approximately 10 minutes. This is likely to occur following an add or repair server scenario.
Use the Start-ClusterGroup and Stop-ClusterGroup cmdlets to restart the update service.
A successful run of these cmdlets should bring the update service online.
Cluster aware updating
Resume node operation failed to resume node.
This is a transient issue and could resolve on its own. Wait for a few minutes and retry the operation. If the issue persists, contact Microsoft Support.
Cluster aware updating
Suspend node operation was stuck for greater than 90 minutes.
This is a transient issue and could resolve on its own. Wait for a few minutes and retry the operation. If the issue persists, contact Microsoft Support.
Known issues for version 2408.2
This software release maps to software version number 2408.2.7.
Release notes for this version include the issues fixed in this release, known issues in this release, and release note issues carried over from previous versions.
SideChannelMitigation is reporting properly in both local cmdlets and Windows Admin Center.
Update
An update would unnecessarily download Solution Builder Extension content that was already added.
Upgrade
Cluster resources weren't in the same group.
Upgrade
Fixed IP pool validation in the Azure portal.
Upgrade
Added validation to ensure the package is the latest version
Upgrade
Validation would fail due to group policies.
Known issues in this release
Microsoft isn't aware of any known issues in this release.
Known issues from previous releases
The following table lists the known issues from previous releases:
Feature
Issue
Workaround
Repair node
After you repair a node and run the command Set-AzureStackLCMUserPassword, you may encounter the following error:
CloudEngine.Actions.InterfaceInvocationFailedException: Type 'ValidateCredentials' of Role 'SecretRotation' raised an exception: Cannot load encryption certificate. The certificate setting 'CN=DscEncryptionCert' does not represent a valid base-64 encoded certificate, nor does it represent a valid certificate by file, directory, thumbprint, or subject name. at Validate-Credentials
Follow these steps to mitigate the issue:
$NewPassword = <Provide new password as secure string>
$OldPassword = <Provide the old/current password as secure string>
if ($eceClusterGroup.State -ne "Online") {Write-AzsSecurityError -Message "ECE cluster group is not in an Online state. Cannot continue with password rotation." -ErrRecord $_}
3. Update the ECE with the new password:
Write-AzsSecurityVerbose -Message "Updating password in ECE" -Verbose
Using an exported Azure VM OS disk as a VHD to create a gallery image for provisioning an Arc VM is unsupported.
Run the command restart-service mochostagent to restart the mochostagent service.
Arc VM management
If you try to enable guest management on a migrated VM, the operation fails with the following error: (InternalError) admission webhook "createupdatevalidationwebhook.infrastructure.azstackhci.microsoft.com" denied the request: OsProfile can't be changed after resource creation
Networking
When a machine is configured with a proxy server that has capital letters in its address, such as HTTPS://10.100.000.00:8080, Arc extensions fail to install or update on the machine in existing builds, including version 2408.2. However, the machine remains Arc connected.
Follow these steps to mitigate the issue:
1. Set the environment values in lowercase. [System.Environment]::SetEnvironmentVariable("HTTPS_PROXY", "https://10.100.000.00:8080", "Machine").
2. Validate that the values were set. [System.Environment]::GetEnvironmentVariable("HTTPS_PROXY", "Machine").
3. Restart Arc services.
Restart-Service himds
Restart-Service ExtensionService
Restart-Service GCArcService
4. Signal the AzcmaAgent with the lowercase proxy information.
& 'C:\Program Files\AzureConnectedMachineAgent\azcmagent.exe' config set proxy.url https://10.100.000.00:8080
& 'C:\Program Files\AzureConnectedMachineAgent\azcmagent.exe' config list
Networking
When Arc machines go down, the "All Clusters" page, in the new portal experience shows a "PartiallyConnected" or "Not Connected Recently status. Even when the Arc machines become healthy, they may not show a "Connected" status.
There's no known workaround for this issue. To check the connectivity status, use the old experience to see if it shows as "Connected".
Security
The SideChannelMitigation security feature may not show an enabled state even if it's enabled.
There's no workaround in this release. If you encounter this issue, contact Microsoft Support to determine next steps.
Arc VM management
The Mochostagent service might appear to be running but can get stuck without updating logs for over a month. You can identify this issue by checking the service logs in C:\programdata\mochostagent\logs to see if logs are being updated.
Run the following command to restart the mochostagent service: restart-service mochostagent.
Upgrade
When upgrading the stamp from 2311 or prior builds to 2408 or later, add node and repair node operations may fail. For example, you could see an error: Type 'AddAsZHostToDomain' of Role 'BareMetal' raised an exception.
There's no workaround in this release. If you encounter this issue, contact Microsoft Support to determine next steps.
Update
When viewing the readiness check results for an Azure Local instance via the Azure Update Manager, there might be multiple readiness checks with the same name.
There's no known workaround in this release. Select View details to view specific information about the readiness check.
Deployment
In some instances, during the registration of Azure Local machines, this error might be seen in the debug logs: Encountered internal server error. One of the mandatory extensions for device deployment might not be installed.
There's an intermittent issue in this release when the Azure portal incorrectly reports the update status as Failed to update or In progress though the update is complete.
Connect to your Azure Local via a remote PowerShell session. To confirm the update status, run the following PowerShell cmdlets:
$Update = get-solutionupdate | ? version -eq "<version string>"
Replace the version string with the version you're running. For example, "10.2405.0.23".
$Update.state
If the update status is Installed, no further action is required on your part. Azure portal refreshes the status correctly within 24 hours. To refresh the status sooner, follow these steps on one of the machines. Restart the Cloud Management cluster group. Stop-ClusterGroup "Cloud Management" Start-ClusterGroup "Cloud Management"
Update
During an initial MOC update, a failure occurs due to the target MOC version not being found in the catalog cache. The follow-up updates and retries show MOC in the target version, without the update succeeding, and as a result the Arc Resource Bridge update fails.
To validate this issue, collect the update logs using Troubleshoot solution updates for Azure Local. The log files should show a similar error message (current version might differ in the error message):
[ERROR: { "errorCode": "InvalidEntityError", "errorResponse": "{\n\"message\": \"the cloud fabric (MOC) is currently at version v0.13.1. A minimum version of 0.15.0 is required for compatibility\"\n}" }]
Follow these steps to mitigate the issue:
1. To find the MOC agent version, run the following command: 'C:\Program Files\AksHci\wssdcloudagent.exe' version.
2. Use the output of the command to find the MOC version from the table below that matches the agent version, and set $initialMocVersion to that MOC version. Set the $targetMocVersion by finding the Azure Local build you're updating to and get the matching MOC version from the following table. Use these values in the mitigation script provided below:
Build
MOC version
Agent version
2311.2
1.0.24.10106
v0.13.0-6-gf13a73f7, v0.11.0-alpha.38,01/06/2024
2402
1.0.25.10203
v0.14.0, v0.13.1, 02/02/2024
2402.1
1.0.25.10302
v0.14.0, v0.13.1, 03/02/2024
2402.2
1.1.1.10314
v0.16.0-1-g04bf0dec, v0.15.1, 03/14/2024
2405/2402.3
1.3.0.10418
v0.17.1, v0.16.5, 04/18/2024
For example, if the agent version is v0.13.0-6-gf13a73f7, v0.11.0-alpha.38,01/06/2024, then $initialMocVersion = "1.0.24.10106" and if you're updating to 2405.0.23, then $targetMocVersion = "1.3.0.10418".
3. Run the following PowerShell commands on the first node:
$initialMocVersion = "<initial version determined from step 2>" $targetMocVersion = "<target version determined from step 2>"
# Import MOC module twice import-module moc import-module moc $verbosePreference = "Continue"
# Clear the SFS catalog cache Remove-Item (Get-MocConfig).manifestCache
# Set version to the current MOC version prior to update, and set state as update failed Set-MocConfigValue -name "version" -value $initialMocVersion Set-MocConfigValue -name "installState" -value ([InstallState]::UpdateFailed)
# Rerun the MOC update to desired version Update-Moc -version $targetMocVersion
4. Resume the update.
AKS on HCI
AKS cluster creation fails with the Error: Invalid AKS network resource id. This issue can occur when the associated logical network name has an underscore.
Underscores aren't supported in logical network names. Make sure to not use underscore in the names for logical networks deployed on your Azure Local.
Repair node
In rare instances, the Repair-Server operation fails with the HealthServiceWaitForDriveFW error. In these cases, the old drives from the repaired node aren't removed and new disks are stuck in the maintenance mode.
To prevent this issue, make sure that you DO NOT drain the node either via the Windows Admin Center or using the Suspend-ClusterNode -Drain PowerShell cmdlet before you start Repair-Server. If the issue occurs, contact Microsoft Support for next steps.
Repair node
This issue is seen when the single machine Azure Local is updated from 2311 to 2402 and then the Repair-Server is performed. The repair operation fails.
Before you repair the single node, follow these steps: 1. Run version 2402 for the ADPrepTool. Follow the steps in Prepare Active Directory. This action is quick and adds the required permissions to the Organizational Unit (OU). 2. Move the computer object from Computers segment to the root OU. Run the following command: Get-ADComputer <HOSTNAME> | Move-ADObject -TargetPath "<OU path>"
Deployment
If you prepare the Active Directory on your own (not using the script and procedure provided by Microsoft), your Active Directory validation could fail with missing Generic All permission. This is due to an issue in the validation check that checks for a dedicated permission entry for msFVE-RecoverInformationobjects – General – Permissions Full control, which is required for BitLocker recovery.
Use the Prepare AD script method or if using your own method, make sure to assign the specific permission msFVE-RecoverInformationobjects – General – Permissions Full control.
Deployment
There's a rare issue in this release where the DNS record is deleted during the Azure Local deployment. When that occurs, the following exception is seen: Type 'PropagatePublicRootCertificate' of Role 'ASCA' raised an exception:<br>The operation on computer 'ASB88RQ22U09' failed: WinRM cannot process the request. The following error occurred while using Kerberos authentication: Cannot find the computer ASB88RQ22U09.local. Verify that the computer exists on the network and that the name provided is spelled correctly at PropagatePublicRootCertificate, C:\NugetStore\Microsoft.AzureStack, at Orchestration.Roles.CertificateAuthority.10.2402.0.14\content\Classes\ASCA\ASCA.psm1: line 38, at C:\CloudDeployment\ECEngine\InvokeInterfaceInternal.psm1: line 127,at Invoke-EceInterfaceInternal, C:\CloudDeployment\ECEngine\InvokeInterfaceInternal.psm1: line 123.
Check the DNS server to see if any DNS records of the machines are missing. Apply the following mitigation on the machines where its DNS record is missing.
Restart the DNS client service. Open a PowerShell session and run the following cmdlet on the affected machine: Taskkill /f /fi "SERVICES eq dnscache"
Deployment
In this release, there's a remote task failure on a multi-node deployment that results in the following exception: ECE RemoteTask orchestration failure with ASRR1N42R01U31 (node pingable - True): A WebException occurred while sending a RestRequest. WebException.Status: ConnectFailure on [https://<URL>](https://<URL>).
The mitigation is to restart the ECE agent on the affected node. On your machine, open a PowerShell session and run the following command: Restart-Service ECEAgent.
Add node
In this release and previous releases, when adding a node to the cluster, isn't possible to update the proxy bypass list string to include the new node. Updating environment variables proxy bypass list on the hosts won't update the proxy bypass list on Azure Resource Bridge or AKS.
There's no workaround in this release. If you encounter this issue, contact Microsoft Support to determine next steps.
Add/Repair node
In this release, when adding or repairing a node, a failure is seen when the software load balancer or network controller VM certificates are being copied from the existing nodes. The failure is because these certificates weren't generated during the deployment/update.
There's no workaround in this release. If you encounter this issue, contact Microsoft Support to determine next steps.
Deployment
In this release, there's a transient issue resulting in the deployment failure with the following exception: Type 'SyncDiagnosticLevel' of Role 'ObservabilityConfig' raised an exception:*<br>*Syncing Diagnostic Level failed with error: The Diagnostic Level does not match. Portal was not set to Enhanced, instead is Basic.
As this is a transient issue, retrying the deployment should fix this. For more information, see how to Rerun the deployment.
Deployment
In this release, there's an issue with the Secrets URI/location field. This is a required field that is marked Not mandatory and results in Azure Resource Manager template deployment failures.
Use the sample parameters file in the Deploy Azure Local via Azure Resource Manager template to ensure that all the inputs are provided in the required format and then try the deployment. If there's a failed deployment, you must also clean up the following resources before you Rerun the deployment: 1. Delete C:\EceStore. 2. Delete C:\CloudDeployment. 3. Delete C:\nugetstore. 4. Remove-Item HKLM:\Software\Microsoft\LCMAzureStackStampInformation.
Security
For new deployments, Secured-core capable devices won't have Dynamic Root of Measurement (DRTM) enabled by default. If you try to enable (DRTM) using the Enable-AzSSecurity cmdlet, you see an error that DRTM setting isn't supported in the current release. Microsoft recommends defense in depth, and UEFI Secure Boot still protects the components in the Static Root of Trust (SRT) boot chain by ensuring that they're loaded only when they're signed and verified.
DRTM isn't supported in this release.
Networking
An environment check fails when a proxy server is used. By design, the bypass list is different for winhttp and wininet, which causes the validation check to fail.
Follow these workaround steps:
1. Clear the proxy bypass list prior to the health check and before starting the deployment or the update.
2. After passing the check, wait for the deployment or update to fail.
3. Set your proxy bypass list again.
Arc VM management
Deployment or update of Arc Resource Bridge could fail when the automatically generated temporary SPN secret during this operation, starts with a hyphen.
Retry the deployment/update. The retry should regenerate the SPN secret and the operation will likely succeed.
Arc VM management
Arc Extensions on Arc VMs stay in "Creating" state indefinitely.
Sign in to the VM, open a command prompt, and type the following: Windows: notepad C:\ProgramData\AzureConnectedMachineAgent\Config\agentconfig.json Linux: sudo vi /var/opt/azcmagent/agentconfig.json Next, find the resourcename property. Delete the GUID that is appended to the end of the resource name, so this property matches the name of the VM. Then restart the VM.
Arc VM management
When a new machine is added to an Azure Local instance, storage path isn't created automatically for the newly created volume.
You can manually create a storage path for any new volumes. For more information, see Create a storage path.
Arc VM management
Restart of Arc VM operation completes after approximately 20 minutes although the VM itself restarts in about a minute.
There's no known workaround in this release.
Arc VM management
In some instances, the status of the logical network shows as Failed in Azure portal. This occurs when you try to delete the logical network without first deleting any resources such as network interfaces associated with that logical network. You should still be able to create resources on this logical network. The status is misleading in this instance.
If the status of this logical network was Succeeded at the time when this network was provisioned, then you can continue to create resources on this network.
Arc VM management
In this release, when you update a VM with a data disk attached to it using the Azure CLI, the operation fails with the following error message: Couldn't find a virtual hard disk with the name.
In rare instances, you may encounter this error while updating your Azure Local: Type 'UpdateArbAndExtensions' of Role 'MocArb' raised an exception: Exception Upgrading ARB and Extension in step [UpgradeArbAndExtensions :Get-ArcHciConfig] UpgradeArb: Invalid applianceyaml = [C:\AksHci\hci-appliance.yaml].
If you see this issue, contact Microsoft Support to assist you with the next steps.
Networking
There's an infrequent DNS client issue in this release that causes the deployment to fail on a two-node cluster with a DNS resolution error: A WebException occurred while sending a RestRequest. WebException.Status: NameResolutionFailure. As a result of the bug, the DNS record of the second node is deleted soon after it's created resulting in a DNS error.
Restart the machine. This operation registers the DNS record, which prevents it from getting deleted.
Azure portal
In some instances, the Azure portal might take a while to update and the view might not be current.
You might need to wait for 30 minutes or more to see the updated view.
Arc VM management
Deleting a network interface on an Arc VM from Azure portal doesn't work in this release.
Providing the OU name in an incorrect syntax isn't detected in the Azure portal. The incorrect syntax includes unsupported characters such as &,",',<,>. The incorrect syntax is detected at a later step during cluster validation.
Make sure that the OU path syntax is correct and doesn't include unsupported characters.
Deployment
Deployments via Azure Resource Manager time out after 2 hours. Deployments that exceed 2 hours show up as failed in the resource group though the cluster is successfully created.
To monitor the deployment in the Azure portal, go to the Azure Local resource and then go to new Deployments entry.
Azure Site Recovery
Azure Site Recovery can't be installed on an Azure Local instance in this release.
There's no known workaround in this release.
Update
When updating the Azure Local instance via the Azure Update Manager, the update progress and results may not be visible in the Azure portal.
To work around this issue, on each machine, add the following registry key (no value needed):
Then on one of the machines, restart the Cloud Management cluster group.
Stop-ClusterGroup "Cloud Management"
Start-ClusterGroup "Cloud Management"
This won't fully remediate the issue as the progress details may still not be displayed for a duration of the update process. To get the latest update details, you can Retrieve the update progress with PowerShell.
Update
In rare instances, if a failed update is stuck in an In progress state in Azure Update Manager, the Try again button is disabled.
To resume the update, run the following PowerShell command: Get-SolutionUpdate | Start-SolutionUpdate.
Update
In some cases, SolutionUpdate commands could fail if run after the Send-DiagnosticData command.
Make sure to close the PowerShell session used for Send-DiagnosticData. Open a new PowerShell session and use it for SolutionUpdate commands.
Update
In rare instances, when applying an update from 2311.0.24 to 2311.2.4, cluster status reports In Progress instead of expected Failed to update.
Retry the update. If the issue persists, contact Microsoft Support.
Update
Attempts to install solution updates can fail at the end of the CAU steps with: There was a failure in a Common Information Model (CIM) operation, that is, an operation performed by software that Cluster-Aware Updating depends on. This rare issue occurs if the Cluster Name or Cluster IP Address resources fail to start after a node reboot and is most typical in small clusters.
If you encounter this issue, contact Microsoft Support for next steps. They can work with you to manually restart the cluster resources and resume the update as needed.
Update
When applying a cluster update to 10.2402.3.11 the Get-SolutionUpdate cmdlet may not respond and eventually fails with a RequestTimeoutException after approximately 10 minutes. This is likely to occur following an add or repair node scenario.
Use the Start-ClusterGroup and Stop-ClusterGroup cmdlets to restart the update service.
A successful run of these cmdlets should bring the update service online.
Cluster aware updating
Resume node operation failed to resume node.
This is a transient issue and could resolve on its own. Wait for a few minutes and retry the operation. If the issue persists, contact Microsoft Support.
Cluster aware updating
Suspend node operation was stuck for greater than 90 minutes.
This is a transient issue and could resolve on its own. Wait for a few minutes and retry the operation. If the issue persists, contact Microsoft Support.
Known issues for version 2408.1
This software release maps to software version number 2408.1.9.
Release notes for this version include the issues fixed in this release, known issues in this release, and release note issues carried over from previous versions.
The MAC address of the VM network interface wouldn't appear if the customer didn't pass the mac address at the time of creation.
Update
MOC node agent would get stuck in a restart pending stage during the update MOC step.
Update
Required permissions weren't granted when upgrading which caused update to fail later.
Upgrade
Added validation to check for an IPv6 address.
Update
SBE interfaces wouldn't execute on all the machines if the hostname in the system was a subset of another hostname.
Known issues in this release
Microsoft isn't aware of any known issues in this release.
Known issues from previous releases
The following table lists the known issues from previous releases:
Feature
Issue
Workaround
Repair server
After you repair a node and run the command Set-AzureStackLCMUserPassword, you may encounter the following error:
CloudEngine.Actions.InterfaceInvocationFailedException: Type 'ValidateCredentials' of Role 'SecretRotation' raised an exception: Cannot load encryption certificate. The certificate setting 'CN=DscEncryptionCert' does not represent a valid base-64 encoded certificate, nor does it represent a valid certificate by file, directory, thumbprint, or subject name. at Validate-Credentials
Follow these steps to mitigate the issue:
$NewPassword = <Provide new password as secure string>
$OldPassword = <Provide the old/current password as secure string>
if ($eceClusterGroup.State -ne "Online") {Write-AzsSecurityError -Message "ECE cluster group is not in an Online state. Cannot continue with password rotation." -ErrRecord $_}
3. Update the ECE with the new password:
Write-AzsSecurityVerbose -Message "Updating password in ECE" -Verbose
Using an exported Azure VM OS disk as a VHD to create a gallery image for provisioning an Arc VM is unsupported.
Run the command restart-service mochostagent to restart the mochostagent service.
Arc VM management
If you try to enable guest management on a migrated VM, the operation fails with the following error: (InternalError) admission webhook "createupdatevalidationwebhook.infrastructure.azstackhci.microsoft.com" denied the request: OsProfile cannot be changed after resource creation
Networking
When a node is configured with a proxy server that has capital letters in its address, such as HTTPS://10.100.000.00:8080, Arc extensions fail to install or update on the node in existing builds, including version 2408.1. However, the node remains Arc connected.
Follow these steps to mitigate the issue:
1. Set the environment values in lowercase. [System.Environment]::SetEnvironmentVariable("HTTPS_PROXY", "https://10.100.000.00:8080", "Machine").
2. Validate that the values were set. [System.Environment]::GetEnvironmentVariable("HTTPS_PROXY", "Machine").
3. Restart Arc services.
Restart-Service himds
Restart-Service ExtensionService
Restart-Service GCArcService
4. Signal the AzcmaAgent with the lowercase proxy information.
& 'C:\Program Files\AzureConnectedMachineAgent\azcmagent.exe' config set proxy.url https://10.100.000.00:8080
& 'C:\Program Files\AzureConnectedMachineAgent\azcmagent.exe' config list
Networking
When Arc machines go down, the "All Clusters" page, in the new portal experience shows a "PartiallyConnected" or "Not Connected Recently status. Even when the Arc machines become healthy, they may not show a "Connected" status.
There's no known workaround for this issue. To check the connectivity status, use the old experience to see if it shows as "Connected".
Security
The SideChannelMitigation security feature may not show an enabled state even if it's enabled.
There's no workaround in this release. If you encounter this issue, contact Microsoft Support to determine next steps.
Arc VM management
The Mochostagent service might appear to be running but can get stuck without updating logs for over a month. You can identify this issue by checking the service logs in C:\programdata\mochostagent\logs to see if logs are being updated.
Run the following command to restart the mochostagent service: restart-service mochostagent.
Upgrade
When upgrading the stamp from 2311 or prior builds to 2408 or later, add node and repair node operations may fail. For example, you could see an error: Type 'AddAsZHostToDomain' of Role 'BareMetal' raised an exception.
There's no workaround in this release. If you encounter this issue, contact Microsoft Support to determine next steps.
Update
When viewing the readiness check results for an Azure Local instance via the Azure Update Manager, there might be multiple readiness checks with the same name.
There's no known workaround in this release. Select View details to view specific information about the readiness check.
Deployment
In some instances, during the registration of Azure Local machines, this error might be seen in the debug logs: Encountered internal server error. One of the mandatory extensions for device deployment might not be installed.
There's an intermittent issue in this release when the Azure portal incorrectly reports the update status as Failed to update or In progress though the update is complete.
$Update = get-solutionupdate| ? version -eq "<version string>"
Replace the version string with the version you're running. For example, "10.2405.0.23".
$Update.state
If the update status is Installed, no further action is required on your part. Azure portal refreshes the status correctly within 24 hours. To refresh the status sooner, follow these steps on one of the cluster nodes. Restart the Cloud Management cluster group. Stop-ClusterGroup "Cloud Management" Start-ClusterGroup "Cloud Management"
Update
During an initial MOC update, a failure occurs due to the target MOC version not being found in the catalog cache. The follow-up updates and retries show MOC in the target version, without the update succeeding, and as a result the Arc Resource Bridge update fails.
To validate this issue, collect the update logs using Troubleshoot solution updates for Azure Local. The log files should show a similar error message (current version might differ in the error message):
[ERROR: { "errorCode": "InvalidEntityError", "errorResponse": "{\n\"message\": \"the cloud fabric (MOC) is currently at version v0.13.1. A minimum version of 0.15.0 is required for compatibility\"\n}" }]
Follow these steps to mitigate the issue:
1. To find the MOC agent version, run the following command: 'C:\Program Files\AksHci\wssdcloudagent.exe' version.
2. Use the output of the command to find the MOC version from the table below that matches the agent version, and set $initialMocVersion to that MOC version. Set the $targetMocVersion by finding the Azure Local build you're updating to and get the matching MOC version from the following table. Use these values in the mitigation script provided below:
Build
MOC version
Agent version
2311.2
1.0.24.10106
v0.13.0-6-gf13a73f7, v0.11.0-alpha.38,01/06/2024
2402
1.0.25.10203
v0.14.0, v0.13.1, 02/02/2024
2402.1
1.0.25.10302
v0.14.0, v0.13.1, 03/02/2024
2402.2
1.1.1.10314
v0.16.0-1-g04bf0dec, v0.15.1, 03/14/2024
2405/2402.3
1.3.0.10418
v0.17.1, v0.16.5, 04/18/2024
For example, if the agent version is v0.13.0-6-gf13a73f7, v0.11.0-alpha.38,01/06/2024, then $initialMocVersion = "1.0.24.10106" and if you are updating to 2405.0.23, then $targetMocVersion = "1.3.0.10418".
3. Run the following PowerShell commands on the first node:
$initialMocVersion = "<initial version determined from step 2>" $targetMocVersion = "<target version determined from step 2>"
# Import MOC module twice import-module moc import-module moc $verbosePreference = "Continue"
# Clear the SFS catalog cache Remove-Item (Get-MocConfig).manifestCache
# Set version to the current MOC version prior to update, and set state as update failed Set-MocConfigValue -name "version" -value $initialMocVersion Set-MocConfigValue -name "installState" -value ([InstallState]::UpdateFailed)
# Rerun the MOC update to desired version Update-Moc -version $targetMocVersion
4. Resume the update.
AKS on Azure Local
AKS cluster creation fails with the Error: Invalid AKS network resource id. This issue can occur when the associated logical network name has an underscore.
Underscores aren't supported in logical network names. Make sure to not use underscore in the names for logical networks deployed on your Azure Local.
Repair server
In rare instances, the Repair-Server operation fails with the HealthServiceWaitForDriveFW error. In these cases, the old drives from the repaired node aren't removed and new disks are stuck in the maintenance mode.
To prevent this issue, make sure that you DO NOT drain the node either via the Windows Admin Center or using the Suspend-ClusterNode -Drain PowerShell cmdlet before you start Repair-Server. If the issue occurs, contact Microsoft Support for next steps.
Repair server
This issue is seen when the single node Azure Local instance is updated from 2311 to 2402 and then the Repair-Server is performed. The repair operation fails.
Before you repair the single node, follow these steps: 1. Run version 2402 for the ADPrepTool. Follow the steps in Prepare Active Directory. This action is quick and adds the required permissions to the Organizational Unit (OU). 2. Move the computer object from Computers segment to the root OU. Run the following command: Get-ADComputer <HOSTNAME> | Move-ADObject -TargetPath "<OU path>"
Deployment
If you prepare the Active Directory on your own (not using the script and procedure provided by Microsoft), your Active Directory validation could fail with missing Generic All permission. This is due to an issue in the validation check that checks for a dedicated permission entry for msFVE-RecoverInformationobjects – General – Permissions Full control, which is required for BitLocker recovery.
Use the Prepare AD script method or if using your own method, make sure to assign the specific permission msFVE-RecoverInformationobjects – General – Permissions Full control.
Deployment
There's a rare issue in this release where the DNS record is deleted during the Azure Local deployment. When that occurs, the following exception is seen: Type 'PropagatePublicRootCertificate' of Role 'ASCA' raised an exception:<br>The operation on computer 'ASB88RQ22U09' failed: WinRM cannot process the request. The following error occurred while using Kerberos authentication: Cannot find the computer ASB88RQ22U09.local. Verify that the computer exists on the network and that the name provided is spelled correctly at PropagatePublicRootCertificate, C:\NugetStore\Microsoft.AzureStack, at Orchestration.Roles.CertificateAuthority.10.2402.0.14\content\Classes\ASCA\ASCA.psm1: line 38, at C:\CloudDeployment\ECEngine\InvokeInterfaceInternal.psm1: line 127,at Invoke-EceInterfaceInternal, C:\CloudDeployment\ECEngine\InvokeInterfaceInternal.psm1: line 123.
Check the DNS server to see if any DNS records of the nodes are missing. Apply the following mitigation on the nodes where its DNS record is missing.
Restart the DNS client service. Open a PowerShell session and run the following cmdlet on the affected node: Taskkill /f /fi "SERVICES eq dnscache"
Deployment
In this release, there's a remote task failure on a multi-node deployment that results in the following exception: ECE RemoteTask orchestration failure with ASRR1N42R01U31 (node pingable - True): A WebException occurred while sending a RestRequest. WebException.Status: ConnectFailure on [https://<URL>](https://<URL>).
The mitigation is to restart the ECE agent on the affected node. On your machine, open a PowerShell session and run the following command: Restart-Service ECEAgent.
Add server
In this release and previous releases, when adding a machine to the system, is not possible to update the proxy bypass list string to include the new machine. Updating environment variables proxy bypass list on the hosts will not update the proxy bypass list on Azure Resource Bridge or AKS.
There's no workaround in this release. If you encounter this issue, contact Microsoft Support to determine next steps.
Add/Repair server
In this release, when adding or repairing a machine, a failure is seen when the software load balancer or network controller VM certificates are being copied from the existing nodes. The failure is because these certificates weren't generated during the deployment/update.
There's no workaround in this release. If you encounter this issue, contact Microsoft Support to determine next steps.
Deployment
In this release, there's a transient issue resulting in the deployment failure with the following exception: Type 'SyncDiagnosticLevel' of Role 'ObservabilityConfig' raised an exception:*<br>*Syncing Diagnostic Level failed with error: The Diagnostic Level does not match. Portal was not set to Enhanced, instead is Basic.
As this is a transient issue, retrying the deployment should fix this. For more information, see how to Rerun the deployment.
Deployment
In this release, there's an issue with the Secrets URI/location field. This is a required field that is marked Not mandatory and results in Azure Resource Manager template deployment failures.
Use the sample parameters file in the Deploy Azure Local via Azure Resource Manager template to ensure that all the inputs are provided in the required format and then try the deployment. If there's a failed deployment, you must also clean up the following resources before you Rerun the deployment: 1. Delete C:\EceStore. 2. Delete C:\CloudDeployment. 3. Delete C:\nugetstore. 4. Remove-Item HKLM:\Software\Microsoft\LCMAzureStackStampInformation.
Security
For new deployments, Secured-core capable devices won't have Dynamic Root of Measurement (DRTM) enabled by default. If you try to enable (DRTM) using the Enable-AzSSecurity cmdlet, you see an error that DRTM setting isn't supported in the current release. Microsoft recommends defense in depth, and UEFI Secure Boot still protects the components in the Static Root of Trust (SRT) boot chain by ensuring that they're loaded only when they're signed and verified.
DRTM isn't supported in this release.
Networking
An environment check fails when a proxy server is used. By design, the bypass list is different for winhttp and wininet, which causes the validation check to fail.
Follow these workaround steps:
1. Clear the proxy bypass list prior to the health check and before starting the deployment or the update.
2. After passing the check, wait for the deployment or update to fail.
3. Set your proxy bypass list again.
Arc VM management
Deployment or update of Arc Resource Bridge could fail when the automatically generated temporary SPN secret during this operation, starts with a hyphen.
Retry the deployment/update. The retry should regenerate the SPN secret and the operation will likely succeed.
Arc VM management
Arc Extensions on Arc VMs stay in "Creating" state indefinitely.
Sign in to the VM, open a command prompt, and type the following: Windows: notepad C:\ProgramData\AzureConnectedMachineAgent\Config\agentconfig.json Linux: sudo vi /var/opt/azcmagent/agentconfig.json Next, find the resourcename property. Delete the GUID that is appended to the end of the resource name, so this property matches the name of the VM. Then restart the VM.
Arc VM management
When a new machine is added to an Azure Local instance, storage path isn't created automatically for the newly created volume.
You can manually create a storage path for any new volumes. For more information, see Create a storage path.
Arc VM management
Restart of Arc VM operation completes after approximately 20 minutes although the VM itself restarts in about a minute.
There's no known workaround in this release.
Arc VM management
In some instances, the status of the logical network shows as Failed in Azure portal. This occurs when you try to delete the logical network without first deleting any resources such as network interfaces associated with that logical network. You should still be able to create resources on this logical network. The status is misleading in this instance.
If the status of this logical network was Succeeded at the time when this network was provisioned, then you can continue to create resources on this network.
Arc VM management
In this release, when you update a VM with a data disk attached to it using the Azure CLI, the operation fails with the following error message: Couldn't find a virtual hard disk with the name.
In rare instances, you may encounter this error while updating your Azure Local instance: Type 'UpdateArbAndExtensions' of Role 'MocArb' raised an exception: Exception Upgrading ARB and Extension in step [UpgradeArbAndExtensions :Get-ArcHciConfig] UpgradeArb: Invalid applianceyaml = [C:\AksHci\hci-appliance.yaml].
If you see this issue, contact Microsoft Support to assist you with the next steps.
Networking
There's an infrequent DNS client issue in this release that causes the deployment to fail on a two-node system with a DNS resolution error: A WebException occurred while sending a RestRequest. WebException.Status: NameResolutionFailure. As a result of the bug, the DNS record of the second node is deleted soon after it's created resulting in a DNS error.
Restart the machine. This operation registers the DNS record, which prevents it from getting deleted.
Azure portal
In some instances, the Azure portal might take a while to update and the view might not be current.
You might need to wait for 30 minutes or more to see the updated view.
Arc VM management
Deleting a network interface on an Arc VM from Azure portal doesn't work in this release.
Providing the OU name in an incorrect syntax isn't detected in the Azure portal. The incorrect syntax includes unsupported characters such as &,",',<,>. The incorrect syntax is detected at a later step during system validation.
Make sure that the OU path syntax is correct and doesn't include unsupported characters.
Deployment
Deployments via Azure Resource Manager time out after 2 hours. Deployments that exceed 2 hours show up as failed in the resource group though the system is successfully created.
To monitor the deployment in the Azure portal, go to the Azure Local instance resource and then go to new Deployments entry.
Azure Site Recovery
Azure Site Recovery can't be installed on an Azure Local instance in this release.
There's no known workaround in this release.
Update
When updating the Azure Local instance via the Azure Update Manager, the update progress and results may not be visible in the Azure portal.
To work around this issue, on each node, add the following registry key (no value needed):
Then on one of the nodes, restart the Cloud Management cluster group.
Stop-ClusterGroup "Cloud Management"
Start-ClusterGroup "Cloud Management"
This won't fully remediate the issue as the progress details may still not be displayed for a duration of the update process. To get the latest update details, you can Retrieve the update progress with PowerShell.
Update
In rare instances, if a failed update is stuck in an In progress state in Azure Update Manager, the Try again button is disabled.
To resume the update, run the following PowerShell command: Get-SolutionUpdate|Start-SolutionUpdate.
Update
In some cases, SolutionUpdate commands could fail if run after the Send-DiagnosticData command.
Make sure to close the PowerShell session used for Send-DiagnosticData. Open a new PowerShell session and use it for SolutionUpdate commands.
Update
In rare instances, when applying an update from 2311.0.24 to 2311.2.4, cluster status reports In Progress instead of expected Failed to update.
Retry the update. If the issue persists, contact Microsoft Support.
Update
Attempts to install solution updates can fail at the end of the CAU steps with: There was a failure in a Common Information Model (CIM) operation, that is, an operation performed by software that Cluster-Aware Updating depends on. This rare issue occurs if the Cluster Name or Cluster IP Address resources fail to start after a node reboot and is most typical in small deployments.
If you encounter this issue, contact Microsoft Support for next steps. They can work with you to manually restart the Azure Local resources and resume the update as needed.
Update
When applying a system update to 10.2402.3.11 the Get-SolutionUpdate cmdlet may not respond and eventually fails with a RequestTimeoutException after approximately 10 minutes. This is likely to occur following an add or repair server scenario.
Use the Start-ClusterGroup and Stop-ClusterGroup cmdlets to restart the update service.
A successful run of these cmdlets should bring the update service online.
Cluster aware updating
Resume node operation failed to resume node.
This is a transient issue and could resolve on its own. Wait for a few minutes and retry the operation. If the issue persists, contact Microsoft Support.
Cluster aware updating
Suspend node operation was stuck for greater than 90 minutes.
This is a transient issue and could resolve on its own. Wait for a few minutes and retry the operation. If the issue persists, contact Microsoft Support.
Known issues for version 2408
This software release maps to software version number 2408.0.29.
Release notes for this version include the issues fixed in this release, known issues in this release, and release note issues carried over from previous versions.
An update issue related to missing resource type ID field in the health checks was fixed.
Updates
An update issue related to different health checks having the same name was fixed.
Arc VM management
In large deployment scenarios, such as extensive AVD host pool deployments or large-scale VM provisioning, you might encounter reliability issues caused by a Hyper-V socket external library problem.
Known issues in this release
The following table lists the known issues in this release:
Feature
Issue
Workaround
Repair server
After you repair a node and run the command Set-AzureStackLCMUserPassword, you may encounter the following error:
CloudEngine.Actions.InterfaceInvocationFailedException: Type 'ValidateCredentials' of Role 'SecretRotation' raised an exception: Cannot load encryption certificate. The certificate setting 'CN=DscEncryptionCert' does not represent a valid base-64 encoded certificate, nor does it represent a valid certificate by file, directory, thumbprint, or subject name. at Validate-Credentials
Follow these steps to mitigate the issue:
$NewPassword = <Provide new password as secure string>
$OldPassword = <Provide the old/current password as secure string>
if ($eceClusterGroup.State -ne "Online") {Write-AzsSecurityError -Message "ECE cluster group is not in an Online state. Cannot continue with password rotation." -ErrRecord $_}
3. Update the ECE with the new password:
Write-AzsSecurityVerbose -Message "Updating password in ECE" -Verbose
Using an exported Azure VM OS disk as a VHD to create a gallery image for provisioning an Arc VM is unsupported.
Run the command restart-service mochostagent to restart the mochostagent service.
Arc VM management
If you try to enable guest management on a migrated VM, the operation fails with the following error: (InternalError) admission webhook "createupdatevalidationwebhook.infrastructure.azstackhci.microsoft.com" denied the request: OsProfile cannot be changed after resource creation
Networking
When a node is configured with a proxy server that has capital letters in its address, such as HTTPS://10.100.000.00:8080, Arc extensions fail to install or update on the node in existing builds, including version 2408. However, the node remains Arc connected.
Follow these steps to mitigate the issue:
1. Set the environment values in lowercase. [System.Environment]::SetEnvironmentVariable("HTTPS_PROXY", "https://10.100.000.00:8080", "Machine").
2. Validate that the values were set. [System.Environment]::GetEnvironmentVariable("HTTPS_PROXY", "Machine").
3. Restart Arc services.
Restart-Service himds
Restart-Service ExtensionService
Restart-Service GCArcService
4. Signal the AzcmaAgent with the lowercase proxy information.
& 'C:\Program Files\AzureConnectedMachineAgent\azcmagent.exe' config set proxy.url https://10.100.000.00:8080
& 'C:\Program Files\AzureConnectedMachineAgent\azcmagent.exe' config list
Networking
When Arc machines go down, the "All Clusters" page, in the new portal experience shows a "PartiallyConnected" or "Not Connected Recently status. Even when the Arc machines become healthy, they may not show a "Connected" status.
There's no known workaround for this issue. To check the connectivity status, use the old experience to see if it shows as "Connected".
Security
The SideChannelMitigation security feature may not show an enabled state even if it's enabled. This happens when using Windows Admin Center (Cluster Security View) or when this cmdlet returns False: Get-AzSSecurity -FeatureName SideChannelMitigation.
There's no workaround in this release to fix the output of these applications. To validate the expected value, run the following cmdlet: Get-ItemProperty 'HKLM:\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management' -name "FeatureSettingsOverride*" The expected output is: FeatureSettingsOverride: 83886152 FeatureSettingsOverrideMask: 3 If your output matches the expected output, you can safely ignore the output from Windows Admin Center and Get-AzSSecurity cmdlet.
Arc VM management
The Mochostagent service might appear to be running but can get stuck without updating logs for over a month. You can identify this issue by checking the service logs in C:\programdata\mochostagent\logs to see if logs are being updated.
Run the following command to restart the mochostagent service: restart-service mochostagent.
Upgrade
When upgrading the stamp from 2311 or prior builds to 2408 or later, add node and repair node operations may fail. For example, you could see an error: Type 'AddAsZHostToDomain' of Role 'BareMetal' raised an exception.
There's no workaround in this release. If you encounter this issue, contact Microsoft Support to determine next steps.
Update
When installing an SBE update for your Azure Local system, some SBE interfaces aren't executed on all the machines if the hostname in the cluster is a subset of another hostname. For example, host-1 is a subset of host-10. This could result in failures in the CAU scan or CAU run.
Microsoft recommends using at least 2 digits for hostname instance counts in your host naming conventions. For more information, see Define your naming convention.
Known issues from previous releases
The following table lists the known issues from previous releases:
Feature
Issue
Workaround
Update
When viewing the readiness check results for an Azure Local instance via the Azure Update Manager, there might be multiple readiness checks with the same name.
There's no known workaround in this release. Select View details to view specific information about the readiness check.
Deployment
In some instances, during the registration of Azure Local machines, this error might be seen in the debug logs: Encountered internal server error. One of the mandatory extensions for device deployment might not be installed.
There's an intermittent issue in this release when the Azure portal incorrectly reports the update status as Failed to update or In progress though the update is complete.
$Update = get-solutionupdate| ? version -eq "<version string>"
Replace the version string with the version you're running. For example, "10.2405.0.23".
$Update.state
If the update status is Installed, no further action is required on your part. Azure portal refreshes the status correctly within 24 hours. To refresh the status sooner, follow these steps on one of the cluster nodes. Restart the Cloud Management cluster group. Stop-ClusterGroup "Cloud Management" Start-ClusterGroup "Cloud Management"
Update
During an initial MOC update, a failure occurs due to the target MOC version not being found in the catalog cache. The follow-up updates and retries show MOC in the target version, without the update succeeding, and as a result the Arc Resource Bridge update fails.
To validate this issue, collect the update logs using Troubleshoot solution updates for Azure Local. The log files should show a similar error message (current version might differ in the error message):
[ERROR: { "errorCode": "InvalidEntityError", "errorResponse": "{\n\"message\": \"the cloud fabric (MOC) is currently at version v0.13.1. A minimum version of 0.15.0 is required for compatibility\"\n}" }]
Follow these steps to mitigate the issue:
1. To find the MOC agent version, run the following command: 'C:\Program Files\AksHci\wssdcloudagent.exe' version.
2. Use the output of the command to find the MOC version from the table below that matches the agent version, and set $initialMocVersion to that MOC version. Set the $targetMocVersion by finding the Azure Local build you're updating to and get the matching MOC version from the following table. Use these values in the mitigation script provided below:
Build
MOC version
Agent version
2311.2
1.0.24.10106
v0.13.0-6-gf13a73f7, v0.11.0-alpha.38,01/06/2024
2402
1.0.25.10203
v0.14.0, v0.13.1, 02/02/2024
2402.1
1.0.25.10302
v0.14.0, v0.13.1, 03/02/2024
2402.2
1.1.1.10314
v0.16.0-1-g04bf0dec, v0.15.1, 03/14/2024
2405/2402.3
1.3.0.10418
v0.17.1, v0.16.5, 04/18/2024
For example, if the agent version is v0.13.0-6-gf13a73f7, v0.11.0-alpha.38,01/06/2024, then $initialMocVersion = "1.0.24.10106" and if you are updating to 2405.0.23, then $targetMocVersion = "1.3.0.10418".
3. Run the following PowerShell commands on the first node:
$initialMocVersion = "<initial version determined from step 2>" $targetMocVersion = "<target version determined from step 2>"
# Import MOC module twice import-module moc import-module moc $verbosePreference = "Continue"
# Clear the SFS catalog cache Remove-Item (Get-MocConfig).manifestCache
# Set version to the current MOC version prior to update, and set state as update failed Set-MocConfigValue -name "version" -value $initialMocVersion Set-MocConfigValue -name "installState" -value ([InstallState]::UpdateFailed)
# Rerun the MOC update to desired version Update-Moc -version $targetMocVersion
4. Resume the update.
AKS on HCI
AKS cluster creation fails with the Error: Invalid AKS network resource id. This issue can occur when the associated logical network name has an underscore.
Underscores aren't supported in logical network names. Make sure to not use underscore in the names for logical networks deployed on your Azure Local instance.
Repair server
In rare instances, the Repair-Server operation fails with the HealthServiceWaitForDriveFW error. In these cases, the old drives from the repaired node aren't removed and new disks are stuck in the maintenance mode.
To prevent this issue, make sure that you DO NOT drain the node either via the Windows Admin Center or using the Suspend-ClusterNode -Drain PowerShell cmdlet before you start Repair-Server. If the issue occurs, contact Microsoft Support for next steps.
Repair server
This issue is seen when the single node Azure Local instance is updated from 2311 to 2402 and then the Repair-Server is performed. The repair operation fails.
Before you repair the single node, follow these steps: 1. Run version 2402 for the ADPrepTool. Follow the steps in Prepare Active Directory. This action is quick and adds the required permissions to the Organizational Unit (OU). 2. Move the computer object from Computers segment to the root OU. Run the following command: Get-ADComputer <HOSTNAME> | Move-ADObject -TargetPath "<OU path>"
Deployment
If you prepare the Active Directory on your own (not using the script and procedure provided by Microsoft), your Active Directory validation could fail with missing Generic All permission. This is due to an issue in the validation check that checks for a dedicated permission entry for msFVE-RecoverInformationobjects – General – Permissions Full control, which is required for BitLocker recovery.
Use the Prepare AD script method or if using your own method, make sure to assign the specific permission msFVE-RecoverInformationobjects – General – Permissions Full control.
Deployment
There's a rare issue in this release where the DNS record is deleted during the Azure Local deployment. When that occurs, the following exception is seen: Type 'PropagatePublicRootCertificate' of Role 'ASCA' raised an exception:<br>The operation on computer 'ASB88RQ22U09' failed: WinRM cannot process the request. The following error occurred while using Kerberos authentication: Cannot find the computer ASB88RQ22U09.local. Verify that the computer exists on the network and that the name provided is spelled correctly at PropagatePublicRootCertificate, C:\NugetStore\Microsoft.AzureStack, at Orchestration.Roles.CertificateAuthority.10.2402.0.14\content\Classes\ASCA\ASCA.psm1: line 38, at C:\CloudDeployment\ECEngine\InvokeInterfaceInternal.psm1: line 127,at Invoke-EceInterfaceInternal, C:\CloudDeployment\ECEngine\InvokeInterfaceInternal.psm1: line 123.
Check the DNS server to see if any DNS records of the cluster nodes are missing. Apply the following mitigation on the nodes where its DNS record is missing.
Restart the DNS client service. Open a PowerShell session and run the following cmdlet on the affected node: Taskkill /f /fi "SERVICES eq dnscache"
Deployment
In this release, there's a remote task failure on a multi-node deployment that results in the following exception: ECE RemoteTask orchestration failure with ASRR1N42R01U31 (node pingable - True): A WebException occurred while sending a RestRequest. WebException.Status: ConnectFailure on [https://<URL>](https://<URL>).
The mitigation is to restart the ECE agent on the affected node. On your machine, open a PowerShell session and run the following command: Restart-Service ECEAgent.
Add server
In this release and previous releases, when adding a machine to the cluster, is not possible to update the proxy bypass list string to include the new machine. Updating environment variables proxy bypass list on the hosts will not update the proxy bypass list on Azure Resource Bridge or AKS.
There's no workaround in this release. If you encounter this issue, contact Microsoft Support to determine next steps.
Add/Repair server
In this release, when adding or repairing a machine, a failure is seen when the software load balancer or network controller VM certificates are being copied from the existing nodes. The failure is because these certificates weren't generated during the deployment/update.
There's no workaround in this release. If you encounter this issue, contact Microsoft Support to determine next steps.
Deployment
In this release, there's a transient issue resulting in the deployment failure with the following exception: Type 'SyncDiagnosticLevel' of Role 'ObservabilityConfig' raised an exception:*<br>*Syncing Diagnostic Level failed with error: The Diagnostic Level does not match. Portal was not set to Enhanced, instead is Basic.
As this is a transient issue, retrying the deployment should fix this. For more information, see how to Rerun the deployment.
Deployment
In this release, there's an issue with the Secrets URI/location field. This is a required field that is marked Not mandatory and results in Azure Resource Manager template deployment failures.
Use the sample parameters file in the Deploy Azure Local via Azure Resource Manager template to ensure that all the inputs are provided in the required format and then try the deployment. If there's a failed deployment, you must also clean up the following resources before you Rerun the deployment: 1. Delete C:\EceStore. 2. Delete C:\CloudDeployment. 3. Delete C:\nugetstore. 4. Remove-Item HKLM:\Software\Microsoft\LCMAzureStackStampInformation.
Security
For new deployments, Secured-core capable devices won't have Dynamic Root of Measurement (DRTM) enabled by default. If you try to enable (DRTM) using the Enable-AzSSecurity cmdlet, you see an error that DRTM setting isn't supported in the current release. Microsoft recommends defense in depth, and UEFI Secure Boot still protects the components in the Static Root of Trust (SRT) boot chain by ensuring that they're loaded only when they're signed and verified.
DRTM isn't supported in this release.
Networking
An environment check fails when a proxy server is used. By design, the bypass list is different for winhttp and wininet, which causes the validation check to fail.
Follow these workaround steps:
1. Clear the proxy bypass list prior to the health check and before starting the deployment or the update.
2. After passing the check, wait for the deployment or update to fail.
3. Set your proxy bypass list again.
Arc VM management
Deployment or update of Arc Resource Bridge could fail when the automatically generated temporary SPN secret during this operation, starts with a hyphen.
Retry the deployment/update. The retry should regenerate the SPN secret and the operation will likely succeed.
Arc VM management
Arc Extensions on Arc VMs stay in "Creating" state indefinitely.
Sign in to the VM, open a command prompt, and type the following: Windows: notepad C:\ProgramData\AzureConnectedMachineAgent\Config\agentconfig.json Linux: sudo vi /var/opt/azcmagent/agentconfig.json Next, find the resourcename property. Delete the GUID that is appended to the end of the resource name, so this property matches the name of the VM. Then restart the VM.
Arc VM management
When a new machine is added to an Azure Local instance, storage path isn't created automatically for the newly created volume.
You can manually create a storage path for any new volumes. For more information, see Create a storage path.
Arc VM management
Restart of Arc VM operation completes after approximately 20 minutes although the VM itself restarts in about a minute.
There's no known workaround in this release.
Arc VM management
In some instances, the status of the logical network shows as Failed in Azure portal. This occurs when you try to delete the logical network without first deleting any resources such as network interfaces associated with that logical network. You should still be able to create resources on this logical network. The status is misleading in this instance.
If the status of this logical network was Succeeded at the time when this network was provisioned, then you can continue to create resources on this network.
Arc VM management
In this release, when you update a VM with a data disk attached to it using the Azure CLI, the operation fails with the following error message: Couldn't find a virtual hard disk with the name.
In rare instances, you may encounter this error while updating your Azure Local instance: Type 'UpdateArbAndExtensions' of Role 'MocArb' raised an exception: Exception Upgrading ARB and Extension in step [UpgradeArbAndExtensions :Get-ArcHciConfig] UpgradeArb: Invalid applianceyaml = [C:\AksHci\hci-appliance.yaml].
If you see this issue, contact Microsoft Support to assist you with the next steps.
Networking
There's an infrequent DNS client issue in this release that causes the deployment to fail on a two-node cluster with a DNS resolution error: A WebException occurred while sending a RestRequest. WebException.Status: NameResolutionFailure. As a result of the bug, the DNS record of the second node is deleted soon after it's created resulting in a DNS error.
Restart the machine. This operation registers the DNS record, which prevents it from getting deleted.
Azure portal
In some instances, the Azure portal might take a while to update and the view might not be current.
You might need to wait for 30 minutes or more to see the updated view.
Arc VM management
Deleting a network interface on an Arc VM from Azure portal doesn't work in this release.
Providing the OU name in an incorrect syntax isn't detected in the Azure portal. The incorrect syntax includes unsupported characters such as &,",',<,>. The incorrect syntax is detected at a later step during cluster validation.
Make sure that the OU path syntax is correct and doesn't include unsupported characters.
Deployment
Deployments via Azure Resource Manager time out after 2 hours. Deployments that exceed 2 hours show up as failed in the resource group though the cluster is successfully created.
To monitor the deployment in the Azure portal, go to the Azure Local instance resource and then go to new Deployments entry.
Azure Site Recovery
Azure Site Recovery can't be installed on an Azure Local instance in this release.
There's no known workaround in this release.
Update
When updating the Azure Local instance via the Azure Update Manager, the update progress and results may not be visible in the Azure portal.
To work around this issue, on each cluster node, add the following registry key (no value needed):
Then on one of the cluster nodes, restart the Cloud Management cluster group.
Stop-ClusterGroup "Cloud Management"
Start-ClusterGroup "Cloud Management"
This won't fully remediate the issue as the progress details may still not be displayed for a duration of the update process. To get the latest update details, you can Retrieve the update progress with PowerShell.
Update
In rare instances, if a failed update is stuck in an In progress state in Azure Update Manager, the Try again button is disabled.
To resume the update, run the following PowerShell command: Get-SolutionUpdate|Start-SolutionUpdate.
Updates
In some cases, SolutionUpdate commands could fail if run after the Send-DiagnosticData command.
Make sure to close the PowerShell session used for Send-DiagnosticData. Open a new PowerShell session and use it for SolutionUpdate commands.
Update
In rare instances, when applying an update from 2311.0.24 to 2311.2.4, cluster status reports In Progress instead of expected Failed to update.
Retry the update. If the issue persists, contact Microsoft Support.
Update
Attempts to install solution updates can fail at the end of the CAU steps with: There was a failure in a Common Information Model (CIM) operation, that is, an operation performed by software that Cluster-Aware Updating depends on. This rare issue occurs if the Cluster Name or Cluster IP Address resources fail to start after a node reboot and is most typical in small clusters.
If you encounter this issue, contact Microsoft Support for next steps. They can work with you to manually restart the cluster resources and resume the update as needed.
Update
When applying a cluster update to 10.2402.3.11 the Get-SolutionUpdate cmdlet may not respond and eventually fails with a RequestTimeoutException after approximately 10 minutes. This is likely to occur following an add or repair server scenario.
Use the Start-ClusterGroup and Stop-ClusterGroup cmdlets to restart the update service.
A successful run of these cmdlets should bring the update service online.
Cluster aware updating
Resume node operation failed to resume node.
This is a transient issue and could resolve on its own. Wait for a few minutes and retry the operation. If the issue persists, contact Microsoft Support.
Cluster aware updating
Suspend node operation was stuck for greater than 90 minutes.
This is a transient issue and could resolve on its own. Wait for a few minutes and retry the operation. If the issue persists, contact Microsoft Support.
Issues for version 2405.3
This software release maps to software version number 2405.3.7.
Release notes for this version include the issues fixed in this release, known issues in this release, and release noted issues carried over from previous versions.
Fixed issues
Here are the fixed issues in this release:
Feature
Issue
Workaround/Comments
Update
In this release, an update issue related to SDN not working once the hosts go through the secret rotation and update, was fixed.
Update
In this release, an update issue related to the Physical Disks environment readiness check incorrectly failing and blocking the update, was fixed
Deployment
In this release, a deployment operation related to null value in cloud deployment, was fixed.
Update
In this release, a health check update to prevent a Summary XML error was fixed.
Known issues in this release
Microsoft isn't aware of any known issues in this release.
Known issues from previous releases
Here are the known issues from previous releases:
Feature
Issue
Workaround
Update
When viewing the readiness check results for an Azure Stack HCI cluster via the Azure Update Manager, there might be multiple readiness checks with the same name.
There's no known workaround in this release. Select View details to view specific information about the readiness check.
Arc VM management
In large deployment scenarios, such as extensive AVD host pool deployments or large-scale VM provisioning, you might encounter reliability issues caused by a Hyper-V socket external library problem.
Follow these steps to mitigate the issue: 1. Run the command Get-service mochostagent (\) get-process (\) kill. Check the output of the command and verify if the handle count is in the thousands.
2. Run the command Get-service mochostagent (\) get-process to terminate the processes.
3. Run the command restart-service mochostagent to restart the mochostagent service.
Deployment
When deploying Azure Local via the Azure portal, you might encounter the following deployment validation failure:
Could not complete the operation. 400: Resource creation validation failed. Details: [{"Code":"AnswerFileValidationFailed","Message":"Errors in Value Validation:\r\nPhysicalNodesValidator found error at deploymentdata.physicalnodes[0].ipv4address: The specified for \u0027deploymentdata.physicalnodes[0].ipv4address\u0027 is not a valid IPv4 address. Example: 192.168.0.1 or 192.168.0.1","Target":null,"Details":null}].
If you go to the Networking tab in Azure portal deployment, within the Network Intent configuration, you could see the following error: The selected physical network adapter is not binded to the management virtual switch.
The deployment via the Azure portal fails with this error: Failed to fetch secret LocalAdminCredential from key vault.
There is no workaround for this issue in this release. If the issue occurs, contact Microsoft Support for next steps.
Deployment
In some instances, during the registration of Azure Stack HCI servers, this error might be seen in the debug logs: Encountered internal server error. One of the mandatory extensions for device deployment might not be installed.
There's an intermittent issue in this release when the Azure portal incorrectly reports the update status as Failed to update or In progress though the update is complete.
Connect to your Azure Local via a remote PowerShell session. To confirm the update status, run the following PowerShell cmdlets:
$Update = get-solutionupdate| ? version -eq "<version string>"
Replace the version string with the version you're running. For example, "10.2405.0.23".
$Update.state
If the update status is Installed, no further action is required on your part. Azure portal refreshes the status correctly within 24 hours. To refresh the status sooner, follow these steps on one of the cluster nodes. Restart the Cloud Management cluster group. Stop-ClusterGroup "Cloud Management" Start-ClusterGroup "Cloud Management"
Update
During an initial MOC update, a failure occurs due to the target MOC version not being found in the catalog cache. The follow-up updates and retries show MOC in the target version, without the update succeeding, and as a result the Arc Resource Bridge update fails.
To validate this issue, collect the update logs using Troubleshoot solution updates for Azure Local. The log files should show a similar error message (current version might differ in the error message):
[ERROR: { "errorCode": "InvalidEntityError", "errorResponse": "{\n\"message\": \"the cloud fabric (MOC) is currently at version v0.13.1. A minimum version of 0.15.0 is required for compatibility\"\n}" }]
Follow these steps to mitigate the issue:
1. To find the MOC agent version, run the following command: 'C:\Program Files\AksHci\wssdcloudagent.exe' version.
2. Use the output of the command to find the MOC version from the table below that matches the agent version, and set $initialMocVersion to that MOC version. Set the $targetMocVersion by finding the Azure Stack HCI build you're updating to and get the matching MOC version from the following table. Use these values in the mitigation script provided below:
Build
MOC version
Agent version
2311.2
1.0.24.10106
v0.13.0-6-gf13a73f7, v0.11.0-alpha.38,01/06/2024
2402
1.0.25.10203
v0.14.0, v0.13.1, 02/02/2024
2402.1
1.0.25.10302
v0.14.0, v0.13.1, 03/02/2024
2402.2
1.1.1.10314
v0.16.0-1-g04bf0dec, v0.15.1, 03/14/2024
2405/2402.3
1.3.0.10418
v0.17.1, v0.16.5, 04/18/2024
For example, if the agent version is v0.13.0-6-gf13a73f7, v0.11.0-alpha.38,01/06/2024, then $initialMocVersion = “1.0.24.10106” and if you are updating to 2405.0.23, then $targetMocVersion = “1.3.0.10418”.
3. Run the following PowerShell commands on the first node:
$initialMocVersion = "<initial version determined from step 2>" $targetMocVersion = "<target version determined from step 2>"
# Import MOC module twice import-module moc import-module moc $verbosePreference = "Continue"
# Clear the SFS catalog cache Remove-Item (Get-MocConfig).manifestCache
# Set version to the current MOC version prior to update, and set state as update failed Set-MocConfigValue -name "version" -value $initialMocVersion Set-MocConfigValue -name "installState" -value ([InstallState]::UpdateFailed)
# Rerun the MOC update to desired version Update-Moc -version $targetMocVersion
4. Resume the update.
AKS on HCI
AKS cluster creation fails with the Error: Invalid AKS network resource id. This issue can occur when the associated logical network name has an underscore.
Underscores aren't supported in logical network names. Make sure to not use underscore in the names for logical networks deployed on your Azure Stack HCI.
Repair server
In rare instances, the Repair-Server operation fails with the HealthServiceWaitForDriveFW error. In these cases, the old drives from the repaired node aren't removed and new disks are stuck in the maintenance mode.
To prevent this issue, make sure that you DO NOT drain the node either via the Windows Admin Center or using the Suspend-ClusterNode -Drain PowerShell cmdlet before you start Repair-Server. If the issue occurs, contact Microsoft Support for next steps.
Repair server
This issue is seen when the single server Azure Stack HCI is updated from 2311 to 2402 and then the Repair-Server is performed. The repair operation fails.
Before you repair the single node, follow these steps: 1. Run version 2402 for the ADPrepTool. Follow the steps in Prepare Active Directory. This action is quick and adds the required permissions to the Organizational Unit (OU). 2. Move the computer object from Computers segment to the root OU. Run the following command: Get-ADComputer <HOSTNAME> | Move-ADObject -TargetPath "<OU path>"
Deployment
If you prepare the Active Directory on your own (not using the script and procedure provided by Microsoft), your Active Directory validation could fail with missing Generic All permission. This is due to an issue in the validation check that checks for a dedicated permission entry for msFVE-RecoverInformationobjects – General – Permissions Full control, which is required for BitLocker recovery.
Use the Prepare AD script method or if using your own method, make sure to assign the specific permission msFVE-RecoverInformationobjects – General – Permissions Full control.
Deployment
There's a rare issue in this release where the DNS record is deleted during the Azure Stack HCI deployment. When that occurs, the following exception is seen: Type 'PropagatePublicRootCertificate' of Role 'ASCA' raised an exception:<br>The operation on computer 'ASB88RQ22U09' failed: WinRM cannot process the request. The following error occurred while using Kerberos authentication: Cannot find the computer ASB88RQ22U09.local. Verify that the computer exists on the network and that the name provided is spelled correctly at PropagatePublicRootCertificate, C:\NugetStore\Microsoft.AzureStack, at Orchestration.Roles.CertificateAuthority.10.2402.0.14\content\Classes\ASCA\ASCA.psm1: line 38, at C:\CloudDeployment\ECEngine\InvokeInterfaceInternal.psm1: line 127,at Invoke-EceInterfaceInternal, C:\CloudDeployment\ECEngine\InvokeInterfaceInternal.psm1: line 123.
Check the DNS server to see if any DNS records of the cluster nodes are missing. Apply the following mitigation on the nodes where its DNS record is missing.
Restart the DNS client service. Open a PowerShell session and run the following cmdlet on the affected node: Taskkill /f /fi "SERVICES eq dnscache"
Deployment
In this release, there's a remote task failure on a multi-node deployment that results in the following exception: ECE RemoteTask orchestration failure with ASRR1N42R01U31 (node pingable - True): A WebException occurred while sending a RestRequest. WebException.Status: ConnectFailure on [https://<URL>](https://<URL>).
The mitigation is to restart the ECE agent on the affected node. On your server, open a PowerShell session and run the following command: Restart-Service ECEAgent.
Add server
In this release and previous releases, when adding a server to the cluster, is not possible to update the proxy bypass list string to include the new server. Updating environment variables proxy bypass list on the hosts will not update the proxy bypass list on Azure Resource Bridge or AKS.
There's no workaround in this release. If you encounter this issue, contact Microsoft Support to determine next steps.
Add/Repair server
In this release, when adding or repairing a server, a failure is seen when the software load balancer or network controller VM certificates are being copied from the existing nodes. The failure is because these certificates weren't generated during the deployment/update.
There's no workaround in this release. If you encounter this issue, contact Microsoft Support to determine next steps.
Deployment
In this release, there's a transient issue resulting in the deployment failure with the following exception: Type 'SyncDiagnosticLevel' of Role 'ObservabilityConfig' raised an exception:*<br>*Syncing Diagnostic Level failed with error: The Diagnostic Level does not match. Portal was not set to Enhanced, instead is Basic.
As this is a transient issue, retrying the deployment should fix this. For more information, see how to Rerun the deployment.
Deployment
In this release, there's an issue with the Secrets URI/location field. This is a required field that is marked Not mandatory and results in Azure Resource Manager template deployment failures.
Use the sample parameters file in the Deploy Azure Local via Azure Resource Manager template to ensure that all the inputs are provided in the required format and then try the deployment. If there's a failed deployment, you must also clean up the following resources before you Rerun the deployment: 1. Delete C:\EceStore. 2. Delete C:\CloudDeployment. 3. Delete C:\nugetstore. 4. Remove-Item HKLM:\Software\Microsoft\LCMAzureStackStampInformation.
Security
For new deployments, Secured-core capable devices won't have Dynamic Root of Measurement (DRTM) enabled by default. If you try to enable (DRTM) using the Enable-AzSSecurity cmdlet, you see an error that DRTM setting isn't supported in the current release. Microsoft recommends defense in depth, and UEFI Secure Boot still protects the components in the Static Root of Trust (SRT) boot chain by ensuring that they're loaded only when they're signed and verified.
DRTM isn't supported in this release.
Networking
An environment check fails when a proxy server is used. By design, the bypass list is different for winhttp and wininet, which causes the validation check to fail.
Follow these workaround steps:
1. Clear the proxy bypass list prior to the health check and before starting the deployment or the update.
2. After passing the check, wait for the deployment or update to fail.
3. Set your proxy bypass list again.
Arc VM management
Deployment or update of Arc Resource Bridge could fail when the automatically generated temporary SPN secret during this operation, starts with a hyphen.
Retry the deployment/update. The retry should regenerate the SPN secret and the operation will likely succeed.
Arc VM management
Arc Extensions on Arc VMs stay in "Creating" state indefinitely.
Sign in to the VM, open a command prompt, and type the following: Windows: notepad C:\ProgramData\AzureConnectedMachineAgent\Config\agentconfig.json Linux: sudo vi /var/opt/azcmagent/agentconfig.json Next, find the resourcename property. Delete the GUID that is appended to the end of the resource name, so this property matches the name of the VM. Then restart the VM.
Arc VM management
When a new server is added to an Azure Stack HCI cluster, storage path isn't created automatically for the newly created volume.
You can manually create a storage path for any new volumes. For more information, see Create a storage path.
Arc VM management
Restart of Arc VM operation completes after approximately 20 minutes although the VM itself restarts in about a minute.
There's no known workaround in this release.
Arc VM management
In some instances, the status of the logical network shows as Failed in Azure portal. This occurs when you try to delete the logical network without first deleting any resources such as network interfaces associated with that logical network. You should still be able to create resources on this logical network. The status is misleading in this instance.
If the status of this logical network was Succeeded at the time when this network was provisioned, then you can continue to create resources on this network.
Arc VM management
In this release, when you update a VM with a data disk attached to it using the Azure CLI, the operation fails with the following error message: Couldn't find a virtual hard disk with the name.
In rare instances, you may encounter this error while updating your Azure Stack HCI: Type 'UpdateArbAndExtensions' of Role 'MocArb' raised an exception: Exception Upgrading ARB and Extension in step [UpgradeArbAndExtensions :Get-ArcHciConfig] UpgradeArb: Invalid applianceyaml = [C:\AksHci\hci-appliance.yaml].
If you see this issue, contact Microsoft Support to assist you with the next steps.
Networking
There's an infrequent DNS client issue in this release that causes the deployment to fail on a two-node cluster with a DNS resolution error: A WebException occurred while sending a RestRequest. WebException.Status: NameResolutionFailure. As a result of the bug, the DNS record of the second node is deleted soon after it's created resulting in a DNS error.
Restart the server. This operation registers the DNS record, which prevents it from getting deleted.
Azure portal
In some instances, the Azure portal might take a while to update and the view might not be current.
You might need to wait for 30 minutes or more to see the updated view.
Arc VM management
Deleting a network interface on an Arc VM from Azure portal doesn't work in this release.
Providing the OU name in an incorrect syntax isn't detected in the Azure portal. The incorrect syntax includes unsupported characters such as &,",',<,>. The incorrect syntax is detected at a later step during cluster validation.
Make sure that the OU path syntax is correct and doesn't include unsupported characters.
Deployment
Deployments via Azure Resource Manager time out after 2 hours. Deployments that exceed 2 hours show up as failed in the resource group though the cluster is successfully created.
To monitor the deployment in the Azure portal, go to the Azure Stack HCI cluster resource and then go to new Deployments entry.
Azure Site Recovery
Azure Site Recovery can't be installed on an Azure Stack HCI cluster in this release.
There's no known workaround in this release.
Update
When updating the Azure Stack HCI cluster via the Azure Update Manager, the update progress and results may not be visible in the Azure portal.
To work around this issue, on each cluster node, add the following registry key (no value needed):
Then on one of the cluster nodes, restart the Cloud Management cluster group.
Stop-ClusterGroup "Cloud Management"
Start-ClusterGroup "Cloud Management"
This won't fully remediate the issue as the progress details may still not be displayed for a duration of the update process. To get the latest update details, you can Retrieve the update progress with PowerShell.
Update
In rare instances, if a failed update is stuck in an In progress state in Azure Update Manager, the Try again button is disabled.
To resume the update, run the following PowerShell command: Get-SolutionUpdate|Start-SolutionUpdate.
Updates
In some cases, SolutionUpdate commands could fail if run after the Send-DiagnosticData command.
Make sure to close the PowerShell session used for Send-DiagnosticData. Open a new PowerShell session and use it for SolutionUpdate commands.
Update
In rare instances, when applying an update from 2311.0.24 to 2311.2.4, cluster status reports In Progress instead of expected Failed to update.
Retry the update. If the issue persists, contact Microsoft Support.
Update
Attempts to install solution updates can fail at the end of the CAU steps with: There was a failure in a Common Information Model (CIM) operation, that is, an operation performed by software that Cluster-Aware Updating depends on. This rare issue occurs if the Cluster Name or Cluster IP Address resources fail to start after a node reboot and is most typical in small clusters.
If you encounter this issue, contact Microsoft Support for next steps. They can work with you to manually restart the cluster resources and resume the update as needed.
Update
When applying a cluster update to 10.2402.3.11 the Get-SolutionUpdate cmdlet may not respond and eventually fails with a RequestTimeoutException after approximately 10 minutes. This is likely to occur following an add or repair server scenario.
Use the Start-ClusterGroup and Stop-ClusterGroup cmdlets to restart the update service.
A successful run of these cmdlets should bring the update service online.
Cluster aware updating
Resume node operation failed to resume node.
This is a transient issue and could resolve on its own. Wait for a few minutes and retry the operation. If the issue persists, contact Microsoft Support.
Cluster aware updating
Suspend node operation was stuck for greater than 90 minutes.
This is a transient issue and could resolve on its own. Wait for a few minutes and retry the operation. If the issue persists, contact Microsoft Support.
Issues for version 2405.2
This software release maps to software version number 2405.2.7.
Release notes for this version include the issues fixed in this release, known issues in this release, and release noted issues carried over from previous versions.
Fixed issues
Here are the fixed issues in this release:
Feature
Issue
Workaround/Comments
Updates
In this release, an update issue related to missing resource type ID field in the health checks, was fixed.
Updates
In this release, an update issue related to different health checks having the same name, was fixed.
Updates
In this release, an issue where Solution Builder Extension Update health checks were missing from the pre-update or daily health checks, was fixed.
Updates
In this release, an issue that caused an inability to view or start new updates due to the update service crashing on servers in a bad state, was fixed.
Updates
In this release, the update service was improved to prevent flooding of actions on the cluster.
Updates
In this release, a health check was added to prevent updates when adding or removing servers fails.
Arc VM management
In earlier releases, any power state change operation of a VM such as start stop, save, and pause, would initially return the state of the VM as running and eventually display the correct state after a refresh 30+ seconds later. In this release, the power state change operation only returns after the VM state is changed to the expected one.
Known issues in this release
Feature
Issue
Workaround
Update
Owing to a bug in SDN infrastructure VMs, SDN stops working once the hosts go through the secret rotation and update.
There is no workaround for this issue in this release. If the issue occurs, contact Microsoft Support for next steps.
Update
Owing to a bug in Environment readiness checker, the Physical Disks environment readiness check incorrectly fails and blocks the update.
Wait for a few minutes and retry the update.
Deployment
In this release, you may receive the following error: Invoke Cloud Deploy Failed With - Value cannot be null.
There is no workaround for this issue in this release. If the issue occurs, contact Microsoft Support for next steps.
Update
In this release, an environment check fails with the following error: Update is in Failed state: HealthCheckFailed. Summary XML from ECE not present.
There is no workaround for this issue in this release. If the issue occurs, contact Microsoft Support for next steps.
Known issues from previous releases
Here are the known issues from previous releases:
Feature
Issue
Workaround
Update
When viewing the readiness check results for an Azure Stack HCI cluster via the Azure Update Manager, there might be multiple readiness checks with the same name.
There's no known workaround in this release. Select View details to view specific information about the readiness check.
Arc VM management
In large deployment scenarios, such as extensive AVD host pool deployments or large-scale VM provisioning, you might encounter reliability issues caused by a Hyper-V socket external library problem.
Follow these steps to mitigate the issue: 1. Run the command Get-service mochostagent (\) get-process (\) kill. Check the output of the command and verify if the handle count is in the thousands.
2. Run the command Get-service mochostagent (\) get-process to terminate the processes.
3. Run the command restart-service mochostagent to restart the mochostagent service.
Deployment
When deploying Azure Local via the Azure portal, you might encounter the following deployment validation failure:
Could not complete the operation. 400: Resource creation validation failed. Details: [{"Code":"AnswerFileValidationFailed","Message":"Errors in Value Validation:\r\nPhysicalNodesValidator found error at deploymentdata.physicalnodes[0].ipv4address: The specified for \u0027deploymentdata.physicalnodes[0].ipv4address\u0027 is not a valid IPv4 address. Example: 192.168.0.1 or 192.168.0.1","Target":null,"Details":null}].
If you go to the Networking tab in Azure portal deployment, within the Network Intent configuration, you could see the following error: The selected physical network adapter is not binded to the management virtual switch.
The deployment via the Azure portal fails with this error: Failed to fetch secret LocalAdminCredential from key vault.
There is no workaround for this issue in this release. If the issue occurs, contact Microsoft Support for next steps.
Deployment
In some instances, during the registration of Azure Stack HCI servers, this error might be seen in the debug logs: Encountered internal server error. One of the mandatory extensions for device deployment might not be installed.
There's an intermittent issue in this release when the Azure portal incorrectly reports the update status as Failed to update or In progress though the update is complete.
Connect to your Azure Local via a remote PowerShell session. To confirm the update status, run the following PowerShell cmdlets:
$Update = get-solutionupdate| ? version -eq "<version string>"
Replace the version string with the version you're running. For example, "10.2405.0.23".
$Update.state
If the update status is Installed, no further action is required on your part. Azure portal refreshes the status correctly within 24 hours. To refresh the status sooner, follow these steps on one of the cluster nodes. Restart the Cloud Management cluster group. Stop-ClusterGroup "Cloud Management" Start-ClusterGroup "Cloud Management"
Update
During an initial MOC update, a failure occurs due to the target MOC version not being found in the catalog cache. The follow-up updates and retries show MOC in the target version, without the update succeeding, and as a result the Arc Resource Bridge update fails.
To validate this issue, collect the update logs using Troubleshoot solution updates for Azure Local. The log files should show a similar error message (current version might differ in the error message):
[ERROR: { "errorCode": "InvalidEntityError", "errorResponse": "{\n\"message\": \"the cloud fabric (MOC) is currently at version v0.13.1. A minimum version of 0.15.0 is required for compatibility\"\n}" }]
Follow these steps to mitigate the issue:
1. To find the MOC agent version, run the following command: 'C:\Program Files\AksHci\wssdcloudagent.exe' version.
2. Use the output of the command to find the MOC version from the table below that matches the agent version, and set $initialMocVersion to that MOC version. Set the $targetMocVersion by finding the Azure Stack HCI build you're updating to and get the matching MOC version from the following table. Use these values in the mitigation script provided below:
Build
MOC version
Agent version
2311.2
1.0.24.10106
v0.13.0-6-gf13a73f7, v0.11.0-alpha.38,01/06/2024
2402
1.0.25.10203
v0.14.0, v0.13.1, 02/02/2024
2402.1
1.0.25.10302
v0.14.0, v0.13.1, 03/02/2024
2402.2
1.1.1.10314
v0.16.0-1-g04bf0dec, v0.15.1, 03/14/2024
2405/2402.3
1.3.0.10418
v0.17.1, v0.16.5, 04/18/2024
For example, if the agent version is v0.13.0-6-gf13a73f7, v0.11.0-alpha.38,01/06/2024, then $initialMocVersion = “1.0.24.10106” and if you are updating to 2405.0.23, then $targetMocVersion = “1.3.0.10418”.
3. Run the following PowerShell commands on the first node:
$initialMocVersion = "<initial version determined from step 2>" $targetMocVersion = "<target version determined from step 2>"
# Import MOC module twice import-module moc import-module moc $verbosePreference = "Continue"
# Clear the SFS catalog cache Remove-Item (Get-MocConfig).manifestCache
# Set version to the current MOC version prior to update, and set state as update failed Set-MocConfigValue -name "version" -value $initialMocVersion Set-MocConfigValue -name "installState" -value ([InstallState]::UpdateFailed)
# Rerun the MOC update to desired version Update-Moc -version $targetMocVersion
4. Resume the update.
AKS on HCI
AKS cluster creation fails with the Error: Invalid AKS network resource id. This issue can occur when the associated logical network name has an underscore.
Underscores aren't supported in logical network names. Make sure to not use underscore in the names for logical networks deployed on your Azure Stack HCI.
Repair server
In rare instances, the Repair-Server operation fails with the HealthServiceWaitForDriveFW error. In these cases, the old drives from the repaired node aren't removed and new disks are stuck in the maintenance mode.
To prevent this issue, make sure that you DO NOT drain the node either via the Windows Admin Center or using the Suspend-ClusterNode -Drain PowerShell cmdlet before you start Repair-Server. If the issue occurs, contact Microsoft Support for next steps.
Repair server
This issue is seen when the single server Azure Stack HCI is updated from 2311 to 2402 and then the Repair-Server is performed. The repair operation fails.
Before you repair the single node, follow these steps: 1. Run version 2402 for the ADPrepTool. Follow the steps in Prepare Active Directory. This action is quick and adds the required permissions to the Organizational Unit (OU). 2. Move the computer object from Computers segment to the root OU. Run the following command: Get-ADComputer <HOSTNAME> | Move-ADObject -TargetPath "<OU path>"
Deployment
If you prepare the Active Directory on your own (not using the script and procedure provided by Microsoft), your Active Directory validation could fail with missing Generic All permission. This is due to an issue in the validation check that checks for a dedicated permission entry for msFVE-RecoverInformationobjects – General – Permissions Full control, which is required for BitLocker recovery.
Use the Prepare AD script method or if using your own method, make sure to assign the specific permission msFVE-RecoverInformationobjects – General – Permissions Full control.
Deployment
There's a rare issue in this release where the DNS record is deleted during the Azure Stack HCI deployment. When that occurs, the following exception is seen: Type 'PropagatePublicRootCertificate' of Role 'ASCA' raised an exception:<br>The operation on computer 'ASB88RQ22U09' failed: WinRM cannot process the request. The following error occurred while using Kerberos authentication: Cannot find the computer ASB88RQ22U09.local. Verify that the computer exists on the network and that the name provided is spelled correctly at PropagatePublicRootCertificate, C:\NugetStore\Microsoft.AzureStack, at Orchestration.Roles.CertificateAuthority.10.2402.0.14\content\Classes\ASCA\ASCA.psm1: line 38, at C:\CloudDeployment\ECEngine\InvokeInterfaceInternal.psm1: line 127,at Invoke-EceInterfaceInternal, C:\CloudDeployment\ECEngine\InvokeInterfaceInternal.psm1: line 123.
Check the DNS server to see if any DNS records of the cluster nodes are missing. Apply the following mitigation on the nodes where its DNS record is missing.
Restart the DNS client service. Open a PowerShell session and run the following cmdlet on the affected node: Taskkill /f /fi "SERVICES eq dnscache"
Deployment
In this release, there's a remote task failure on a multi-node deployment that results in the following exception: ECE RemoteTask orchestration failure with ASRR1N42R01U31 (node pingable - True): A WebException occurred while sending a RestRequest. WebException.Status: ConnectFailure on [https://<URL>](https://<URL>).
The mitigation is to restart the ECE agent on the affected node. On your server, open a PowerShell session and run the following command: Restart-Service ECEAgent.
Add server
In this release and previous releases, when adding a server to the cluster, is not possible to update the proxy bypass list string to include the new server. Updating environment variables proxy bypass list on the hosts will not update the proxy bypass list on Azure Resource Bridge or AKS.
There's no workaround in this release. If you encounter this issue, contact Microsoft Support to determine next steps.
Add/Repair server
In this release, when adding or repairing a server, a failure is seen when the software load balancer or network controller VM certificates are being copied from the existing nodes. The failure is because these certificates weren't generated during the deployment/update.
There's no workaround in this release. If you encounter this issue, contact Microsoft Support to determine next steps.
Deployment
In this release, there's a transient issue resulting in the deployment failure with the following exception: Type 'SyncDiagnosticLevel' of Role 'ObservabilityConfig' raised an exception:*<br>*Syncing Diagnostic Level failed with error: The Diagnostic Level does not match. Portal was not set to Enhanced, instead is Basic.
As this is a transient issue, retrying the deployment should fix this. For more information, see how to Rerun the deployment.
Deployment
In this release, there's an issue with the Secrets URI/location field. This is a required field that is marked Not mandatory and results in Azure Resource Manager template deployment failures.
Use the sample parameters file in the Deploy Azure Local via Azure Resource Manager template to ensure that all the inputs are provided in the required format and then try the deployment. If there's a failed deployment, you must also clean up the following resources before you Rerun the deployment: 1. Delete C:\EceStore. 2. Delete C:\CloudDeployment. 3. Delete C:\nugetstore. 4. Remove-Item HKLM:\Software\Microsoft\LCMAzureStackStampInformation.
Security
For new deployments, Secured-core capable devices won't have Dynamic Root of Measurement (DRTM) enabled by default. If you try to enable (DRTM) using the Enable-AzSSecurity cmdlet, you see an error that DRTM setting isn't supported in the current release. Microsoft recommends defense in depth, and UEFI Secure Boot still protects the components in the Static Root of Trust (SRT) boot chain by ensuring that they're loaded only when they're signed and verified.
DRTM isn't supported in this release.
Networking
An environment check fails when a proxy server is used. By design, the bypass list is different for winhttp and wininet, which causes the validation check to fail.
Follow these workaround steps:
1. Clear the proxy bypass list prior to the health check and before starting the deployment or the update.
2. After passing the check, wait for the deployment or update to fail.
3. Set your proxy bypass list again.
Arc VM management
Deployment or update of Arc Resource Bridge could fail when the automatically generated temporary SPN secret during this operation, starts with a hyphen.
Retry the deployment/update. The retry should regenerate the SPN secret and the operation will likely succeed.
Arc VM management
Arc Extensions on Arc VMs stay in "Creating" state indefinitely.
Sign in to the VM, open a command prompt, and type the following: Windows: notepad C:\ProgramData\AzureConnectedMachineAgent\Config\agentconfig.json Linux: sudo vi /var/opt/azcmagent/agentconfig.json Next, find the resourcename property. Delete the GUID that is appended to the end of the resource name, so this property matches the name of the VM. Then restart the VM.
Arc VM management
When a new server is added to an Azure Stack HCI cluster, storage path isn't created automatically for the newly created volume.
You can manually create a storage path for any new volumes. For more information, see Create a storage path.
Arc VM management
Restart of Arc VM operation completes after approximately 20 minutes although the VM itself restarts in about a minute.
There's no known workaround in this release.
Arc VM management
In some instances, the status of the logical network shows as Failed in Azure portal. This occurs when you try to delete the logical network without first deleting any resources such as network interfaces associated with that logical network. You should still be able to create resources on this logical network. The status is misleading in this instance.
If the status of this logical network was Succeeded at the time when this network was provisioned, then you can continue to create resources on this network.
Arc VM management
In this release, when you update a VM with a data disk attached to it using the Azure CLI, the operation fails with the following error message: Couldn't find a virtual hard disk with the name.
In rare instances, you may encounter this error while updating your Azure Stack HCI: Type 'UpdateArbAndExtensions' of Role 'MocArb' raised an exception: Exception Upgrading ARB and Extension in step [UpgradeArbAndExtensions :Get-ArcHciConfig] UpgradeArb: Invalid applianceyaml = [C:\AksHci\hci-appliance.yaml].
If you see this issue, contact Microsoft Support to assist you with the next steps.
Networking
There's an infrequent DNS client issue in this release that causes the deployment to fail on a two-node cluster with a DNS resolution error: A WebException occurred while sending a RestRequest. WebException.Status: NameResolutionFailure. As a result of the bug, the DNS record of the second node is deleted soon after it's created resulting in a DNS error.
Restart the server. This operation registers the DNS record, which prevents it from getting deleted.
Azure portal
In some instances, the Azure portal might take a while to update and the view might not be current.
You might need to wait for 30 minutes or more to see the updated view.
Arc VM management
Deleting a network interface on an Arc VM from Azure portal doesn't work in this release.
Providing the OU name in an incorrect syntax isn't detected in the Azure portal. The incorrect syntax includes unsupported characters such as &,",',<,>. The incorrect syntax is detected at a later step during cluster validation.
Make sure that the OU path syntax is correct and doesn't include unsupported characters.
Deployment
Deployments via Azure Resource Manager time out after 2 hours. Deployments that exceed 2 hours show up as failed in the resource group though the cluster is successfully created.
To monitor the deployment in the Azure portal, go to the Azure Stack HCI cluster resource and then go to new Deployments entry.
Azure Site Recovery
Azure Site Recovery can't be installed on an Azure Stack HCI cluster in this release.
There's no known workaround in this release.
Update
When updating the Azure Stack HCI cluster via the Azure Update Manager, the update progress and results may not be visible in the Azure portal.
To work around this issue, on each cluster node, add the following registry key (no value needed):
Then on one of the cluster nodes, restart the Cloud Management cluster group.
Stop-ClusterGroup "Cloud Management"
Start-ClusterGroup "Cloud Management"
This won't fully remediate the issue as the progress details may still not be displayed for a duration of the update process. To get the latest update details, you can Retrieve the update progress with PowerShell.
Update
In rare instances, if a failed update is stuck in an In progress state in Azure Update Manager, the Try again button is disabled.
To resume the update, run the following PowerShell command: Get-SolutionUpdate|Start-SolutionUpdate.
Updates
In some cases, SolutionUpdate commands could fail if run after the Send-DiagnosticData command.
Make sure to close the PowerShell session used for Send-DiagnosticData. Open a new PowerShell session and use it for SolutionUpdate commands.
Update
In rare instances, when applying an update from 2311.0.24 to 2311.2.4, cluster status reports In Progress instead of expected Failed to update.
Retry the update. If the issue persists, contact Microsoft Support.
Update
Attempts to install solution updates can fail at the end of the CAU steps with: There was a failure in a Common Information Model (CIM) operation, that is, an operation performed by software that Cluster-Aware Updating depends on. This rare issue occurs if the Cluster Name or Cluster IP Address resources fail to start after a node reboot and is most typical in small clusters.
If you encounter this issue, contact Microsoft Support for next steps. They can work with you to manually restart the cluster resources and resume the update as needed.
Update
When applying a cluster update to 10.2402.3.11 the Get-SolutionUpdate cmdlet may not respond and eventually fails with a RequestTimeoutException after approximately 10 minutes. This is likely to occur following an add or repair server scenario.
Use the Start-ClusterGroup and Stop-ClusterGroup cmdlets to restart the update service.
A successful run of these cmdlets should bring the update service online.
Cluster aware updating
Resume node operation failed to resume node.
This is a transient issue and could resolve on its own. Wait for a few minutes and retry the operation. If the issue persists, contact Microsoft Support.
Cluster aware updating
Suspend node operation was stuck for greater than 90 minutes.
This is a transient issue and could resolve on its own. Wait for a few minutes and retry the operation. If the issue persists, contact Microsoft Support.
Issues for version 2405.1
This software release maps to software version number 2405.1.4.
Release notes for this version include the issues fixed in this release, known issues in this release, and release noted issues carried over from previous versions.
Fixed issues
Here are the fixed issues in this release:
Feature
Issue
Workaround/Comments
Updates
An update issue was fixed. This issue caused the update to fail after the Cluster-Aware Updating (CAU) step although a CAU rerun in this case would fix the issue.
Observability
In this release, an issue was fixed that resulted in proactive log collection being disabled by default after the extension was installed.
Updates
An issue was fixed where the Agent Lifecycle Manager (ALM) failed to restart services after secret rotation.
Updates
In this release, an issue was fixed where using the PowerShell command Start-SolutionUpdate to retry a failed solution update failed.
Updates
An issue was fixed that caused a Solution Builder Extension update to fail to download.
Updates
An issue was fixed where the updates failed during the Service Principal Name (SPN) verification based on the deployment SPN settings.
Updates
An issue was fixed where the update of Arc Resource Bridge (ARB) takes a long time and the update fails.
Updates
An issue was fixed where the Solution Builder Update health checks were missing from the preupdate or daily health checks.
Add server Repair server
During Add-Server, the cluster storage network shouldn't be expected to be the same as the storage VLAN ID.
Networking
AzStackHci_Network_Test_Infra_IP_Connection needs to honor the severity of the endpoint definition.
Known issues in this release
Microsoft isn't aware of any known issues in this release.
Known issues from previous releases
Here are the known issues from previous releases:
Feature
Issue
Workaround
Update
When viewing the readiness check results for an Azure Stack HCI cluster via the Azure Update Manager, there might be multiple readiness checks with the same name.
There's no known workaround in this release. Select View details to view specific information about the readiness check.
Arc VM management
In large deployment scenarios, such as extensive AVD host pool deployments or large-scale VM provisioning, you might encounter reliability issues caused by a Hyper-V socket external library problem.
Follow these steps to mitigate the issue: 1. Run the command Get-service mochostagent (\) get-process (\) kill. Check the output of the command and verify if the handle count is in the thousands.
2. Run the command Get-service mochostagent (\) get-process to terminate the processes.
3. Run the command restart-service mochostagent to restart the mochostagent service.
Deployment
When deploying Azure Local via the Azure portal, you might encounter the following deployment validation failure:
Could not complete the operation. 400: Resource creation validation failed. Details: [{"Code":"AnswerFileValidationFailed","Message":"Errors in Value Validation:\r\nPhysicalNodesValidator found error at deploymentdata.physicalnodes[0].ipv4address: The specified for \u0027deploymentdata.physicalnodes[0].ipv4address\u0027 is not a valid IPv4 address. Example: 192.168.0.1 or 192.168.0.1","Target":null,"Details":null}].
If you go to the Networking tab in Azure portal deployment, within the Network Intent configuration, you could see the following error: The selected physical network adapter is not binded to the management virtual switch.
The deployment via the Azure portal fails with this error: Failed to fetch secret LocalAdminCredential from key vault.
There is no workaround for this issue in this release. If the issue occurs, contact Microsoft Support for next steps.
Deployment
In some instances, during the registration of Azure Stack HCI servers, this error might be seen in the debug logs: Encountered internal server error. One of the mandatory extensions for device deployment might not be installed.
There's an intermittent issue in this release when the Azure portal incorrectly reports the update status as Failed to update or In progress though the update is complete.
Connect to your Azure Local via a remote PowerShell session. To confirm the update status, run the following PowerShell cmdlets:
$Update = get-solutionupdate| ? version -eq "<version string>"
Replace the version string with the version you're running. For example, "10.2405.0.23".
$Update.state
If the update status is Installed, no further action is required on your part. Azure portal refreshes the status correctly within 24 hours. To refresh the status sooner, follow these steps on one of the cluster nodes. Restart the Cloud Management cluster group. Stop-ClusterGroup "Cloud Management" Start-ClusterGroup "Cloud Management"
Update
During an initial MOC update, a failure occurs due to the target MOC version not being found in the catalog cache. The follow-up updates and retries show MOC in the target version, without the update succeeding, and as a result the Arc Resource Bridge update fails.
To validate this issue, collect the update logs using Troubleshoot solution updates for Azure Local. The log files should show a similar error message (current version might differ in the error message):
[ERROR: { "errorCode": "InvalidEntityError", "errorResponse": "{\n\"message\": \"the cloud fabric (MOC) is currently at version v0.13.1. A minimum version of 0.15.0 is required for compatibility\"\n}" }]
Follow these steps to mitigate the issue:
1. To find the MOC agent version, run the following command: 'C:\Program Files\AksHci\wssdcloudagent.exe' version.
2. Use the output of the command to find the MOC version from the table below that matches the agent version, and set $initialMocVersion to that MOC version. Set the $targetMocVersion by finding the Azure Stack HCI build you're updating to and get the matching MOC version from the following table. Use these values in the mitigation script provided below:
Build
MOC version
Agent version
2311.2
1.0.24.10106
v0.13.0-6-gf13a73f7, v0.11.0-alpha.38,01/06/2024
2402
1.0.25.10203
v0.14.0, v0.13.1, 02/02/2024
2402.1
1.0.25.10302
v0.14.0, v0.13.1, 03/02/2024
2402.2
1.1.1.10314
v0.16.0-1-g04bf0dec, v0.15.1, 03/14/2024
2405/2402.3
1.3.0.10418
v0.17.1, v0.16.5, 04/18/2024
For example, if the agent version is v0.13.0-6-gf13a73f7, v0.11.0-alpha.38,01/06/2024, then $initialMocVersion = “1.0.24.10106” and if you are updating to 2405.0.23, then $targetMocVersion = “1.3.0.10418”.
3. Run the following PowerShell commands on the first node:
$initialMocVersion = "<initial version determined from step 2>" $targetMocVersion = "<target version determined from step 2>"
# Import MOC module twice import-module moc import-module moc $verbosePreference = "Continue"
# Clear the SFS catalog cache Remove-Item (Get-MocConfig).manifestCache
# Set version to the current MOC version prior to update, and set state as update failed Set-MocConfigValue -name "version" -value $initialMocVersion Set-MocConfigValue -name "installState" -value ([InstallState]::UpdateFailed)
# Rerun the MOC update to desired version Update-Moc -version $targetMocVersion
4. Resume the update.
AKS on HCI
AKS cluster creation fails with the Error: Invalid AKS network resource id. This issue can occur when the associated logical network name has an underscore.
Underscores aren't supported in logical network names. Make sure to not use underscore in the names for logical networks deployed on your Azure Stack HCI.
Repair server
In rare instances, the Repair-Server operation fails with the HealthServiceWaitForDriveFW error. In these cases, the old drives from the repaired node aren't removed and new disks are stuck in the maintenance mode.
To prevent this issue, make sure that you DO NOT drain the node either via the Windows Admin Center or using the Suspend-ClusterNode -Drain PowerShell cmdlet before you start Repair-Server. If the issue occurs, contact Microsoft Support for next steps.
Repair server
This issue is seen when the single server Azure Stack HCI is updated from 2311 to 2402 and then the Repair-Server is performed. The repair operation fails.
Before you repair the single node, follow these steps: 1. Run version 2402 for the ADPrepTool. Follow the steps in Prepare Active Directory. This action is quick and adds the required permissions to the Organizational Unit (OU). 2. Move the computer object from Computers segment to the root OU. Run the following command: Get-ADComputer <HOSTNAME> | Move-ADObject -TargetPath "<OU path>"
Deployment
If you prepare the Active Directory on your own (not using the script and procedure provided by Microsoft), your Active Directory validation could fail with missing Generic All permission. This is due to an issue in the validation check that checks for a dedicated permission entry for msFVE-RecoverInformationobjects – General – Permissions Full control, which is required for BitLocker recovery.
Use the Prepare AD script method or if using your own method, make sure to assign the specific permission msFVE-RecoverInformationobjects – General – Permissions Full control.
Deployment
There's a rare issue in this release where the DNS record is deleted during the Azure Stack HCI deployment. When that occurs, the following exception is seen: Type 'PropagatePublicRootCertificate' of Role 'ASCA' raised an exception:<br>The operation on computer 'ASB88RQ22U09' failed: WinRM cannot process the request. The following error occurred while using Kerberos authentication: Cannot find the computer ASB88RQ22U09.local. Verify that the computer exists on the network and that the name provided is spelled correctly at PropagatePublicRootCertificate, C:\NugetStore\Microsoft.AzureStack, at Orchestration.Roles.CertificateAuthority.10.2402.0.14\content\Classes\ASCA\ASCA.psm1: line 38, at C:\CloudDeployment\ECEngine\InvokeInterfaceInternal.psm1: line 127,at Invoke-EceInterfaceInternal, C:\CloudDeployment\ECEngine\InvokeInterfaceInternal.psm1: line 123.
Check the DNS server to see if any DNS records of the cluster nodes are missing. Apply the following mitigation on the nodes where its DNS record is missing.
Restart the DNS client service. Open a PowerShell session and run the following cmdlet on the affected node: Taskkill /f /fi "SERVICES eq dnscache"
Deployment
In this release, there's a remote task failure on a multi-node deployment that results in the following exception: ECE RemoteTask orchestration failure with ASRR1N42R01U31 (node pingable - True): A WebException occurred while sending a RestRequest. WebException.Status: ConnectFailure on [https://<URL>](https://<URL>).
The mitigation is to restart the ECE agent on the affected node. On your server, open a PowerShell session and run the following command: Restart-Service ECEAgent.
Add server
In this release and previous releases, when adding a server to the cluster, is not possible to update the proxy bypass list string to include the new server. Updating environment variables proxy bypass list on the hosts will not update the proxy bypass list on Azure Resource Bridge or AKS.
There's no workaround in this release. If you encounter this issue, contact Microsoft Support to determine next steps.
Add/Repair server
In this release, when adding or repairing a server, a failure is seen when the software load balancer or network controller VM certificates are being copied from the existing nodes. The failure is because these certificates weren't generated during the deployment/update.
There's no workaround in this release. If you encounter this issue, contact Microsoft Support to determine next steps.
Deployment
In this release, there's a transient issue resulting in the deployment failure with the following exception: Type 'SyncDiagnosticLevel' of Role 'ObservabilityConfig' raised an exception:*<br>*Syncing Diagnostic Level failed with error: The Diagnostic Level does not match. Portal was not set to Enhanced, instead is Basic.
As this is a transient issue, retrying the deployment should fix this. For more information, see how to Rerun the deployment.
Deployment
In this release, there's an issue with the Secrets URI/location field. This is a required field that is marked Not mandatory and results in Azure Resource Manager template deployment failures.
Use the sample parameters file in the Deploy Azure Local via Azure Resource Manager template to ensure that all the inputs are provided in the required format and then try the deployment. If there's a failed deployment, you must also clean up the following resources before you Rerun the deployment: 1. Delete C:\EceStore. 2. Delete C:\CloudDeployment. 3. Delete C:\nugetstore. 4. Remove-Item HKLM:\Software\Microsoft\LCMAzureStackStampInformation.
Security
For new deployments, Secured-core capable devices won't have Dynamic Root of Measurement (DRTM) enabled by default. If you try to enable (DRTM) using the Enable-AzSSecurity cmdlet, you see an error that DRTM setting isn't supported in the current release. Microsoft recommends defense in depth, and UEFI Secure Boot still protects the components in the Static Root of Trust (SRT) boot chain by ensuring that they're loaded only when they're signed and verified.
DRTM isn't supported in this release.
Networking
An environment check fails when a proxy server is used. By design, the bypass list is different for winhttp and wininet, which causes the validation check to fail.
Follow these workaround steps:
1. Clear the proxy bypass list prior to the health check and before starting the deployment or the update.
2. After passing the check, wait for the deployment or update to fail.
3. Set your proxy bypass list again.
Arc VM management
Deployment or update of Arc Resource Bridge could fail when the automatically generated temporary SPN secret during this operation, starts with a hyphen.
Retry the deployment/update. The retry should regenerate the SPN secret and the operation will likely succeed.
Arc VM management
Arc Extensions on Arc VMs stay in "Creating" state indefinitely.
Sign in to the VM, open a command prompt, and type the following: Windows: notepad C:\ProgramData\AzureConnectedMachineAgent\Config\agentconfig.json Linux: sudo vi /var/opt/azcmagent/agentconfig.json Next, find the resourcename property. Delete the GUID that is appended to the end of the resource name, so this property matches the name of the VM. Then restart the VM.
Arc VM management
When a new server is added to an Azure Stack HCI cluster, storage path isn't created automatically for the newly created volume.
You can manually create a storage path for any new volumes. For more information, see Create a storage path.
Arc VM management
Restart of Arc VM operation completes after approximately 20 minutes although the VM itself restarts in about a minute.
There's no known workaround in this release.
Arc VM management
In some instances, the status of the logical network shows as Failed in Azure portal. This occurs when you try to delete the logical network without first deleting any resources such as network interfaces associated with that logical network. You should still be able to create resources on this logical network. The status is misleading in this instance.
If the status of this logical network was Succeeded at the time when this network was provisioned, then you can continue to create resources on this network.
Arc VM management
In this release, when you update a VM with a data disk attached to it using the Azure CLI, the operation fails with the following error message: Couldn't find a virtual hard disk with the name.
In rare instances, you may encounter this error while updating your Azure Stack HCI: Type 'UpdateArbAndExtensions' of Role 'MocArb' raised an exception: Exception Upgrading ARB and Extension in step [UpgradeArbAndExtensions :Get-ArcHciConfig] UpgradeArb: Invalid applianceyaml = [C:\AksHci\hci-appliance.yaml].
If you see this issue, contact Microsoft Support to assist you with the next steps.
Networking
There's an infrequent DNS client issue in this release that causes the deployment to fail on a two-node cluster with a DNS resolution error: A WebException occurred while sending a RestRequest. WebException.Status: NameResolutionFailure. As a result of the bug, the DNS record of the second node is deleted soon after it's created resulting in a DNS error.
Restart the server. This operation registers the DNS record, which prevents it from getting deleted.
Azure portal
In some instances, the Azure portal might take a while to update and the view might not be current.
You might need to wait for 30 minutes or more to see the updated view.
Arc VM management
Deleting a network interface on an Arc VM from Azure portal doesn't work in this release.
Providing the OU name in an incorrect syntax isn't detected in the Azure portal. The incorrect syntax includes unsupported characters such as &,",',<,>. The incorrect syntax is detected at a later step during cluster validation.
Make sure that the OU path syntax is correct and doesn't include unsupported characters.
Deployment
Deployments via Azure Resource Manager time out after 2 hours. Deployments that exceed 2 hours show up as failed in the resource group though the cluster is successfully created.
To monitor the deployment in the Azure portal, go to the Azure Stack HCI cluster resource and then go to new Deployments entry.
Azure Site Recovery
Azure Site Recovery can't be installed on an Azure Stack HCI cluster in this release.
There's no known workaround in this release.
Update
When updating the Azure Stack HCI cluster via the Azure Update Manager, the update progress and results may not be visible in the Azure portal.
To work around this issue, on each cluster node, add the following registry key (no value needed):
Then on one of the cluster nodes, restart the Cloud Management cluster group.
Stop-ClusterGroup "Cloud Management"
Start-ClusterGroup "Cloud Management"
This won't fully remediate the issue as the progress details may still not be displayed for a duration of the update process. To get the latest update details, you can Retrieve the update progress with PowerShell.
Update
In rare instances, if a failed update is stuck in an In progress state in Azure Update Manager, the Try again button is disabled.
To resume the update, run the following PowerShell command: Get-SolutionUpdate|Start-SolutionUpdate.
Updates
In some cases, SolutionUpdate commands could fail if run after the Send-DiagnosticData command.
Make sure to close the PowerShell session used for Send-DiagnosticData. Open a new PowerShell session and use it for SolutionUpdate commands.
Update
In rare instances, when applying an update from 2311.0.24 to 2311.2.4, cluster status reports In Progress instead of expected Failed to update.
Retry the update. If the issue persists, contact Microsoft Support.
Update
Attempts to install solution updates can fail at the end of the CAU steps with: There was a failure in a Common Information Model (CIM) operation, that is, an operation performed by software that Cluster-Aware Updating depends on. This rare issue occurs if the Cluster Name or Cluster IP Address resources fail to start after a node reboot and is most typical in small clusters.
If you encounter this issue, contact Microsoft Support for next steps. They can work with you to manually restart the cluster resources and resume the update as needed.
Update
When applying a cluster update to 10.2402.3.11 the Get-SolutionUpdate cmdlet may not respond and eventually fails with a RequestTimeoutException after approximately 10 minutes. This is likely to occur following an add or repair server scenario.
Use the Start-ClusterGroup and Stop-ClusterGroup cmdlets to restart the update service.
A successful run of these cmdlets should bring the update service online.
Cluster aware updating
Resume node operation failed to resume node.
This is a transient issue and could resolve on its own. Wait for a few minutes and retry the operation. If the issue persists, contact Microsoft Support.
Cluster aware updating
Suspend node operation was stuck for greater than 90 minutes.
This is a transient issue and could resolve on its own. Wait for a few minutes and retry the operation. If the issue persists, contact Microsoft Support.
Issues for version 2405
This software release maps to software version number 2405.0.24.
Release notes for this version include the issues fixed in this release, known issues in this release, and release noted issues carried over from previous versions.
Fixed issues
Here are the fixed issues in this release:
Feature
Issue
Workaround/Comments
Active Directory
During cluster deployments that use a large Active Directory, an issue that can cause timeouts when adding users to the local administrator group, is fixed.
Deployment
New ARM templates are released for cluster creation that simplify the dependency resource creation. These templates include some fixes that addressed the missing mandatory fields.
Deployment
The secret rotation PowerShell command Set-AzureStackLCMUserPassword supports a new parameter to skip the confirmation message.
Deployment
Improved the reliability of secret rotation when services aren't restarting in a timely manner.
Deployment
Fixed an issue so that the deployment is enabled when a disjoint namespace is used.
Deployment
Fixed an issue in deployment when setting the diagnostic level in Azure and the device.
SBE
A new PowerShell command is released that can be used to update the SBE partner property values provided at deployment time.
SBE
Fixed an issue that prevents the update service to respond to requests after an SBE only update run.
Add server Repair server
An issue is fixed that prevents a node from joining Active Directory during an add server operation.
Networking
Improved the reliability of Network ATC when setting up the host networking configuration with certain network adapter types.
Networking
Improved reliability when detecting firmware versions for disk drives.
Updates
Improved the reliability of update notifications for health check results sent from the device to AUM (Azure Update Manager). In certain cases, the message size could be too large and caused no results to be shown in AUM.
Updates
Fixed a file lock issue that can cause update failures for the trusted launch VM agent (IGVM).
Updates
Fixed an issue that prevented the orchestrator agent from being restarted during an update run.
Updates
Fixed a rare condition where it took a long time for the update service to discover or start an update.
Updates
Fixed an issue for Cluster-Aware Updating (CAU) interaction with the orchestrator when an update in progress is reported by CAU.
Updates
The naming schema for updates was adjusted to allow the identification of feature versus cumulative updates.
Updates
Improved the reliability of reporting the cluster update progress to the orchestrator.
Azure Arc
Resolved an issue where the Azure Arc connection was lost when the Hybrid Instance Metadata service (HIMDS) restarted, breaking Azure portal functionality. The device now automatically reinitiates the Azure Arc connection in these cases.
Known issues in this release
Here are the known issues in this release:
Feature
Issue
Workaround/Comments
Arc VM management
In large deployment scenarios, such as extensive AVD host pool deployments or large-scale VM provisioning, you might encounter reliability issues caused by a Hyper-V socket external library problem.
Follow these steps to mitigate the issue: 1. Run the command Get-service mochostagent (\) get-process (\) kill. Check the output of the command and verify if the handle count is in the thousands.
2. Run the command Get-service mochostagent (\) get-process to terminate the processes.
3. Run the command restart-service mochostagent to restart the mochostagent service.
Deployment
When deploying Azure Local via the Azure portal, you might encounter the following deployment validation failure:
Could not complete the operation. 400: Resource creation validation failed. Details: [{"Code":"AnswerFileValidationFailed","Message":"Errors in Value Validation:\r\nPhysicalNodesValidator found error at deploymentdata.physicalnodes[0].ipv4address: The specified for \u0027deploymentdata.physicalnodes[0].ipv4address\u0027 is not a valid IPv4 address. Example: 192.168.0.1 or 192.168.0.1","Target":null,"Details":null}].
If you go to the Networking tab in Azure portal deployment, within the Network Intent configuration, you could see the following error: The selected physical network adapter is not binded to the management virtual switch.
The deployment via the Azure portal fails with this error: Failed to fetch secret LocalAdminCredential from key vault.
There is no workaround for this issue in this release. If the issue occurs, contact Microsoft Support for next steps.
Deployment
The new ISO image for the Azure Stack HCI, version 23H2 operating system was rolled back to a previous version owing to compatibility issues with some hardware configurations.
If you encounter any issues in Arc registration, roll back to the previous version. No action is required for you if you have already successfully deployed the newer image. Both the ISO images are the same operating system build version.
Update
When viewing the readiness check results for an Azure Stack HCI cluster via the Azure Update Manager, there may be multiple readiness checks with the same name.
There's no known workaround in this release. Select View details to view specific information about the readiness check.
Deployment
In some instances, during the registration of Azure Stack HCI servers, this error may be seen in the debug logs: Encountered internal server error. One of the mandatory extensions for device deployment may not be installed.
There's an intermittent issue in this release when the Azure portal incorrectly reports the update status as Failed to update or In progress though the update is complete.
Connect to your Azure Local via a remote PowerShell session. To confirm the update status, run the following PowerShell cmdlets:
$Update = get-solutionupdate| ? version -eq "<version string>"
Replace the version string with the version you're running. For example, "10.2405.0.23".
$Update.state
If the update status is Installed, no further action is required on your part. Azure portal refreshes the status correctly within 24 hours. To refresh the status sooner, follow these steps on one of the cluster nodes. Restart the Cloud Management cluster group. Stop-ClusterGroup "Cloud Management" Start-ClusterGroup "Cloud Management"
Update
During an initial MOC update, a failure occurs due to the target MOC version not being found in the catalog cache. The follow-up updates and retries show MOC in the target version, without the update succeeding, and as a result the Arc Resource Bridge update fails.
To validate this issue, collect the update logs using Troubleshoot solution updates for Azure Local. The log files should show a similar error message (current version might differ in the error message):
[ERROR: { "errorCode": "InvalidEntityError", "errorResponse": "{\n\"message\": \"the cloud fabric (MOC) is currently at version v0.13.1. A minimum version of 0.15.0 is required for compatibility\"\n}" }]
Follow these steps to mitigate the issue:
1. To find the MOC agent version, run the following command: 'C:\Program Files\AksHci\wssdcloudagent.exe' version.
2. Use the output of the command to find the MOC version from the table below that matches the agent version, and set $initialMocVersion to that MOC version. Set the $targetMocVersion by finding the Azure Stack HCI build you are updating to and get the matching MOC version from the table below. Use these values in the mitigation script provided below:
Build
MOC version
Agent version
2311.2
1.0.24.10106
v0.13.0-6-gf13a73f7, v0.11.0-alpha.38,01/06/2024
2402
1.0.25.10203
v0.14.0, v0.13.1, 02/02/2024
2402.1
1.0.25.10302
v0.14.0, v0.13.1, 03/02/2024
2402.2
1.1.1.10314
v0.16.0-1-g04bf0dec, v0.15.1, 03/14/2024
2405/2402.3
1.3.0.10418
v0.17.1, v0.16.5, 04/18/2024
For example, if the agent version is v0.13.0-6-gf13a73f7, v0.11.0-alpha.38,01/06/2024, then $initialMocVersion = "1.0.24.10106" and if we are updating to 2405.0.23, then $targetMocVersion = "1.3.0.10418".
3. Run the following PowerShell commands on the first node:
$initialMocVersion = "<initial version determined from step 2>" $targetMocVersion = "<target version determined from step 2>"
# Import MOC module twice import-module moc import-module moc $verbosePreference = "Continue"
# Clear the SFS catalog cache Remove-Item (Get-MocConfig).manifestCache
# Set version to the current MOC version prior to update, and set state as update failed Set-MocConfigValue -name "version" -value $initialMocVersion Set-MocConfigValue -name "installState" -value ([InstallState]::UpdateFailed)
# Rerun the MOC update to desired version Update-Moc -version $targetMocVersion
4. Resume the update.
Security
The SideChannelMitigation security feature may not show an enabled state even if it's enabled. This happens when using Windows Admin Center (Cluster Security View) or when this cmdlet returns False: Get-AzSSecurity -FeatureName SideChannelMitigation.
There's no workaround in this release to fix the output of these applications. To validate the expected value, run the following cmdlet: Get-ItemProperty 'HKLM:\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management' -name "FeatureSettingsOverride*" The expected output is: FeatureSettingsOverride: 83886152 FeatureSettingsOverrideMask: 3 If your output matches the expected output, you can safely ignore the output from Windows Admin Center and Get-AzSSecurity cmdlet.
Known issues from previous releases
Here are the known issues from previous releases:
Feature
Issue
Workaround
AKS on HCI
AKS cluster creation fails with the Error: Invalid AKS network resource id. This issue can occur when the associated logical network name has an underscore.
Underscores aren't supported in logical network names. Make sure to not use underscore in the names for logical networks deployed on your Azure Stack HCI.
Repair server
In rare instances, the Repair-Server operation fails with the HealthServiceWaitForDriveFW error. In these cases, the old drives from the repaired node aren't removed and new disks are stuck in the maintenance mode.
To prevent this issue, make sure that you DO NOT drain the node either via the Windows Admin Center or using the Suspend-ClusterNode -Drain PowerShell cmdlet before you start Repair-Server. If the issue occurs, contact Microsoft Support for next steps.
Repair server
This issue is seen when the single server Azure Stack HCI is updated from 2311 to 2402 and then the Repair-Server is performed. The repair operation fails.
Before you repair the single node, follow these steps: 1. Run version 2402 for the ADPrepTool. Follow the steps in Prepare Active Directory. This action is quick and adds the required permissions to the Organizational Unit (OU). 2. Move the computer object from Computers segment to the root OU. Run the following command: Get-ADComputer <HOSTNAME> | Move-ADObject -TargetPath "<OU path>"
Deployment
If you prepare the Active Directory on your own (not using the script and procedure provided by Microsoft), your Active Directory validation could fail with missing Generic All permission. This is due to an issue in the validation check that checks for a dedicated permission entry for msFVE-RecoverInformationobjects – General – Permissions Full control, which is required for BitLocker recovery.
Use the Prepare AD script method or if using your own method, make sure to assign the specific permission msFVE-RecoverInformationobjects – General – Permissions Full control.
Deployment
There's a rare issue in this release where the DNS record is deleted during the Azure Stack HCI deployment. When that occurs, the following exception is seen: Type 'PropagatePublicRootCertificate' of Role 'ASCA' raised an exception:<br>The operation on computer 'ASB88RQ22U09' failed: WinRM cannot process the request. The following error occurred while using Kerberos authentication: Cannot find the computer ASB88RQ22U09.local. Verify that the computer exists on the network and that the name provided is spelled correctly at PropagatePublicRootCertificate, C:\NugetStore\Microsoft.AzureStack, at Orchestration.Roles.CertificateAuthority.10.2402.0.14\content\Classes\ASCA\ASCA.psm1: line 38, at C:\CloudDeployment\ECEngine\InvokeInterfaceInternal.psm1: line 127,at Invoke-EceInterfaceInternal, C:\CloudDeployment\ECEngine\InvokeInterfaceInternal.psm1: line 123.
Check the DNS server to see if any DNS records of the cluster nodes are missing. Apply the following mitigation on the nodes where its DNS record is missing.
Restart the DNS client service. Open a PowerShell session and run the following cmdlet on the affected node: Taskkill /f /fi "SERVICES eq dnscache"
Deployment
In this release, there's a remote task failure on a multi-node deployment that results in the following exception: ECE RemoteTask orchestration failure with ASRR1N42R01U31 (node pingable - True): A WebException occurred while sending a RestRequest. WebException.Status: ConnectFailure on [https://<URL>](https://<URL>).
The mitigation is to restart the ECE agent on the affected node. On your server, open a PowerShell session and run the following command: Restart-Service ECEAgent.
Add/Repair server
In this release, when adding or repairing a server, a failure is seen when the software load balancer or network controller VM certificates are being copied from the existing nodes. The failure is because these certificates weren't generated during the deployment/update.
There's no workaround in this release. If you encounter this issue, contact Microsoft Support to determine next steps.
Deployment
In this release, there's a transient issue resulting in the deployment failure with the following exception: Type 'SyncDiagnosticLevel' of Role 'ObservabilityConfig' raised an exception:*<br>*Syncing Diagnostic Level failed with error: The Diagnostic Level does not match. Portal was not set to Enhanced, instead is Basic.
As this is a transient issue, retrying the deployment should fix this. For more information, see how to Rerun the deployment.
Deployment
In this release, there's an issue with the Secrets URI/location field. This is a required field that is marked Not mandatory and results in Azure Resource Manager template deployment failures.
Use the sample parameters file in the Deploy Azure Local via Azure Resource Manager template to ensure that all the inputs are provided in the required format and then try the deployment. If there's a failed deployment, you must also clean up the following resources before you Rerun the deployment: 1. Delete C:\EceStore. 2. Delete C:\CloudDeployment. 3. Delete C:\nugetstore. 4. Remove-Item HKLM:\Software\Microsoft\LCMAzureStackStampInformation.
Security
For new deployments, Secured-core capable devices won't have Dynamic Root of Measurement (DRTM) enabled by default. If you try to enable (DRTM) using the Enable-AzSSecurity cmdlet, you see an error that DRTM setting isn't supported in the current release. Microsoft recommends defense in depth, and UEFI Secure Boot still protects the components in the Static Root of Trust (SRT) boot chain by ensuring that they're loaded only when they're signed and verified.
DRTM isn't supported in this release.
Networking
An environment check fails when a proxy server is used. By design, the bypass list is different for winhttp and wininet, which causes the validation check to fail.
Follow these workaround steps:
1. Clear the proxy bypass list prior to the health check and before starting the deployment or the update.
2. After passing the check, wait for the deployment or update to fail.
3. Set your proxy bypass list again.
Arc VM management
Deployment or update of Arc Resource Bridge could fail when the automatically generated temporary SPN secret during this operation, starts with a hyphen.
Retry the deployment/update. The retry should regenerate the SPN secret and the operation will likely succeed.
Arc VM management
Arc Extensions on Arc VMs stay in "Creating" state indefinitely.
Sign in to the VM, open a command prompt, and type the following: Windows: notepad C:\ProgramData\AzureConnectedMachineAgent\Config\agentconfig.json Linux: sudo vi /var/opt/azcmagent/agentconfig.json Next, find the resourcename property. Delete the GUID that is appended to the end of the resource name, so this property matches the name of the VM. Then restart the VM.
Arc VM management
When a new server is added to an Azure Stack HCI cluster, storage path isn't created automatically for the newly created volume.
You can manually create a storage path for any new volumes. For more information, see Create a storage path.
Arc VM management
Restart of Arc VM operation completes after approximately 20 minutes although the VM itself restarts in about a minute.
There's no known workaround in this release.
Arc VM management
In some instances, the status of the logical network shows as Failed in Azure portal. This occurs when you try to delete the logical network without first deleting any resources such as network interfaces associated with that logical network. You should still be able to create resources on this logical network. The status is misleading in this instance.
If the status of this logical network was Succeeded at the time when this network was provisioned, then you can continue to create resources on this network.
Arc VM management
In this release, when you update a VM with a data disk attached to it using the Azure CLI, the operation fails with the following error message: Couldn't find a virtual hard disk with the name.
In rare instances, you may encounter this error while updating your Azure Stack HCI: Type 'UpdateArbAndExtensions' of Role 'MocArb' raised an exception: Exception Upgrading ARB and Extension in step [UpgradeArbAndExtensions :Get-ArcHciConfig] UpgradeArb: Invalid applianceyaml = [C:\AksHci\hci-appliance.yaml].
If you see this issue, contact Microsoft Support to assist you with the next steps.
Networking
There's an infrequent DNS client issue in this release that causes the deployment to fail on a two-node cluster with a DNS resolution error: A WebException occurred while sending a RestRequest. WebException.Status: NameResolutionFailure. As a result of the bug, the DNS record of the second node is deleted soon after it's created resulting in a DNS error.
Restart the server. This operation registers the DNS record, which prevents it from getting deleted.
Azure portal
In some instances, the Azure portal might take a while to update and the view might not be current.
You might need to wait for 30 minutes or more to see the updated view.
Arc VM management
Deleting a network interface on an Arc VM from Azure portal doesn't work in this release.
Providing the OU name in an incorrect syntax isn't detected in the Azure portal. The incorrect syntax includes unsupported characters such as &,",',<,>. The incorrect syntax is detected at a later step during cluster validation.
Make sure that the OU path syntax is correct and doesn't include unsupported characters.
Deployment
Deployments via Azure Resource Manager time out after 2 hours. Deployments that exceed 2 hours show up as failed in the resource group though the cluster is successfully created.
To monitor the deployment in the Azure portal, go to the Azure Stack HCI cluster resource and then go to new Deployments entry.
Azure Site Recovery
Azure Site Recovery can't be installed on an Azure Stack HCI cluster in this release.
There's no known workaround in this release.
Update
When updating the Azure Stack HCI cluster via the Azure Update Manager, the update progress and results may not be visible in the Azure portal.
To work around this issue, on each cluster node, add the following registry key (no value needed):
Then on one of the cluster nodes, restart the Cloud Management cluster group.
Stop-ClusterGroup "Cloud Management"
Start-ClusterGroup "Cloud Management"
This won't fully remediate the issue as the progress details may still not be displayed for a duration of the update process. To get the latest update details, you can Retrieve the update progress with PowerShell.
Update
In rare instances, if a failed update is stuck in an In progress state in Azure Update Manager, the Try again button is disabled.
To resume the update, run the following PowerShell command: Get-SolutionUpdate|Start-SolutionUpdate.
Updates
In some cases, SolutionUpdate commands could fail if run after the Send-DiagnosticData command.
Make sure to close the PowerShell session used for Send-DiagnosticData. Open a new PowerShell session and use it for SolutionUpdate commands.
Update
In rare instances, when applying an update from 2311.0.24 to 2311.2.4, cluster status reports In Progress instead of expected Failed to update.
Retry the update. If the issue persists, contact Microsoft Support.
Update
Attempts to install solution updates can fail at the end of the CAU steps with: There was a failure in a Common Information Model (CIM) operation, that is, an operation performed by software that Cluster-Aware Updating depends on. This rare issue occurs if the Cluster Name or Cluster IP Address resources fail to start after a node reboot and is most typical in small clusters.
If you encounter this issue, contact Microsoft Support for next steps. They can work with you to manually restart the cluster resources and resume the update as needed.
Update
When applying a cluster update to 10.2402.3.11 the Get-SolutionUpdate cmdlet may not respond and eventually fails with a RequestTimeoutException after approximately 10 minutes. This is likely to occur following an add or repair server scenario.
Use the Start-ClusterGroup and Stop-ClusterGroup cmdlets to restart the update service.
A successful run of these cmdlets should bring the update service online.
Cluster aware updating
Resume node operation failed to resume node.
This is a transient issue and could resolve on its own. Wait for a few minutes and retry the operation. If the issue persists, contact Microsoft Support.
Cluster aware updating
Suspend node operation was stuck for greater than 90 minutes.
This is a transient issue and could resolve on its own. Wait for a few minutes and retry the operation. If the issue persists, contact Microsoft Support.
Issues for version 2402.4
This software release maps to software version number 2402.4.4.
Release notes for this version include the issues fixed in this release, known issues in this release, and release noted issues carried over from previous versions.
Fixed issues
Microsoft isn't aware of any fixed issues in this release.
Known issues in this release
Microsoft isn't aware of any known issues in this release.
Known issues from previous releases
Here are the known issues from previous releases:
Feature
Issue
Workaround
AKS on HCI
AKS cluster creation fails with the Error: Invalid AKS network resource id. This issue can occur when the associated logical network name has an underscore.
Underscores aren't supported in logical network names. Make sure to not use underscore in the names for logical networks deployed on your Azure Stack HCI.
Repair server
In rare instances, the Repair-Server operation fails with the HealthServiceWaitForDriveFW error. In these cases, the old drives from the repaired node aren't removed and new disks are stuck in the maintenance mode.
To prevent this issue, make sure that you DO NOT drain the node either via the Windows Admin Center or using the Suspend-ClusterNode -Drain PowerShell cmdlet before you start Repair-Server. If the issue occurs, contact Microsoft Support for next steps.
Repair server
This issue is seen when the single server Azure Stack HCI is updated from 2311 to 2402 and then the Repair-Server is performed. The repair operation fails.
Before you repair the single node, follow these steps: 1. Run version 2402 for the ADPrepTool. Follow the steps in Prepare Active Directory. This action is quick and adds the required permissions to the Organizational Unit (OU). 2. Move the computer object from Computers segment to the root OU. Run the following command: Get-ADComputer <HOSTNAME> | Move-ADObject -TargetPath "<OU path>"
Deployment
If you prepare the Active Directory on your own (not using the script and procedure provided by Microsoft), your Active Directory validation could fail with missing Generic All permission. This is due to an issue in the validation check that checks for a dedicated permission entry for msFVE-RecoverInformationobjects – General – Permissions Full control, which is required for BitLocker recovery.
Use the Prepare AD script method or if using your own method, make sure to assign the specific permission msFVE-RecoverInformationobjects – General – Permissions Full control.
Deployment
There's a rare issue in this release where the DNS record is deleted during the Azure Stack HCI deployment. When that occurs, the following exception is seen: Type 'PropagatePublicRootCertificate' of Role 'ASCA' raised an exception:<br>The operation on computer 'ASB88RQ22U09' failed: WinRM cannot process the request. The following error occurred while using Kerberos authentication: Cannot find the computer ASB88RQ22U09.local. Verify that the computer exists on the network and that the name provided is spelled correctly at PropagatePublicRootCertificate, C:\NugetStore\Microsoft.AzureStack, at Orchestration.Roles.CertificateAuthority.10.2402.0.14\content\Classes\ASCA\ASCA.psm1: line 38, at C:\CloudDeployment\ECEngine\InvokeInterfaceInternal.psm1: line 127,at Invoke-EceInterfaceInternal, C:\CloudDeployment\ECEngine\InvokeInterfaceInternal.psm1: line 123.
Check the DNS server to see if any DNS records of the cluster nodes are missing. Apply the following mitigation on the nodes where its DNS record is missing.
Restart the DNS client service. Open a PowerShell session and run the following cmdlet on the affected node: Taskkill /f /fi "SERVICES eq dnscache"
Deployment
In this release, there's a remote task failure on a multi-node deployment that results in the following exception: ECE RemoteTask orchestration failure with ASRR1N42R01U31 (node pingable - True): A WebException occurred while sending a RestRequest. WebException.Status: ConnectFailure on [https://<URL>](https://<URL>).
The mitigation is to restart the ECE agent on the affected node. On your server, open a PowerShell session and run the following command: Restart-Service ECEAgent.
Add/Repair server
In this release, when adding or repairing a server, a failure is seen when the software load balancer or network controller VM certificates are being copied from the existing nodes. The failure is because these certificates weren't generated during the deployment/update.
There's no workaround in this release. If you encounter this issue, contact Microsoft Support to determine next steps.
Deployment
In this release, there's a transient issue resulting in the deployment failure with the following exception: Type 'SyncDiagnosticLevel' of Role 'ObservabilityConfig' raised an exception:*<br>*Syncing Diagnostic Level failed with error: The Diagnostic Level does not match. Portal was not set to Enhanced, instead is Basic.
As this is a transient issue, retrying the deployment should fix this. For more information, see how to Rerun the deployment.
Deployment
In this release, there's an issue with the Secrets URI/location field. This is a required field that is marked Not mandatory and results in Azure Resource Manager template deployment failures.
Use the sample parameters file in the Deploy Azure Local via Azure Resource Manager template to ensure that all the inputs are provided in the required format and then try the deployment. If there's a failed deployment, you must also clean up the following resources before you Rerun the deployment: 1. Delete C:\EceStore. 2. Delete C:\CloudDeployment. 3. Delete C:\nugetstore. 4. Remove-Item HKLM:\Software\Microsoft\LCMAzureStackStampInformation.
Security
For new deployments, Secured-core capable devices won't have Dynamic Root of Measurement (DRTM) enabled by default. If you try to enable (DRTM) using the Enable-AzSSecurity cmdlet, you see an error that DRTM setting isn't supported in the current release. Microsoft recommends defense in depth, and UEFI Secure Boot still protects the components in the Static Root of Trust (SRT) boot chain by ensuring that they're loaded only when they're signed and verified.
DRTM isn't supported in this release.
Networking
An environment check fails when a proxy server is used. By design, the bypass list is different for winhttp and wininet, which causes the validation check to fail.
Follow these workaround steps:
1. Clear the proxy bypass list prior to the health check and before starting the deployment or the update.
2. After passing the check, wait for the deployment or update to fail.
3. Set your proxy bypass list again.
Arc VM management
Deployment or update of Arc Resource Bridge could fail when the automatically generated temporary SPN secret during this operation, starts with a hyphen.
Retry the deployment/update. The retry should regenerate the SPN secret and the operation will likely succeed.
Arc VM management
Arc Extensions on Arc VMs stay in "Creating" state indefinitely.
Sign in to the VM, open a command prompt, and type the following: Windows: notepad C:\ProgramData\AzureConnectedMachineAgent\Config\agentconfig.json Linux: sudo vi /var/opt/azcmagent/agentconfig.json Next, find the resourcename property. Delete the GUID that is appended to the end of the resource name, so this property matches the name of the VM. Then restart the VM.
Arc VM management
When a new server is added to an Azure Stack HCI cluster, storage path isn't created automatically for the newly created volume.
You can manually create a storage path for any new volumes. For more information, see Create a storage path.
Arc VM management
Restart of Arc VM operation completes after approximately 20 minutes although the VM itself restarts in about a minute.
There's no known workaround in this release.
Arc VM management
In some instances, the status of the logical network shows as Failed in Azure portal. This occurs when you try to delete the logical network without first deleting any resources such as network interfaces associated with that logical network. You should still be able to create resources on this logical network. The status is misleading in this instance.
If the status of this logical network was Succeeded at the time when this network was provisioned, then you can continue to create resources on this network.
Arc VM management
In this release, when you update a VM with a data disk attached to it using the Azure CLI, the operation fails with the following error message: Couldn't find a virtual hard disk with the name.
In rare instances, you may encounter this error while updating your Azure Stack HCI: Type 'UpdateArbAndExtensions' of Role 'MocArb' raised an exception: Exception Upgrading ARB and Extension in step [UpgradeArbAndExtensions :Get-ArcHciConfig] UpgradeArb: Invalid applianceyaml = [C:\AksHci\hci-appliance.yaml].
If you see this issue, contact Microsoft Support to assist you with the next steps.
Networking
There's an infrequent DNS client issue in this release that causes the deployment to fail on a two-node cluster with a DNS resolution error: A WebException occurred while sending a RestRequest. WebException.Status: NameResolutionFailure. As a result of the bug, the DNS record of the second node is deleted soon after it's created resulting in a DNS error.
Restart the server. This operation registers the DNS record, which prevents it from getting deleted.
Azure portal
In some instances, the Azure portal might take a while to update and the view might not be current.
You might need to wait for 30 minutes or more to see the updated view.
Arc VM management
Deleting a network interface on an Arc VM from Azure portal doesn't work in this release.
Providing the OU name in an incorrect syntax isn't detected in the Azure portal. The incorrect syntax includes unsupported characters such as &,",',<,>. The incorrect syntax is detected at a later step during cluster validation.
Make sure that the OU path syntax is correct and does not include unsupported characters.
Deployment
Deployments via Azure Resource Manager time out after 2 hours. Deployments that exceed 2 hours show up as failed in the resource group though the cluster is successfully created.
To monitor the deployment in the Azure portal, go to the Azure Stack HCI cluster resource and then go to new Deployments entry.
Azure Site Recovery
Azure Site Recovery can't be installed on an Azure Stack HCI cluster in this release.
There's no known workaround in this release.
Update
When updating the Azure Stack HCI cluster via the Azure Update Manager, the update progress and results may not be visible in the Azure portal.
To work around this issue, on each cluster node, add the following registry key (no value needed):
Then on one of the cluster nodes, restart the Cloud Management cluster group.
Stop-ClusterGroup "Cloud Management"
Start-ClusterGroup "Cloud Management"
This won't fully remediate the issue as the progress details may still not be displayed for a duration of the update process. To get the latest update details, you can Retrieve the update progress with PowerShell.
Update
In rare instances, if a failed update is stuck in an In progress state in Azure Update Manager, the Try again button is disabled.
To resume the update, run the following PowerShell command: Get-SolutionUpdate|Start-SolutionUpdate.
Updates
In some cases, SolutionUpdate commands could fail if run after the Send-DiagnosticData command.
Make sure to close the PowerShell session used for Send-DiagnosticData. Open a new PowerShell session and use it for SolutionUpdate commands.
Update
In rare instances, when applying an update from 2311.0.24 to 2311.2.4, cluster status reports In Progress instead of expected Failed to update.
Retry the update. If the issue persists, contact Microsoft Support.
Update
Attempts to install solution updates can fail at the end of the CAU steps with: There was a failure in a Common Information Model (CIM) operation, that is, an operation performed by software that Cluster-Aware Updating depends on. This rare issue occurs if the Cluster Name or Cluster IP Address resources fail to start after a node reboot and is most typical in small clusters.
If you encounter this issue, contact Microsoft Support for next steps. They can work with you to manually restart the cluster resources and resume the update as needed.
Update
When applying a cluster update to 10.2402.4.11 the Get-SolutionUpdate cmdlet may not respond and eventually fails with a RequestTimeoutException after approximately 10 minutes. This is likely to occur following an add or repair server scenario.
Use the Start-ClusterGroup and Stop-ClusterGroup cmdlets to restart the update service.
A successful run of these cmdlets should bring the update service online.
Cluster aware updating
Resume node operation failed to resume node.
This is a transient issue and could resolve on its own. Wait for a few minutes and retry the operation. If the issue persists, contact Microsoft Support.
Cluster aware updating
Suspend node operation was stuck for greater than 90 minutes.
This is a transient issue and could resolve on its own. Wait for a few minutes and retry the operation. If the issue persists, contact Microsoft Support.
Issues for version 2402.3
This software release maps to software version number 2402.3.10.
Release notes for this version include the issues fixed in this release, known issues in this release, and release noted issues carried over from previous versions.
Fixed issues
Microsoft isn't aware of any fixed issues in this release.
Known issues in this release
Microsoft isn't aware of any known issues in this release.
Known issues from previous releases
Here are the known issues from previous releases:
Feature
Issue
Workaround
AKS on HCI
AKS cluster creation fails with the Error: Invalid AKS network resource id. This issue can occur when the associated logical network name has an underscore.
Underscores aren't supported in logical network names. Make sure to not use underscore in the names for logical networks deployed on your Azure Stack HCI.
Repair server
In rare instances, the Repair-Server operation fails with the HealthServiceWaitForDriveFW error. In these cases, the old drives from the repaired node aren't removed and new disks are stuck in the maintenance mode.
To prevent this issue, make sure that you DO NOT drain the node either via the Windows Admin Center or using the Suspend-ClusterNode -Drain PowerShell cmdlet before you start Repair-Server. If the issue occurs, contact Microsoft Support for next steps.
Repair server
This issue is seen when the single server Azure Stack HCI is updated from 2311 to 2402 and then the Repair-Server is performed. The repair operation fails.
Before you repair the single node, follow these steps: 1. Run version 2402 for the ADPrepTool. Follow the steps in Prepare Active Directory. This action is quick and adds the required permissions to the Organizational Unit (OU). 2. Move the computer object from Computers segment to the root OU. Run the following command: Get-ADComputer <HOSTNAME> | Move-ADObject -TargetPath "<OU path>"
Deployment
If you prepare the Active Directory on your own (not using the script and procedure provided by Microsoft), your Active Directory validation could fail with missing Generic All permission. This is due to an issue in the validation check that checks for a dedicated permission entry for msFVE-RecoverInformationobjects – General – Permissions Full control, which is required for BitLocker recovery.
Use the Prepare AD script method or if using your own method, make sure to assign the specific permission msFVE-RecoverInformationobjects – General – Permissions Full control.
Deployment
There's a rare issue in this release where the DNS record is deleted during the Azure Stack HCI deployment. When that occurs, the following exception is seen: Type 'PropagatePublicRootCertificate' of Role 'ASCA' raised an exception:<br>The operation on computer 'ASB88RQ22U09' failed: WinRM cannot process the request. The following error occurred while using Kerberos authentication: Cannot find the computer ASB88RQ22U09.local. Verify that the computer exists on the network and that the name provided is spelled correctly at PropagatePublicRootCertificate, C:\NugetStore\Microsoft.AzureStack, at Orchestration.Roles.CertificateAuthority.10.2402.0.14\content\Classes\ASCA\ASCA.psm1: line 38, at C:\CloudDeployment\ECEngine\InvokeInterfaceInternal.psm1: line 127,at Invoke-EceInterfaceInternal, C:\CloudDeployment\ECEngine\InvokeInterfaceInternal.psm1: line 123.
Check the DNS server to see if any DNS records of the cluster nodes are missing. Apply the following mitigation on the nodes where its DNS record is missing.
Restart the DNS client service. Open a PowerShell session and run the following cmdlet on the affected node: Taskkill /f /fi "SERVICES eq dnscache"
Deployment
In this release, there's a remote task failure on a multi-node deployment that results in the following exception: ECE RemoteTask orchestration failure with ASRR1N42R01U31 (node pingable - True): A WebException occurred while sending a RestRequest. WebException.Status: ConnectFailure on [https://<URL>](https://<URL>).
The mitigation is to restart the ECE agent on the affected node. On your server, open a PowerShell session and run the following command: Restart-Service ECEAgent.
Add/Repair server
In this release, when adding or repairing a server, a failure is seen when the software load balancer or network controller VM certificates are being copied from the existing nodes. The failure is because these certificates weren't generated during the deployment/update.
There's no workaround in this release. If you encounter this issue, contact Microsoft Support to determine next steps.
Deployment
In this release, there's a transient issue resulting in the deployment failure with the following exception: Type 'SyncDiagnosticLevel' of Role 'ObservabilityConfig' raised an exception:*<br>*Syncing Diagnostic Level failed with error: The Diagnostic Level does not match. Portal was not set to Enhanced, instead is Basic.
As this is a transient issue, retrying the deployment should fix this. For more information, see how to Rerun the deployment.
Deployment
In this release, there's an issue with the Secrets URI/location field. This is a required field that is marked Not mandatory and results in Azure Resource Manager template deployment failures.
Use the sample parameters file in the Deploy Azure Local via Azure Resource Manager template to ensure that all the inputs are provided in the required format and then try the deployment. If there's a failed deployment, you must also clean up the following resources before you Rerun the deployment: 1. Delete C:\EceStore. 2. Delete C:\CloudDeployment. 3. Delete C:\nugetstore. 4. Remove-Item HKLM:\Software\Microsoft\LCMAzureStackStampInformation.
Security
For new deployments, Secured-core capable devices won't have Dynamic Root of Measurement (DRTM) enabled by default. If you try to enable (DRTM) using the Enable-AzSSecurity cmdlet, you see an error that DRTM setting isn't supported in the current release. Microsoft recommends defense in depth, and UEFI Secure Boot still protects the components in the Static Root of Trust (SRT) boot chain by ensuring that they're loaded only when they're signed and verified.
DRTM isn't supported in this release.
Networking
An environment check fails when a proxy server is used. By design, the bypass list is different for winhttp and wininet, which causes the validation check to fail.
Follow these workaround steps:
1. Clear the proxy bypass list prior to the health check and before starting the deployment or the update.
2. After passing the check, wait for the deployment or update to fail.
3. Set your proxy bypass list again.
Arc VM management
Deployment or update of Arc Resource Bridge could fail when the automatically generated temporary SPN secret during this operation, starts with a hyphen.
Retry the deployment/update. The retry should regenerate the SPN secret and the operation will likely succeed.
Arc VM management
Arc Extensions on Arc VMs stay in "Creating" state indefinitely.
Sign in to the VM, open a command prompt, and type the following: Windows: notepad C:\ProgramData\AzureConnectedMachineAgent\Config\agentconfig.json Linux: sudo vi /var/opt/azcmagent/agentconfig.json Next, find the resourcename property. Delete the GUID that is appended to the end of the resource name, so this property matches the name of the VM. Then restart the VM.
Arc VM management
When a new server is added to an Azure Stack HCI cluster, storage path isn't created automatically for the newly created volume.
You can manually create a storage path for any new volumes. For more information, see Create a storage path.
Arc VM management
Restart of Arc VM operation completes after approximately 20 minutes although the VM itself restarts in about a minute.
There's no known workaround in this release.
Arc VM management
In some instances, the status of the logical network shows as Failed in Azure portal. This occurs when you try to delete the logical network without first deleting any resources such as network interfaces associated with that logical network. You should still be able to create resources on this logical network. The status is misleading in this instance.
If the status of this logical network was Succeeded at the time when this network was provisioned, then you can continue to create resources on this network.
Arc VM management
In this release, when you update a VM with a data disk attached to it using the Azure CLI, the operation fails with the following error message: Couldn't find a virtual hard disk with the name.
In rare instances, you may encounter this error while updating your Azure Stack HCI: Type 'UpdateArbAndExtensions' of Role 'MocArb' raised an exception: Exception Upgrading ARB and Extension in step [UpgradeArbAndExtensions :Get-ArcHciConfig] UpgradeArb: Invalid applianceyaml = [C:\AksHci\hci-appliance.yaml].
If you see this issue, contact Microsoft Support to assist you with the next steps.
Networking
There's an infrequent DNS client issue in this release that causes the deployment to fail on a two-node cluster with a DNS resolution error: A WebException occurred while sending a RestRequest. WebException.Status: NameResolutionFailure. As a result of the bug, the DNS record of the second node is deleted soon after it's created resulting in a DNS error.
Restart the server. This operation registers the DNS record, which prevents it from getting deleted.
Azure portal
In some instances, the Azure portal might take a while to update and the view might not be current.
You might need to wait for 30 minutes or more to see the updated view.
Arc VM management
Deleting a network interface on an Arc VM from Azure portal doesn't work in this release.
Providing the OU name in an incorrect syntax isn't detected in the Azure portal. The incorrect syntax includes unsupported characters such as &,",',<,>. The incorrect syntax is detected at a later step during cluster validation.
Make sure that the OU path syntax is correct and does not include unsupported characters.
Deployment
Deployments via Azure Resource Manager time out after 2 hours. Deployments that exceed 2 hours show up as failed in the resource group though the cluster is successfully created.
To monitor the deployment in the Azure portal, go to the Azure Stack HCI cluster resource and then go to new Deployments entry.
Azure Site Recovery
Azure Site Recovery can't be installed on an Azure Stack HCI cluster in this release.
There's no known workaround in this release.
Update
When updating the Azure Stack HCI cluster via the Azure Update Manager, the update progress and results may not be visible in the Azure portal.
To work around this issue, on each cluster node, add the following registry key (no value needed):
Then on one of the cluster nodes, restart the Cloud Management cluster group.
Stop-ClusterGroup "Cloud Management"
Start-ClusterGroup "Cloud Management"
This won't fully remediate the issue as the progress details may still not be displayed for a duration of the update process. To get the latest update details, you can Retrieve the update progress with PowerShell.
Update
In rare instances, if a failed update is stuck in an In progress state in Azure Update Manager, the Try again button is disabled.
To resume the update, run the following PowerShell command: Get-SolutionUpdate|Start-SolutionUpdate.
Updates
In some cases, SolutionUpdate commands could fail if run after the Send-DiagnosticData command.
Make sure to close the PowerShell session used for Send-DiagnosticData. Open a new PowerShell session and use it for SolutionUpdate commands.
Update
In rare instances, when applying an update from 2311.0.24 to 2311.2.4, cluster status reports In Progress instead of expected Failed to update.
Retry the update. If the issue persists, contact Microsoft Support.
Update
Attempts to install solution updates can fail at the end of the CAU steps with: There was a failure in a Common Information Model (CIM) operation, that is, an operation performed by software that Cluster-Aware Updating depends on. This rare issue occurs if the Cluster Name or Cluster IP Address resources fail to start after a node reboot and is most typical in small clusters.
If you encounter this issue, contact Microsoft Support for next steps. They can work with you to manually restart the cluster resources and resume the update as needed.
Update
When applying a cluster update to 10.2402.3.11 the Get-SolutionUpdate cmdlet may not respond and eventually fails with a RequestTimeoutException after approximately 10 minutes. This is likely to occur following an add or repair server scenario.
Use the Start-ClusterGroup and Stop-ClusterGroup cmdlets to restart the update service.
A successful run of these cmdlets should bring the update service online.
Cluster aware updating
Resume node operation failed to resume node.
This is a transient issue and could resolve on its own. Wait for a few minutes and retry the operation. If the issue persists, contact Microsoft Support.
Cluster aware updating
Suspend node operation was stuck for greater than 90 minutes.
This is a transient issue and could resolve on its own. Wait for a few minutes and retry the operation. If the issue persists, contact Microsoft Support.
Issues for version 2402.2
This software release maps to software version number 2402.2.12.
Release notes for this version include the issues fixed in this release, known issues in this release, and release noted issues carried over from previous versions.
Fixed issues
Microsoft isn't aware of any fixed issues in this release.
Known issues in this release
Here are the known issues in this release:
Feature
Issue
Workaround/Comments
Updates
Attempts to install solution updates can fail at the end of the CAU steps with: There was a failure in a Common Information Model (CIM) operation, that is, an operation performed by software that Cluster-Aware Updating depends on. This rare issue occurs if the Cluster Name or Cluster IP Address resources fail to start after a node reboot and is most typical in small clusters.
If you encounter this issue, contact Microsoft Support for next steps. They can work with you to manually restart the cluster resources and resume the update as needed.
Updates
When applying a cluster update to 10.2402.2.11 the Get-SolutionUpdate cmdlet may not respond and eventually fails with a RequestTimeoutException after approximately 10 minutes. This is likely to occur following an add or repair server scenario.
Use the Start-ClusterGroup and Stop-ClusterGroup cmdlets to restart the update service.
A successful run of these cmdlets should bring the update service online.
Known issues from previous releases
Here are the known issues from previous releases:
Feature
Issue
Workaround
AKS on HCI
AKS cluster creation fails with the Error: Invalid AKS network resource id. This issue can occur when the associated logical network name has an underscore.
Underscores aren't supported in logical network names. Make sure to not use underscore in the names for logical networks deployed on your Azure Local.
Repair server
In rare instances, the Repair-Server operation fails with the HealthServiceWaitForDriveFW error. In these cases, the old drives from the repaired node aren't removed and new disks are stuck in the maintenance mode.
To prevent this issue, make sure that you DO NOT drain the node either via the Windows Admin Center or using the Suspend-ClusterNode -Drain PowerShell cmdlet before you start Repair-Server. If the issue occurs, contact Microsoft Support for next steps.
Repair server
This issue is seen when the single server Azure Local is updated from 2311 to 2402 and then the Repair-Server is performed. The repair operation fails.
Before you repair the single node, follow these steps: 1. Run version 2402 for the ADPrepTool. Follow the steps in Prepare Active Directory. This action is quick and adds the required permissions to the Organizational Unit (OU). 2. Move the computer object from Computers segment to the root OU. Run the following command: Get-ADComputer <HOSTNAME> | Move-ADObject -TargetPath "<OU path>"
Deployment
If you prepare the Active Directory on your own (not using the script and procedure provided by Microsoft), your Active Directory validation could fail with missing Generic All permission. This is due to an issue in the validation check that checks for a dedicated permission entry for msFVE-RecoverInformationobjects – General – Permissions Full control, which is required for BitLocker recovery.
Use the Prepare AD script method or if using your own method, make sure to assign the specific permission msFVE-RecoverInformationobjects – General – Permissions Full control.
Deployment
There's a rare issue in this release where the DNS record is deleted during the Azure Local deployment. When that occurs, the following exception is seen: Type 'PropagatePublicRootCertificate' of Role 'ASCA' raised an exception:<br>The operation on computer 'ASB88RQ22U09' failed: WinRM cannot process the request. The following error occurred while using Kerberos authentication: Cannot find the computer ASB88RQ22U09.local. Verify that the computer exists on the network and that the name provided is spelled correctly at PropagatePublicRootCertificate, C:\NugetStore\Microsoft.AzureStack, at Orchestration.Roles.CertificateAuthority.10.2402.0.14\content\Classes\ASCA\ASCA.psm1: line 38, at C:\CloudDeployment\ECEngine\InvokeInterfaceInternal.psm1: line 127,at Invoke-EceInterfaceInternal, C:\CloudDeployment\ECEngine\InvokeInterfaceInternal.psm1: line 123.
Check the DNS server to see if any DNS records of the cluster nodes are missing. Apply the following mitigation on the nodes where its DNS record is missing.
Restart the DNS client service. Open a PowerShell session and run the following cmdlet on the affected node: Taskkill /f /fi "SERVICES eq dnscache"
Deployment
In this release, there's a remote task failure on a multi-node deployment that results in the following exception: ECE RemoteTask orchestration failure with ASRR1N42R01U31 (node pingable - True): A WebException occurred while sending a RestRequest. WebException.Status: ConnectFailure on [https://<URL>](https://<URL>).
The mitigation is to restart the ECE agent on the affected node. On your server, open a PowerShell session and run the following command: Restart-Service ECEAgent.
Add/Repair server
In this release, when adding or repairing a server, a failure is seen when the software load balancer or network controller VM certificates are being copied from the existing nodes. The failure is because these certificates weren't generated during the deployment/update.
There's no workaround in this release. If you encounter this issue, contact Microsoft Support to determine next steps.
Deployment
In this release, there's a transient issue resulting in the deployment failure with the following exception: Type 'SyncDiagnosticLevel' of Role 'ObservabilityConfig' raised an exception:*<br>*Syncing Diagnostic Level failed with error: The Diagnostic Level does not match. Portal was not set to Enhanced, instead is Basic.
As this is a transient issue, retrying the deployment should fix this. For more information, see how to Rerun the deployment.
Deployment
In this release, there's an issue with the Secrets URI/location field. This is a required field that is marked Not mandatory and results in Azure Resource Manager template deployment failures.
Use the sample parameters file in the Deploy Azure Local via Azure Resource Manager template to ensure that all the inputs are provided in the required format and then try the deployment. If there's a failed deployment, you must also clean up the following resources before you Rerun the deployment: 1. Delete C:\EceStore. 2. Delete C:\CloudDeployment. 3. Delete C:\nugetstore. 4. Remove-Item HKLM:\Software\Microsoft\LCMAzureStackStampInformation.
Security
For new deployments, Secured-core capable devices won't have Dynamic Root of Measurement (DRTM) enabled by default. If you try to enable (DRTM) using the Enable-AzSSecurity cmdlet, you see an error that DRTM setting isn't supported in the current release. Microsoft recommends defense in depth, and UEFI Secure Boot still protects the components in the Static Root of Trust (SRT) boot chain by ensuring that they're loaded only when they're signed and verified.
DRTM isn't supported in this release.
Networking
The environment checker for 2402 builds still requires to have the same proxy bypass string for WinInet, WinHttp and Environment Variables and fails when a proxy server is used. By design, the bypass list is different for winhttp, wininet and environment variables, which causes the validation check to fail. This issue is fixed in 2405 and later builds.
If you see this issue, contact Microsoft Support to assist you with the next steps.
Arc VM management
Deployment or update of Arc Resource Bridge could fail when the automatically generated temporary SPN secret during this operation, starts with a hyphen.
Retry the deployment/update. The retry should regenerate the SPN secret and the operation will likely succeed.
Arc VM management
Arc Extensions on Arc VMs stay in "Creating" state indefinitely.
Sign in to the VM, open a command prompt, and type the following: Windows: notepad C:\ProgramData\AzureConnectedMachineAgent\Config\agentconfig.json Linux: sudo vi /var/opt/azcmagent/agentconfig.json Next, find the resourcename property. Delete the GUID that is appended to the end of the resource name, so this property matches the name of the VM. Then restart the VM.
Arc VM management
When a new server is added to an Azure Local instance, storage path isn't created automatically for the newly created volume.
You can manually create a storage path for any new volumes. For more information, see Create a storage path.
Arc VM management
Restart of Arc VM operation completes after approximately 20 minutes although the VM itself restarts in about a minute.
There's no known workaround in this release.
Arc VM management
In some instances, the status of the logical network shows as Failed in Azure portal. This occurs when you try to delete the logical network without first deleting any resources such as network interfaces associated with that logical network. You should still be able to create resources on this logical network. The status is misleading in this instance.
If the status of this logical network was Succeeded at the time when this network was provisioned, then you can continue to create resources on this network.
Arc VM management
In this release, when you update a VM with a data disk attached to it using the Azure CLI, the operation fails with the following error message: Couldn't find a virtual hard disk with the name.
In rare instances, you may encounter this error while updating your Azure Local: Type 'UpdateArbAndExtensions' of Role 'MocArb' raised an exception: Exception Upgrading ARB and Extension in step [UpgradeArbAndExtensions :Get-ArcHciConfig] UpgradeArb: Invalid applianceyaml = [C:\AksHci\hci-appliance.yaml].
If you see this issue, contact Microsoft Support to assist you with the next steps.
Networking
There's an infrequent DNS client issue in this release that causes the deployment to fail on a two-node cluster with a DNS resolution error: A WebException occurred while sending a RestRequest. WebException.Status: NameResolutionFailure. As a result of the bug, the DNS record of the second node is deleted soon after it's created resulting in a DNS error.
Restart the server. This operation registers the DNS record, which prevents it from getting deleted.
Azure portal
In some instances, the Azure portal might take a while to update and the view might not be current.
You might need to wait for 30 minutes or more to see the updated view.
Arc VM management
Deleting a network interface on an Arc VM from Azure portal doesn't work in this release.
Providing the OU name in an incorrect syntax isn't detected in the Azure portal. The incorrect syntax includes unsupported characters such as &,",',<,>. The incorrect syntax is detected at a later step during cluster validation.
Make sure that the OU path syntax is correct and does not include unsupported characters.
Deployment
Deployments via Azure Resource Manager time out after 2 hours. Deployments that exceed 2 hours show up as failed in the resource group though the cluster is successfully created.
To monitor the deployment in the Azure portal, go to the Azure Local cluster resource and then go to new Deployments entry.
Azure Site Recovery
Azure Site Recovery can't be installed on an Azure Local in this release.
There's no known workaround in this release.
Update
When updating the Azure Local instance via the Azure Update Manager, the update progress and results may not be visible in the Azure portal.
To work around this issue, on each cluster node, add the following registry key (no value needed):
Then on one of the cluster nodes, restart the Cloud Management cluster group.
Stop-ClusterGroup "Cloud Management"
Start-ClusterGroup "Cloud Management"
This won't fully remediate the issue as the progress details may still not be displayed for a duration of the update process. To get the latest update details, you can Retrieve the update progress with PowerShell.
Updates
In rare instances, if a failed update is stuck in an In progress state in Azure Update Manager, the Try again button is disabled.
To resume the update, run the following PowerShell command: Get-SolutionUpdate|Start-SolutionUpdate.
Updates
In some cases, SolutionUpdate commands could fail if run after the Send-DiagnosticData command.
Make sure to close the PowerShell session used for Send-DiagnosticData. Open a new PowerShell session and use it for SolutionUpdate commands.
Updates
In rare instances, when applying an update from 2311.0.24 to 2311.2.4, cluster status reports In Progress instead of expected Failed to update.
Retry the update. If the issue persists, contact Microsoft Support.
Cluster aware updating
Resume node operation failed to resume node.
This is a transient issue and could resolve on its own. Wait for a few minutes and retry the operation. If the issue persists, contact Microsoft Support.
Cluster aware updating
Suspend node operation was stuck for greater than 90 minutes.
This is a transient issue and could resolve on its own. Wait for a few minutes and retry the operation. If the issue persists, contact Microsoft Support.
Issues for version 2402.1
This software release maps to software version number 10.2402.1.5.
Release notes for this version include the issues fixed in this release, known issues in this release, and release noted issues carried over from previous versions.
Fixed issues
Here are the issues fixed in this release:
Feature
Issue
Workaround/Comments
Updates
In this release, there's a health check issue owing to which a single server Azure Local can't be updated from the Azure portal.
When you create a disk or a network interface in this release with underscore in the name, the operation fails.
Make sure to not use underscore in the names for disks or network interfaces.
Arc VM management
If the resource group used to deploy an Arc VM on your Azure Local has an underscore in the name, the guest agent installation fails. As a result, you won't be able to enable guest management.
Make sure that there are no underscores in the resource groups used to deploy Arc VMs.
Known issues in this release
Here are the known issues in this release:
Feature
Issue
Workaround/Comments
AKS on HCI
AKS cluster creation fails with the Error: Invalid AKS network resource id. This issue can occur when the associated logical network name has an underscore.
Underscores aren't supported in logical network names. Make sure to not use underscore in the names for logical networks deployed on your Azure Local.
Known issues from previous releases
Here are the known issues from previous releases:
Feature
Issue
Workaround
Repair server
In rare instances, the Repair-Server operation fails with the HealthServiceWaitForDriveFW error. In these cases, the old drives from the repaired node aren't removed and new disks are stuck in the maintenance mode.
To prevent this issue, make sure that you DO NOT drain the node either via the Windows Admin Center or using the Suspend-ClusterNode -Drain PowerShell cmdlet before you start Repair-Server. If the issue occurs, contact Microsoft Support for next steps.
Repair server
This issue is seen when the single server Azure Local is updated from 2311 to 2402 and then the Repair-Server is performed. The repair operation fails.
Before you repair the single node, follow these steps: 1. Run version 2402 for the ADPrepTool. Follow the steps in Prepare Active Directory. This action is quick and adds the required permissions to the Organizational Unit (OU). 2. Move the computer object from Computers segment to the root OU. Run the following command: Get-ADComputer <HOSTNAME> | Move-ADObject -TargetPath "<OU path>"
Deployment
If you prepare the Active Directory on your own (not using the script and procedure provided by Microsoft), your Active Directory validation could fail with missing Generic All permission. This is due to an issue in the validation check that checks for a dedicated permission entry for msFVE-RecoverInformationobjects – General – Permissions Full control, which is required for BitLocker recovery.
Use the Prepare AD script method or if using your own method, make sure to assign the specific permission msFVE-RecoverInformationobjects – General – Permissions Full control.
Deployment
There's a rare issue in this release where the DNS record is deleted during the Azure Local deployment. When that occurs, the following exception is seen: Type 'PropagatePublicRootCertificate' of Role 'ASCA' raised an exception:<br>The operation on computer 'ASB88RQ22U09' failed: WinRM cannot process the request. The following error occurred while using Kerberos authentication: Cannot find the computer ASB88RQ22U09.local. Verify that the computer exists on the network and that the name provided is spelled correctly at PropagatePublicRootCertificate, C:\NugetStore\Microsoft.AzureStack, at Orchestration.Roles.CertificateAuthority.10.2402.0.14\content\Classes\ASCA\ASCA.psm1: line 38, at C:\CloudDeployment\ECEngine\InvokeInterfaceInternal.psm1: line 127,at Invoke-EceInterfaceInternal, C:\CloudDeployment\ECEngine\InvokeInterfaceInternal.psm1: line 123.
Check the DNS server to see if any DNS records of the cluster nodes are missing. Apply the following mitigation on the nodes where its DNS record is missing.
Restart the DNS client service. Open a PowerShell session and run the following cmdlet on the affected node: Taskkill /f /fi "SERVICES eq dnscache"
Deployment
In this release, there's a remote task failure on a multi-node deployment that results in the following exception: ECE RemoteTask orchestration failure with ASRR1N42R01U31 (node pingable - True): A WebException occurred while sending a RestRequest. WebException.Status: ConnectFailure on [https://<URL>](https://<URL>).
The mitigation is to restart the ECE agent on the affected node. On your server, open a PowerShell session and run the following command: Restart-Service ECEAgent.
Add/Repair server
In this release, when adding or repairing a server, a failure is seen when the software load balancer or network controller VM certificates are being copied from the existing nodes. The failure is because these certificates weren't generated during the deployment/update.
There's no workaround in this release. If you encounter this issue, contact Microsoft Support to determine next steps.
Deployment
In this release, there's a transient issue resulting in the deployment failure with the following exception: Type 'SyncDiagnosticLevel' of Role 'ObservabilityConfig' raised an exception:*<br>*Syncing Diagnostic Level failed with error: The Diagnostic Level does not match. Portal was not set to Enhanced, instead is Basic.
As this is a transient issue, retrying the deployment should fix this. For more information, see how to Rerun the deployment.
Deployment
In this release, there's an issue with the Secrets URI/location field. This is a required field that is marked Not mandatory and results in Azure Resource Manager template deployment failures.
Use the sample parameters file in the Deploy Azure Local via Azure Resource Manager template to ensure that all the inputs are provided in the required format and then try the deployment. If there's a failed deployment, you must also clean up the following resources before you Rerun the deployment: 1. Delete C:\EceStore. 2. Delete C:\CloudDeployment. 3. Delete C:\nugetstore. 4. Remove-Item HKLM:\Software\Microsoft\LCMAzureStackStampInformation.
Security
For new deployments, Secured-core capable devices won't have Dynamic Root of Measurement (DRTM) enabled by default. If you try to enable (DRTM) using the Enable-AzSSecurity cmdlet, you'll see an error that DRTM setting isn't supported in the current release. Microsoft recommends defense in depth, and UEFI Secure Boot still protects the components in the Static Root of Trust (SRT) boot chain by ensuring that they're loaded only when they're signed and verified.
DRTM isn't supported in this release.
Networking
An environment check fails when a proxy server is used. By design, the bypass list is different for winhttp and wininet, which causes the validation check to fail.
Follow these workaround steps:
1. Clear the proxy bypass list prior to the health check and before starting the deployment or the update.
2. After passing the check, wait for the deployment or update to fail.
3. Set your proxy bypass list again.
Arc VM management
Deployment or update of Arc Resource Bridge could fail when the automatically generated temporary SPN secret during this operation, starts with a hyphen.
Retry the deployment/update. The retry should regenerate the SPN secret and the operation will likely succeed.
Arc VM management
Arc Extensions on Arc VMs stay in "Creating" state indefinitely.
Sign in to the VM, open a command prompt, and type the following: Windows: notepad C:\ProgramData\AzureConnectedMachineAgent\Config\agentconfig.json Linux: sudo vi /var/opt/azcmagent/agentconfig.json Next, find the resourcename property. Delete the GUID that is appended to the end of the resource name, so this property matches the name of the VM. Then restart the VM.
Arc VM management
When a new server is added to an Azure Local instance, storage path isn't created automatically for the newly created volume.
You can manually create a storage path for any new volumes. For more information, see Create a storage path.
Arc VM management
Restart of Arc VM operation completes after approximately 20 minutes although the VM itself restarts in about a minute.
There's no known workaround in this release.
Arc VM management
In some instances, the status of the logical network shows as Failed in Azure portal. This occurs when you try to delete the logical network without first deleting any resources such as network interfaces associated with that logical network. You should still be able to create resources on this logical network. The status is misleading in this instance.
If the status of this logical network was Succeeded at the time when this network was provisioned, then you can continue to create resources on this network.
Arc VM management
In this release, when you update a VM with a data disk attached to it using the Azure CLI, the operation fails with the following error message: Couldn't find a virtual hard disk with the name.
In rare instances, you may encounter this error while updating your Azure Local: Type 'UpdateArbAndExtensions' of Role 'MocArb' raised an exception: Exception Upgrading ARB and Extension in step [UpgradeArbAndExtensions :Get-ArcHciConfig] UpgradeArb: Invalid applianceyaml = [C:\AksHci\hci-appliance.yaml].
If you see this issue, contact Microsoft Support to assist you with the next steps.
Networking
There's an infrequent DNS client issue in this release that causes the deployment to fail on a two-node cluster with a DNS resolution error: A WebException occurred while sending a RestRequest. WebException.Status: NameResolutionFailure. As a result of the bug, the DNS record of the second node is deleted soon after it's created resulting in a DNS error.
Restart the server. This operation registers the DNS record, which prevents it from getting deleted.
Azure portal
In some instances, the Azure portal might take a while to update and the view might not be current.
You might need to wait for 30 minutes or more to see the updated view.
Arc VM management
Deleting a network interface on an Arc VM from Azure portal doesn't work in this release.
Providing the OU name in an incorrect syntax isn't detected in the Azure portal. The incorrect syntax includes unsupported characters such as &,",',<,>. The incorrect syntax is detected at a later step during cluster validation.
Make sure that the OU path syntax is correct and does not include unsupported characters.
Deployment
Deployments via Azure Resource Manager time out after 2 hours. Deployments that exceed 2 hours show up as failed in the resource group though the cluster is successfully created.
To monitor the deployment in the Azure portal, go to the Azure Local cluster resource and then go to new Deployments entry.
Azure Site Recovery
Azure Site Recovery can't be installed on an Azure Local instance in this release.
There's no known workaround in this release.
Update
When updating the Azure Local instance via the Azure Update Manager, the update progress and results may not be visible in the Azure portal.
To work around this issue, on each cluster node, add the following registry key (no value needed):
Then on one of the cluster nodes, restart the Cloud Management cluster group.
Stop-ClusterGroup "Cloud Management"
Start-ClusterGroup "Cloud Management"
This won't fully remediate the issue as the progress details may still not be displayed for a duration of the update process. To get the latest update details, you can Retrieve the update progress with PowerShell.
Updates
In rare instances, if a failed update is stuck in an In progress state in Azure Update Manager, the Try again button is disabled.
To resume the update, run the following PowerShell command: Get-SolutionUpdate|Start-SolutionUpdate.
Updates
In some cases, SolutionUpdate commands could fail if run after the Send-DiagnosticData command.
Make sure to close the PowerShell session used for Send-DiagnosticData. Open a new PowerShell session and use it for SolutionUpdate commands.
Updates
In rare instances, when applying an update from 2311.0.24 to 2311.2.4, cluster status reports In Progress instead of expected Failed to update.
Retry the update. If the issue persists, contact Microsoft Support.
Cluster aware updating
Resume node operation failed to resume node.
This is a transient issue and could resolve on its own. Wait for a few minutes and retry the operation. If the issue persists, contact Microsoft Support.
Cluster aware updating
Suspend node operation was stuck for greater than 90 minutes.
This is a transient issue and could resolve on its own. Wait for a few minutes and retry the operation. If the issue persists, contact Microsoft Support.
Issues for version 2402
This software release maps to software version number 10.2402.0.23.
Release notes for this version include the issues fixed in this release, known issues in this release, and release noted issues carried over from previous versions.
Fixed issues
Here are the issues fixed in this release:
Feature
Issue
Workaround/Comments
Deployment
The first deployment step: Before Cloud Deployment when Deploying via Azure portal can take from 45 minutes to an hour to complete.
Deployment
There's a sporadic heartbeat reliability issue in this release due to which the registration encounters the error: HCI registration failed. Error: Arc integration failed.
This issue is intermittent. Try rerunning the deployment. For more information, see Rerun the deployment.
Deployment
There's an intermittent issue in this release where the Arc integration validation fails with this error: Validator failed. Can't retrieve the dynamic parameters for the cmdlet. PowerShell Gallery is currently unavailable. Please try again later.
This issue is intermittent. Try rerunning the deployment. For more information, see Rerun the deployment.
Deployment
In some instances, running the Arc registration script doesn't install the mandatory extensions, Azure Edge device Management or Azure Edge Lifecycle Manager.
The issue was fixed in this release. The extensions remediate themselves and get into a successful deployment state.
Update
When you try to change your AzureStackLCMUserPassword using command: Set-AzureStackLCMUserPassword, you might encounter this error:
Can't find an object with identity: 'object id'*.
There's no known workaround in this release.
Known issues in this release
Here are the known issues in this release:
Feature
Issue
Workaround/Comments
Repair server
In rare instances, the Repair-Server operation fails with the HealthServiceWaitForDriveFW error. In these cases, the old drives from the repaired node aren't removed and new disks are stuck in the maintenance mode.
To prevent this issue, make sure that you DO NOT drain the node either via the Windows Admin Center or using the Suspend-ClusterNode -Drain PowerShell cmdlet before you start Repair-Server. If the issue occurs, contact Microsoft Support for next steps.
Repair server
This issue is seen when the single server Azure Stack HCI is updated from 2311 to 2402 and then the Repair-Server is performed. The repair operation fails.
Before you repair the single node, follow these steps: 1. Run version 2402 for the ADPrepTool. Follow the steps in Prepare Active Directory. This action is quick and adds the required permissions to the Organizational Unit (OU). 2. Move the computer object from Computers segment to the root OU. Run the following command: Get-ADComputer <HOSTNAME> | Move-ADObject -TargetPath "<OU path>"
Deployment
If you prepare the Active Directory on your own (not using the script and procedure provided by Microsoft), your Active Directory validation could fail with missing Generic All permission. This is due to an issue in the validation check that checks for a dedicated permission entry for msFVE-RecoverInformationobjects – General – Permissions Full control, which is required for BitLocker recovery.
Use the Prepare AD script method or if using your own method, make sure to assign the specific permission msFVE-RecoverInformationobjects – General – Permissions Full control.
Deployment
There's a rare issue in this release where the DNS record is deleted during the Azure Stack HCI deployment. When that occurs, the following exception is seen: Type 'PropagatePublicRootCertificate' of Role 'ASCA' raised an exception:<br>The operation on computer 'ASB88RQ22U09' failed: WinRM cannot process the request. The following error occurred while using Kerberos authentication: Cannot find the computer ASB88RQ22U09.local. Verify that the computer exists on the network and that the name provided is spelled correctly at PropagatePublicRootCertificate, C:\NugetStore\Microsoft.AzureStack, at Orchestration.Roles.CertificateAuthority.10.2402.0.14\content\Classes\ASCA\ASCA.psm1: line 38, at C:\CloudDeployment\ECEngine\InvokeInterfaceInternal.psm1: line 127,at Invoke-EceInterfaceInternal, C:\CloudDeployment\ECEngine\InvokeInterfaceInternal.psm1: line 123.
Check the DNS server to see if any DNS records of the cluster nodes are missing. Apply the following mitigation on the nodes where its DNS record is missing.
Restart the DNS client service. Open a PowerShell session and run the following cmdlet on the affected node: Taskkill /f /fi "SERVICES eq dnscache"
Deployment
In this release, there's a remote task failure on a multi-node deployment that results in the following exception: ECE RemoteTask orchestration failure with ASRR1N42R01U31 (node pingable - True): A WebException occurred while sending a RestRequest. WebException.Status: ConnectFailure on [https://<URL>](https://<URL>).
The mitigation is to restart the ECE agent on the affected node. On your server, open a PowerShell session and run the following command: Restart-Service ECEAgent.
Updates
In this release, there's a health check issue owing to which a single server Azure Stack HCI can't be updated from the Azure portal.
In this release, when adding or repairing a server, a failure is seen when the software load balancer or network controller VM certificates are being copied from the existing nodes. The failure is because these certificates weren't generated during the deployment/update.
There's no workaround in this release. If you encounter this issue, contact Microsoft Support to determine next steps.
Deployment
In this release, there's a transient issue resulting in the deployment failure with the following exception: Type 'SyncDiagnosticLevel' of Role 'ObservabilityConfig' raised an exception:*<br>*Syncing Diagnostic Level failed with error: The Diagnostic Level does not match. Portal was not set to Enhanced, instead is Basic.
As this is a transient issue, retrying the deployment should fix this. For more information, see how to Rerun the deployment.
Deployment
In this release, there's an issue with the Secrets URI/location field. This is a required field that is marked Not mandatory and results in the Azure Resource Manager template deployment failures.
Use the sample parameters file in the Deploy Azure Local via Azure Resource Manager template to ensure that all the inputs are provided in the required format and then try the deployment. If there's a failed deployment, you must also clean up the following resources before you Rerun the deployment: 1. Delete C:\EceStore. 2. Delete C:\CloudDeployment. 3. Delete C:\nugetstore. 4. Remove-Item HKLM:\Software\Microsoft\LCMAzureStackStampInformation.
Security
For new deployments, Secured-core capable devices won't have Dynamic Root of Measurement (DRTM) enabled by default. If you try to enable (DRTM) using the Enable-AzSSecurity cmdlet, you'll see an error that DRTM setting isn't supported in the current release. Microsoft recommends defense in depth, and UEFI Secure Boot still protects the components in the Static Root of Trust (SRT) boot chain by ensuring that they're loaded only when they're signed and verified.
DRTM is not supported in this release.
Networking
An environment check fails when a proxy server is used. By design, the bypass list is different for winhttp and wininet, which causes the validation check to fail.
Follow these workaround steps:
1. Clear the proxy bypass list prior to the health check and before starting the deployment or the update.
2. After passing the check, wait for the deployment or update to fail.
3. Set your proxy bypass list again.
Known issues from previous releases
Here are the known issues from previous releases:
Feature
Issue
Workaround
Arc VM management
Deployment or update of Arc Resource Bridge could fail when the automatically generated temporary SPN secret during this operation, starts with a hyphen.
Retry the deployment/update. The retry should regenerate the SPN secret and the operation will likely succeed.
Arc VM management
Arc Extensions on Arc VMs stay in "Creating" state indefinitely.
Sign in to the VM, open a command prompt, and type the following: Windows: notepad C:\ProgramData\AzureConnectedMachineAgent\Config\agentconfig.json Linux: sudo vi /var/opt/azcmagent/agentconfig.json Next, find the resourcename property. Delete the GUID that is appended to the end of the resource name, so this property matches the name of the VM. Then restart the VM.
Arc VM management
When a new server is added to an Azure Stack HCI cluster, storage path isn't created automatically for the newly created volume.
You can manually create a storage path for any new volumes. For more information, see Create a storage path.
Arc VM management
Restart of Arc VM operation completes after approximately 20 minutes although the VM itself restarts in about a minute.
There's no known workaround in this release.
Arc VM management
In some instances, the status of the logical network shows as Failed in Azure portal. This occurs when you try to delete the logical network without first deleting any resources such as network interfaces associated with that logical network. You should still be able to create resources on this logical network. The status is misleading in this instance.
If the status of this logical network was Succeeded at the time when this network was provisioned, then you can continue to create resources on this network.
Arc VM management
In this release, when you update a VM with a data disk attached to it using the Azure CLI, the operation fails with the following error message: Couldn't find a virtual hard disk with the name.
In rare instances, you may encounter this error while updating your Azure Stack HCI: Type 'UpdateArbAndExtensions' of Role 'MocArb' raised an exception: Exception Upgrading ARB and Extension in step [UpgradeArbAndExtensions :Get-ArcHciConfig] UpgradeArb: Invalid applianceyaml = [C:\AksHci\hci-appliance.yaml].
If you see this issue, contact Microsoft Support to assist you with the next steps.
Networking
There's an infrequent DNS client issue in this release that causes the deployment to fail on a two-node cluster with a DNS resolution error: A WebException occurred while sending a RestRequest. WebException.Status: NameResolutionFailure. As a result of the bug, the DNS record of the second node is deleted soon after it's created resulting in a DNS error.
Restart the server. This operation registers the DNS record, which prevents it from getting deleted.
Azure portal
In some instances, the Azure portal might take a while to update and the view might not be current.
You might need to wait for 30 minutes or more to see the updated view.
Arc VM management
Deleting a network interface on an Arc VM from Azure portal doesn't work in this release.
When you create a disk or a network interface in this release with underscore in the name, the operation fails.
Make sure to not use underscore in the names for disks or network interfaces.
Deployment
Providing the OU name in an incorrect syntax isn't detected in the Azure portal. The incorrect syntax includes unsupported characters such as &,",',<,>. The incorrect syntax is detected at a later step during cluster validation.
Make sure that OU path syntax is correct and does not include unsupported characters.
Deployment
Deployments via Azure Resource Manager time out after 2 hours. Deployments that exceed 2 hours show up as failed in the resource group though the cluster is successfully created.
To monitor the deployment in the Azure portal, go to the Azure Stack HCI cluster resource and then go to new Deployments entry.
Azure Site Recovery
Azure Site Recovery can't be installed on an Azure Stack HCI cluster in this release.
There's no known workaround in this release.
Update
When updating the Azure Stack HCI cluster via the Azure Update Manager, the update progress and results may not be visible in the Azure portal.
To work around this issue, on each cluster node, add the following registry key (no value needed):
Then on one of the cluster nodes, restart the Cloud Management cluster group.
Stop-ClusterGroup "Cloud Management"
Start-ClusterGroup "Cloud Management"
This won't fully remediate the issue as the progress details may still not be displayed for a duration of the update process. To get the latest update details, you can Retrieve the update progress with PowerShell.
Update
In this release, if you run the Test-CauRun cmdlet prior to actually applying the 2311.2 update, you see an error message regarding a missing firewall rule to remotely shut down the Azure Stack HCI system.
No action is required on your part as the missing rule is automatically created when 2311.2 updates are applied.
When applying future updates, make sure to run the Get-SolutionUpdateEnvironment cmdlet instead of Test-CauRun.
Updates
In rare instances, if a failed update is stuck in an In progress state in Azure Update Manager, the Try again button is disabled.
To resume the update, run the following PowerShell command: Get-SolutionUpdate|Start-SolutionUpdate.
Updates
In some cases, SolutionUpdate commands could fail if run after the Send-DiagnosticData command.
Make sure to close the PowerShell session used for Send-DiagnosticData. Open a new PowerShell session and use it for SolutionUpdate commands.
Updates
In rare instances, when applying an update from 2311.0.24 to 2311.2.4, cluster status reports In Progress instead of expected Failed to update.
Retry the update. If the issue persists, contact Microsoft Support.
Arc VM management
If the resource group used to deploy an Arc VM on your Azure Stack HCI has an underscore in the name, the guest agent installation fails. As a result, you won't be able to enable guest management.
Make sure that there are no underscores in the resource groups used to deploy Arc VMs.
Cluster aware updating
Resume node operation failed to resume node.
This is a transient issue and could resolve on its own. Wait for a few minutes and retry the operation. If the issue persists, contact Microsoft Support.
Cluster aware updating
Suspend node operation was stuck for greater than 90 minutes.
This is a transient issue and could resolve on its own. Wait for a few minutes and retry the operation. If the issue persists, contact Microsoft Support.
Issues for version 2311.5
This software release maps to software version number 10.2311.5.6. This release only supports updates from 2311 release.
Release notes for this version include the issues fixed in this release, known issues in this release, and release noted issues carried over from previous versions.
Fixed issues
Microsoft isn't currently aware of any fixed issues with this release.
Known issues in this release
Here are the known issues in this release:
Feature
Issue
Workaround/Comments
Update
Download operations don't terminate after the specified timeout of 6 hours.
Microsoft is actively working to resolve this issue. If you encounter this issue, contact Microsoft Support for next steps.
Update
Update is in Failed state: DownloadFailed. Summary XML from ECE not present.
Microsoft is actively working to resolve this issue. If you encounter this issue, contact Microsoft Support for next steps.
Updates
When applying a cluster update to 10.2311.5.6 the Get-SolutionUpdate cmdlet may not respond and eventually fails with a RequestTimeoutException after approximately 10 minutes. This is likely to occur following an add or repair server scenario.
Use the Start-ClusterGroup and Stop-ClusterGroup cmdlets to restart the update service.
Get-ClusterGroup -Name "Azure Local Update Service Cluster Group" | Stop-ClusterGroup
Get-ClusterGroup -Name "Azure Local Update Service Cluster Group" | Start-ClusterGroup
A successful run of these cmdlets should bring the update service online.
Known issues from previous releases
Here are the known issues from previous releases:
Feature
Issue
Workaround/Comments
Arc VM management
When a new node is added into the cluster, it fails with: Node add failed at ERROR: An older version of the Arc VM cluster extension is installed on your cluster.
Make sure you have the latest firewall requirements including endpoint https://hciarcvmscontainerregistry.azurecr.io on port 443. This endpoint is required for Azure Local Arc VM container registry.
Arc VM management
When you update to 10.2311.4.5, the arcbridge status shows "UpgradeFailed" on the Azure portal.
Make sure you have the latest firewall requirements including endpoint https://hciarcvmscontainerregistry.azurecr.io on port 443. This endpoint is required for Azure Local Arc VM container registry.
Arc VM management
Deployment or update of Arc Resource Bridge could fail when the automatically generated temporary SPN secret during this operation, starts with a hyphen.
Retry the deployment/update. The retry should regenerate the SPN secret and the operation will likely succeed.
Arc VM management
Arc Extensions on Arc VMs stay in "Creating" state indefinitely.
Sign in to the VM, open a command prompt, and type the following: Windows: notepad C:\ProgramData\AzureConnectedMachineAgent\Config\agentconfig.json Linux: sudo vi /var/opt/azcmagent/agentconfig.json Next, find the resourcename property. Delete the GUID that is appended to the end of the resource name, so this property matches the name of the VM. Then restart the VM.
Arc VM management
When a new server is added to an Azure Local instance, storage path isn't created automatically for the newly created volume.
You can manually create a storage path for any new volumes. For more information, see Create a storage path.
Arc VM management
Restart of Arc VM operation completes after approximately 20 minutes although the VM itself restarts in about a minute.
There's no known workaround in this release.
Arc VM management
In some instances, the status of the logical network shows as Failed in Azure portal. This occurs when you try to delete the logical network without first deleting any resources such as network interfaces associated with that logical network. You should still be able to create resources on this logical network. The status is misleading in this instance.
If the status of this logical network was Succeeded at the time when this network was provisioned, then you can continue to create resources on this network.
Arc VM management
In this release, when you update a VM with a data disk attached to it using the Azure CLI, the operation fails with the following error message: Couldn't find a virtual hard disk with the name.
There's a sporadic heartbeat reliability issue in this release due to which the registration encounters the error: HCI registration failed. Error: Arc integration failed.
This issue is intermittent. Try rerunning the deployment. For more information, see Rerun the deployment.
Deployment
There's an intermittent issue in this release where the Arc integration validation fails with this error: Validator failed. Can't retrieve the dynamic parameters for the cmdlet. PowerShell Gallery is currently unavailable. Please try again later.
This issue is intermittent. Try rerunning the deployment. For more information, see Rerun the deployment.
Deployment
The AzureEdgeRemoteSupport extension shows as "Failed" in the cluster view and "Succeeded" in the node view. Additionally, the node view displays an incorrect extension name, "AzureEdgeEdgeRemoteSupport".
This issue is cosmetic and doesn't impact extension functionality. You may want to follow these steps to manually mitigate the issue:
1. In the Azure portal, navigate to the resource group for your nodes.
2. Go to each Arc node and uninstall the Remote Support extension.
3. Allow up to 12 hours for the Azure Local Resource Provider to update the extensions.
This procedure enables the reinstallation of the extension, ensuring it displays the correct name, AzureEdgeRemoteSupport, and resolves any failures observed in the cluster view.
Optionally, you can use the cmdlet sync-azurestackhci to force a sync on any of the cluster nodes.
Update
In rare instances, you may encounter this error while updating your Azure Local: Type 'UpdateArbAndExtensions' of Role 'MocArb' raised an exception: Exception Upgrading ARB and Extension in step [UpgradeArbAndExtensions :Get-ArcHciConfig] UpgradeArb: Invalid applianceyaml = [C:\AksHci\hci-appliance.yaml].
If you see this issue, contact Microsoft Support to assist you with the next steps.
Update
In this release, there's a health check issue preventing a single server running Azure Local from being updated via the Azure portal.
There's an infrequent DNS client issue in this release that causes the deployment to fail on a two-node cluster with a DNS resolution error: A WebException occurred while sending a RestRequest. WebException.Status: NameResolutionFailure. As a result of the bug, the DNS record of the second node is deleted soon after it's created resulting in a DNS error.
Restart the server. This operation registers the DNS record, which prevents it from getting deleted.
Azure portal
In some instances, the Azure portal might take a while to update and the view might not be current.
You might need to wait for 30 minutes or more to see the updated view.
Arc VM management
Deleting a network interface on an Arc VM from Azure portal doesn't work in this release.
Providing the OU name in an incorrect syntax isn't detected in the Azure portal. The incorrect syntax is however detected at a later step during cluster validation.
There's no known workaround in this release.
Deployment
In some instances, running the Arc registration script doesn't install the mandatory extensions, Azure Edge device Management or Azure Edge Lifecycle Manager.
The workaround is to run the script again and make sure that all the mandatory extensions are installed before you Deploy via Azure portal.
Deployment
The first deployment step: Before Cloud Deployment when Deploying via Azure portal can take from 45 minutes to an hour to complete.
Deployment
Deployments via Azure Resource Manager time out after 2 hours. Deployments that exceed 2 hours show up as failed in the resource group though the cluster is successfully created.
To monitor the deployment in the Azure portal, go to the Azure Local sresource and then go to new Deployments entry.
Azure Site Recovery
Azure Site Recovery can't be installed on an Azure Local instance in this release.
There's no known workaround in this release.
Update
When updating the Azure Local instance via the Azure Update Manager, the update progress and results may not be visible in the Azure portal.
To work around this issue, on each cluster node, add the following registry key (no value needed):
Then on one of the cluster nodes, restart the Cloud Management cluster group.
Stop-ClusterGroup "Cloud Management"
Start-ClusterGroup "Cloud Management"
This won't fully remediate the issue as the progress details may still not be displayed for a duration of the update process. To get the latest update details, you can Retrieve the update progress with PowerShell.
Updates
In rare instances, if a failed update is stuck in an In progress state in Azure Update Manager, the Try again button is disabled.
To resume the update, run the following PowerShell command: Get-SolutionUpdate|Start-SolutionUpdate.
Updates
In some cases, SolutionUpdate commands could fail if run after the Send-DiagnosticData command.
Make sure to close the PowerShell session used for Send-DiagnosticData. Open a new PowerShell session and use it for SolutionUpdate commands.
Updates
In rare instances, when applying an update from 2311.0.24 to 2311.2.4, cluster status reports In Progress instead of expected Failed to update.
Retry the update. If the issue persists, contact Microsoft Support.
Cluster aware updating
Resume node operation failed to resume node.
This is a transient issue and could resolve on its own. Wait for a few minutes and retry the operation. If the issue persists, contact Microsoft Support.
Cluster aware updating
Suspend node operation was stuck for greater than 90 minutes.
This is a transient issue and could resolve on its own. Wait for a few minutes and retry the operation. If the issue persists, contact Microsoft Support.
Security
In this release, if you enable Dynamic Root of Measurement (DRTM) using the Enable-AzSSecurity cmdlet, you receive the following error: DRTM setting is not supported on current release at C:\ProgramFiles\WindowsPowerShell\Modules\AzureStackOSConfigAgent\AzureStackOSConfigAgent psm1:4307 char:17 + ...throw "DRTM setting is not supported on current release".
DRTM is not supported in this release.
Issues for version 2311.4
This software release maps to software version number 10.2311.4.6. This release only supports updates from 2311 release.
Release notes for this version include the issues fixed in this release, known issues in this release, and release noted issues carried over from previous versions.
Fixed issues
Here are the fixed issues in this release:
Feature
Issue
Workaround/Comments
Arc VM management
When you create a disk or a network interface in this release with underscore in the name, the operation fails.
Make sure to not use underscore in the names for disks or network interfaces.
Arc VM management
If the resource group used to deploy an Arc VM on your Azure Local has an underscore in the name, the guest agent installation fails. As a result, you won't be able to enable guest management.
Make sure that there are no underscores in the resource groups used to deploy Arc VMs.
Known issues in this release
Here are the known issues in this release:
Feature
Issue
Workaround/Comments
Arc VM management
When a new node is added into the cluster, it fails with: Node add failed at ERROR: An older version of the Arc VM cluster extension is installed on your cluster.
Make sure you have the latest firewall requirements including endpoint https://hciarcvmscontainerregistry.azurecr.io on port 443. This endpoint is required for Azure Local Arc VM container registry.
Arc VM management
When you update to 10.2311.4.5, the arcbridge status shows "UpgradeFailed" on the Azure portal.
Make sure you have the latest firewall requirements including endpoint https://hciarcvmscontainerregistry.azurecr.io on port 443. This endpoint is required for Azure Local Arc VM container registry.
Known issues from previous releases
Here are the known issues from previous releases:
Feature
Issue
Workaround/Comments
Arc VM management
Deployment or update of Arc Resource Bridge could fail when the automatically generated temporary SPN secret during this operation, starts with a hyphen.
Retry the deployment/update. The retry should regenerate the SPN secret and the operation will likely succeed.
Arc VM management
Arc Extensions on Arc VMs stay in "Creating" state indefinitely.
Sign in to the VM, open a command prompt, and type the following: Windows: notepad C:\ProgramData\AzureConnectedMachineAgent\Config\agentconfig.json Linux: sudo vi /var/opt/azcmagent/agentconfig.json Next, find the resourcename property. Delete the GUID that is appended to the end of the resource name, so this property matches the name of the VM. Then restart the VM.
Arc VM management
When a new server is added to an Azure Local instance, storage path isn't created automatically for the newly created volume.
You can manually create a storage path for any new volumes. For more information, see Create a storage path.
Arc VM management
Restart of Arc VM operation completes after approximately 20 minutes although the VM itself restarts in about a minute.
There's no known workaround in this release.
Arc VM management
In some instances, the status of the logical network shows as Failed in Azure portal. This occurs when you try to delete the logical network without first deleting any resources such as network interfaces associated with that logical network. You should still be able to create resources on this logical network. The status is misleading in this instance.
If the status of this logical network was Succeeded at the time when this network was provisioned, then you can continue to create resources on this network.
Arc VM management
In this release, when you update a VM with a data disk attached to it using the Azure CLI, the operation fails with the following error message: Couldn't find a virtual hard disk with the name.
There's a sporadic heartbeat reliability issue in this release due to which the registration encounters the error: HCI registration failed. Error: Arc integration failed.
This issue is intermittent. Try rerunning the deployment. For more information, see Rerun the deployment.
Deployment
There's an intermittent issue in this release where the Arc integration validation fails with this error: Validator failed. Can't retrieve the dynamic parameters for the cmdlet. PowerShell Gallery is currently unavailable. Please try again later.
This issue is intermittent. Try rerunning the deployment. For more information, see Rerun the deployment.
Deployment
The AzureEdgeRemoteSupport extension shows as "Failed" in the cluster view and "Succeeded" in the node view. Additionally, the node view displays an incorrect extension name, "AzureEdgeEdgeRemoteSupport".
This issue is cosmetic and doesn't impact extension functionality. You may want to follow these steps to manually mitigate the issue:
1. In the Azure portal, navigate to the resource group for your nodes.
2. Go to each Arc node and uninstall the Remote Support extension.
3. Allow up to 12 hours for the Azure Local Resource Provider to update the extensions.
This procedure enables the reinstallation of the extension, ensuring it displays the correct name, AzureEdgeRemoteSupport, and resolves any failures observed in the cluster view.
Optionally, you can use the cmdlet sync-azurestackhci to force a sync on any of the cluster nodes.
Update
In rare instances, you may encounter this error while updating your Azure Local : Type 'UpdateArbAndExtensions' of Role 'MocArb' raised an exception: Exception Upgrading ARB and Extension in step [UpgradeArbAndExtensions :Get-ArcHciConfig] UpgradeArb: Invalid applianceyaml = [C:\AksHci\hci-appliance.yaml].
If you see this issue, contact Microsoft Support to assist you with the next steps.
Update
In this release, there's a health check issue preventing a single server running Azure Local from being updated via the Azure portal.
There's an infrequent DNS client issue in this release that causes the deployment to fail on a two-node cluster with a DNS resolution error: A WebException occurred while sending a RestRequest. WebException.Status: NameResolutionFailure. As a result of the bug, the DNS record of the second node is deleted soon after it's created resulting in a DNS error.
Restart the server. This operation registers the DNS record, which prevents it from getting deleted.
Azure portal
In some instances, the Azure portal might take a while to update and the view might not be current.
You might need to wait for 30 minutes or more to see the updated view.
Arc VM management
Deleting a network interface on an Arc VM from Azure portal doesn't work in this release.
Providing the OU name in an incorrect syntax isn't detected in the Azure portal. The incorrect syntax is however detected at a later step during cluster validation.
There's no known workaround in this release.
Deployment
In some instances, running the Arc registration script doesn't install the mandatory extensions, Azure Edge device Management or Azure Edge Lifecycle Manager.
The workaround is to run the script again and make sure that all the mandatory extensions are installed before you Deploy via Azure portal.
Deployment
The first deployment step: Before Cloud Deployment when Deploying via Azure portal can take from 45 minutes to an hour to complete.
Deployment
Deployments via Azure Resource Manager time out after 2 hours. Deployments that exceed 2 hours show up as failed in the resource group though the cluster is successfully created.
To monitor the deployment in the Azure portal, go to the Azure Local resource and then go to new Deployments entry.
Azure Site Recovery
Azure Site Recovery can't be installed on an Azure Local instance in this release.
There's no known workaround in this release.
Update
When updating the Azure Local instance via the Azure Update Manager, the update progress and results may not be visible in the Azure portal.
To work around this issue, on each cluster node, add the following registry key (no value needed):
Then on one of the cluster nodes, restart the Cloud Management cluster group.
Stop-ClusterGroup "Cloud Management"
Start-ClusterGroup "Cloud Management"
This won't fully remediate the issue as the progress details may still not be displayed for a duration of the update process. To get the latest update details, you can Retrieve the update progress with PowerShell.
Updates
In rare instances, if a failed update is stuck in an In progress state in Azure Update Manager, the Try again button is disabled.
To resume the update, run the following PowerShell command: Get-SolutionUpdate|Start-SolutionUpdate.
Updates
In some cases, SolutionUpdate commands could fail if run after the Send-DiagnosticData command.
Make sure to close the PowerShell session used for Send-DiagnosticData. Open a new PowerShell session and use it for SolutionUpdate commands.
Updates
In rare instances, when applying an update from 2311.0.24 to 2311.2.4, cluster status reports In Progress instead of expected Failed to update.
Retry the update. If the issue persists, contact Microsoft Support.
Cluster aware updating
Resume node operation failed to resume node.
This is a transient issue and could resolve on its own. Wait for a few minutes and retry the operation. If the issue persists, contact Microsoft Support.
Cluster aware updating
Suspend node operation was stuck for greater than 90 minutes.
This is a transient issue and could resolve on its own. Wait for a few minutes and retry the operation. If the issue persists, contact Microsoft Support.
Security
In this release, if you enable Dynamic Root of Measurement (DRTM) using the Enable-AzSSecurity cmdlet, you receive the following error: DRTM setting is not supported on current release at C:\ProgramFiles\WindowsPowerShell\Modules\AzureStackOSConfigAgent\AzureStackOSConfigAgent psm1:4307 char:17 + ...throw "DRTM setting is not supported on current release".
DRTM is not supported in this release.
Issues for version 2311.3
This software release maps to software version number 10.2311.3.12. This release only supports updates from 2311 release.
Release notes for this version include the issues fixed in this release, known issues in this release, and release noted issues carried over from previous versions.
Fixed issues
Here are the issues fixed in this release:
Feature
Issue
Workaround
Update
When you try to change your AzureStackLCMUserPassword using command: Set-AzureStackLCMUserPassword, you might encounter this error:
Can't find an object with identity: 'object id'*.
If the issue occurs, contact Microsoft Support for next steps.
Known issues in this release
Here are the known issues in this release:
Feature
Issue
Workaround
Security
In this release, if you enable Dynamic Root of Measurement (DRTM) using the Enable-AzSSecurity cmdlet, you receive the following error: DRTM setting is not supported on current release at C:\ProgramFiles\WindowsPowerShell\Modules\AzureStackOSConfigAgent\AzureStackOSConfigAgent psm1:4307 char:17 + ...throw "DRTM setting is not supported on current release".
DRTM is not supported in this release.
Known issues from previous releases
Here are the known issues from previous releases:
Feature
Issue
Workaround
Arc VM management
Deployment or update of Arc Resource Bridge could fail when the automatically generated temporary SPN secret during this operation, starts with a hyphen.
Retry the deployment/update. The retry should regenerate the SPN secret and the operation will likely succeed.
Arc VM management
Arc Extensions on Arc VMs stay in "Creating" state indefinitely.
Sign in to the VM, open a command prompt, and type the following: Windows: notepad C:\ProgramData\AzureConnectedMachineAgent\Config\agentconfig.json Linux: sudo vi /var/opt/azcmagent/agentconfig.json Next, find the resourcename property. Delete the GUID that is appended to the end of the resource name, so this property matches the name of the VM. Then restart the VM.
Arc VM management
When a new server is added to an Azure Local instance, storage path isn't created automatically for the newly created volume.
You can manually create a storage path for any new volumes. For more information, see Create a storage path.
Arc VM management
Restart of Arc VM operation completes after approximately 20 minutes although the VM itself restarts in about a minute.
There's no known workaround in this release.
Arc VM management
In some instances, the status of the logical network shows as Failed in Azure portal. This occurs when you try to delete the logical network without first deleting any resources such as network interfaces associated with that logical network. You should still be able to create resources on this logical network. The status is misleading in this instance.
If the status of this logical network was Succeeded at the time when this network was provisioned, then you can continue to create resources on this network.
Arc VM management
In this release, when you update a VM with a data disk attached to it using the Azure CLI, the operation fails with the following error message: Couldn't find a virtual hard disk with the name.
There's a sporadic heartbeat reliability issue in this release due to which the registration encounters the error: HCI registration failed. Error: Arc integration failed.
This issue is intermittent. Try rerunning the deployment. For more information, see Rerun the deployment.
Deployment
There's an intermittent issue in this release where the Arc integration validation fails with this error: Validator failed. Can't retrieve the dynamic parameters for the cmdlet. PowerShell Gallery is currently unavailable. Please try again later.
This issue is intermittent. Try rerunning the deployment. For more information, see Rerun the deployment.
Deployment
The AzureEdgeRemoteSupport extension shows as "Failed" in the cluster view and "Succeeded" in the node view. Additionally, the node view displays an incorrect extension name, "AzureEdgeEdgeRemoteSupport".
This issue is cosmetic and doesn't impact extension functionality. You may want to follow these steps to manually mitigate the issue:
1. In the Azure portal, navigate to the resource group for your nodes.
2. Go to each Arc node and uninstall the Remote Support extension.
3. Allow up to 12 hours for the Azure Local Resource Provider to update the extensions.
This procedure enables the reinstallation of the extension, ensuring it displays the correct name, AzureEdgeRemoteSupport, and resolves any failures observed in the cluster view.
Optionally, you can use the cmdlet sync-azurestackhci to force a sync on any of the cluster nodes.
Update
In rare instances, you may encounter this error while updating your Azure Local: Type 'UpdateArbAndExtensions' of Role 'MocArb' raised an exception: Exception Upgrading ARB and Extension in step [UpgradeArbAndExtensions :Get-ArcHciConfig] UpgradeArb: Invalid applianceyaml = [C:\AksHci\hci-appliance.yaml].
If you see this issue, contact Microsoft Support to assist you with the next steps.
Update
In this release, there's a health check issue preventing a single server running Azure Local from being updated via the Azure portal.
There's an infrequent DNS client issue in this release that causes the deployment to fail on a two-node cluster with a DNS resolution error: A WebException occurred while sending a RestRequest. WebException.Status: NameResolutionFailure. As a result of the bug, the DNS record of the second node is deleted soon after it's created resulting in a DNS error.
Restart the server. This operation registers the DNS record, which prevents it from getting deleted.
Azure portal
In some instances, the Azure portal might take a while to update and the view might not be current.
You might need to wait for 30 minutes or more to see the updated view.
Arc VM management
Deleting a network interface on an Arc VM from Azure portal doesn't work in this release.
When you create a disk or a network interface in this release with underscore in the name, the operation fails.
Make sure to not use underscore in the names for disks or network interfaces.
Deployment
Providing the OU name in an incorrect syntax isn't detected in the Azure portal. The incorrect syntax is however detected at a later step during cluster validation.
There's no known workaround in this release.
Deployment
In some instances, running the Arc registration script doesn't install the mandatory extensions, Azure Edge device Management or Azure Edge Lifecycle Manager.
The workaround is to run the script again and make sure that all the mandatory extensions are installed before you Deploy via Azure portal.
Deployment
The first deployment step: Before Cloud Deployment when Deploying via Azure portal can take from 45 minutes to an hour to complete.
Deployment
Deployments via Azure Resource Manager time out after 2 hours. Deployments that exceed 2 hours show up as failed in the resource group though the cluster is successfully created.
To monitor the deployment in the Azure portal, go to the Azure Local resource and then go to new Deployments entry.
Azure Site Recovery
Azure Site Recovery can't be installed on an Azure Local instance in this release.
There's no known workaround in this release.
Update
When updating the Azure Local instance via the Azure Update Manager, the update progress and results may not be visible in the Azure portal.
To work around this issue, on each cluster node, add the following registry key (no value needed):
Then on one of the cluster nodes, restart the Cloud Management cluster group.
Stop-ClusterGroup "Cloud Management"
Start-ClusterGroup "Cloud Management"
This won't fully remediate the issue as the progress details may still not be displayed for a duration of the update process. To get the latest update details, you can Retrieve the update progress with PowerShell.
Updates
In rare instances, if a failed update is stuck in an In progress state in Azure Update Manager, the Try again button is disabled.
To resume the update, run the following PowerShell command: Get-SolutionUpdate|Start-SolutionUpdate.
Updates
In some cases, SolutionUpdate commands could fail if run after the Send-DiagnosticData command.
Make sure to close the PowerShell session used for Send-DiagnosticData. Open a new PowerShell session and use it for SolutionUpdate commands.
Updates
In rare instances, when applying an update from 2311.0.24 to 2311.2.4, cluster status reports In Progress instead of expected Failed to update.
Retry the update. If the issue persists, contact Microsoft Support.
Arc VM management
If the resource group used to deploy an Arc VM on your Azure Local has an underscore in the name, the guest agent installation fails. As a result, you won't be able to enable guest management.
Make sure that there are no underscores in the resource groups used to deploy Arc VMs.
Cluster aware updating
Resume node operation failed to resume node.
This is a transient issue and could resolve on its own. Wait for a few minutes and retry the operation. If the issue persists, contact Microsoft Support.
Cluster aware updating
Suspend node operation was stuck for greater than 90 minutes.
This is a transient issue and could resolve on its own. Wait for a few minutes and retry the operation. If the issue persists, contact Microsoft Support.
Issues for version 2311.2
This software release maps to software version number 10.2311.2.7. This release only supports updates from 2311 release.
Release notes for this version include the issues fixed in this release, known issues in this release, and release noted issues carried over from previous versions.
Fixed issues
Here are the issues fixed in this release:
Feature
Issue
Workaround/Comments
Add server and repair server
In this release, add server and repair server scenarios might fail with the following error:
CloudEngine.Actions.InterfaceInvocationFailedException: Type 'AddNewNodeConfiguration' of Role 'BareMetal' raised an exception: The term 'Trace-Execution' is not recognized as the name of a cmdlet, function, script file, or operable program.
Follow these steps to work around this error:
1. Create a copy of the required PowerShell modules on the new node.
When you update 2310 to 2311 software, the service principal doesn't migrate.
If you encounter an issue with the software, use PowerShell to migrate the service principal.
Deployment
If you select Review + Create and you haven't filled out all the tabs, the deployment begins and then eventually fails.
There's no known workaround in this release.
Deployment
This issue is seen if an incorrect subscription or resource group was used during registration. When you register the server a second time with Arc, the Azure Edge Lifecycle Manager extension fails during the registration, but the extension state is reported as Ready.
Before you run the registration the second time:
Make sure to delete the following folders from your servers: C:\ecestore, C:\CloudDeployment, and C:\nugetstore. Delete the registry key using the PowerShell cmdlet: Remove-Item HKLM:\Software\Microsoft\LCMAzureStackStampInformation
Deployment
A new storage account is created for each run of the deployment. Existing storage accounts aren't supported in this release.
Deployment
A new key vault is created for each run of the deployment. Existing key vaults aren't supported in this release.
Deployment
On server hardware, a USB network adapter is created to access the Baseboard Management Controller (BMC). This adapter can cause the cluster validation to fail during the deployment.
Make sure to disable the BMC network adapter before you begin cloud deployment.
Deployment
The network direct intent overrides defined on the template aren't working in this release.
Use the Azure Resource Manager template to override this parameter and disable RDMA for the intents.
Known issues in this release
Here are the known issues in this release:
Feature
Issue
Workaround/Comments
Update
In this release, if you run the Test-CauRun cmdlet before actually applying the 2311.2 update, you see an error message regarding a missing firewall rule to remotely shut down the Azure Local system.
No action is required on your part as the missing rule is automatically created when 2311.2 updates are applied.
When applying future updates, make sure to run the Get-SolutionUpdateEnvironment cmdlet instead of Test-CauRun.
Updates
In rare instances, if a failed update is stuck in an In progress state in Azure Update Manager, the Try again button is disabled.
To resume the update, run the following PowerShell command: Get-SolutionUpdate|Start-SolutionUpdate.
Updates
In some cases, SolutionUpdate commands could fail if run after the Send-DiagnosticData command.
Make sure to close the PowerShell session used for Send-DiagnosticData. Open a new PowerShell session and use it for SolutionUpdate commands.
Updates
In very rare instances, when applying an update from 2311.0.24 to 2311.2.4, cluster status reports In Progress instead of expected Failed to update.
Retry the update. If the issue persists, contact Microsoft Support.
Arc VM management
If the resource group used to deploy an Arc VM on your Azure Local has an underscore in the name, the guest agent installation will fail. As a result, you won't be able to enable guest management.
Make sure that there are no underscores in the resource groups used to deploy Arc VMs.
Cluster aware updating
Resume node operation failed to resume node.
This is a transient issue and could resolve on its own. Wait for a few minutes and retry the operation. If the issue persists, contact Microsoft Support.
Cluster aware updating
Suspend node operation was stuck for greater than 90 minutes.
This is a transient issue and could resolve on its own. Wait for a few minutes and retry the operation. If the issue persists, contact Microsoft Support.
Known issues from previous releases
Here are the known issues from previous releases:
Feature
Issue
Workaround
Arc VM management
Deployment or update of Arc Resource Bridge could fail when the automatically generated temporary SPN secret during this operation, starts with a hyphen.
Retry the deployment/update. The retry should regenerate the SPN secret and the operation will likely succeed.
Arc VM management
Arc Extensions on Arc VMs stay in "Creating" state indefinitely.
Log in to the VM, open a command prompt, and type the following: Windows: notepad C:\ProgramData\AzureConnectedMachineAgent\Config\agentconfig.json Linux: sudo vi /var/opt/azcmagent/agentconfig.json Next, find the resourcename property. Delete the GUID that is appended to the end of the resource name, so this property matches the name of the VM. Then restart the VM.
Arc VM management
When a new server is added to an Azure Local instance, storage path isn't created automatically for the newly created volume.
You can manually create a storage path for any new volumes. For more information, see Create a storage path.
Arc VM management
Restart of Arc VM operation completes after approximately 20 minutes although the VM itself restarts in about a minute.
There's no known workaround in this release.
Arc VM management
In some instances, the status of the logical network shows as Failed in Azure portal. This occurs when you try to delete the logical network without first deleting any resources such as network interfaces associated with that logical network. You should still be able to create resources on this logical network. The status is misleading in this instance.
If the status of this logical network was Succeeded at the time when this network was provisioned, then you can continue to create resources on this network.
Arc VM management
In this release, when you update a VM with a data disk attached to it using the Azure CLI, the operation fails with the following error message: Couldn't find a virtual hard disk with the name.
There's a sporadic heartbeat reliability issue in this release due to which the registration encounters the error: HCI registration failed. Error: Arc integration failed.
This issue is intermittent. Try rerunning the deployment. For more information, see Rerun the deployment.
Deployment
There's an intermittent issue in this release where the Arc integration validation fails with this error: Validator failed. Cannot retrieve the dynamic parameters for the cmdlet. PowerShell Gallery is currently unavailable. Please try again later.
This issue is intermittent. Try rerunning the deployment. For more information, see Rerun the deployment.
Deployment
The AzureEdgeRemoteSupport extension shows as "Failed" in the cluster view and "Succeeded" in the node view. Additionally, the node view displays an incorrect extension name, "AzureEdgeEdgeRemoteSupport".
This issue is cosmetic and doesn't impact extension functionality. You may want to follow these steps to manually mitigate the issue:
1. In the Azure portal, navigate to the resource group for your nodes.
2. Go to each Arc node and uninstall the Remote Support extension.
3. Allow up to 12 hours for the Azure Local Resource Provider to update the extensions.
This procedure enables the reinstallation of the extension, ensuring it displays the correct name, AzureEdgeRemoteSupport, and resolves any failures observed in the cluster view.
Optionally, you can use the cmdlet sync-azurestackhci to force a sync on any of the cluster nodes.
Update
In rare instances, you may encounter this error while updating your Azure Local: Type 'UpdateArbAndExtensions' of Role 'MocArb' raised an exception: Exception Upgrading ARB and Extension in step [UpgradeArbAndExtensions :Get-ArcHciConfig] UpgradeArb: Invalid applianceyaml = [C:\AksHci\hci-appliance.yaml].
If you see this issue, contact Microsoft Support to assist you with the next steps.
Update
When you try to change your AzureStackLCMUserPassword using command: Set-AzureStackLCMUserPassword, you might encounter this error:
Cannot find an object with identity: 'object id'.
If the issue occurs, contact Microsoft Support for next steps.
Update
In this release, there's a health check issue preventing a single server running Azure Local from being updated via the Azure portal.
There's an infrequent DNS client issue in this release that causes the deployment to fail on a two-node cluster with a DNS resolution error: A WebException occurred while sending a RestRequest. WebException.Status: NameResolutionFailure. As a result of the bug, the DNS record of the second node is deleted soon after it's created resulting in a DNS error.
Restart the server. This operation registers the DNS record, which prevents it from getting deleted.
Azure portal
In some instances, the Azure portal might take a while to update and the view might not be current.
You might need to wait for 30 minutes or more to see the updated view.
Arc VM management
Deleting a network interface on an Arc VM from Azure portal doesn't work in this release.
When you create a disk or a network interface in this release with underscore in the name, the operation fails.
Make sure to not use underscore in the names for disks or network interfaces.
Deployment
Providing the OU name in an incorrect syntax isn't detected in the Azure portal. The incorrect syntax is however detected at a later step during cluster validation.
There's no known workaround in this release.
Deployment
In some instances, running the Arc registration script doesn't install the mandatory extensions, Azure Edge device Management or Azure Edge Lifecycle Manager.
Run the script again and make sure that all the mandatory extensions are installed before you Deploy via Azure portal.
Deployment
The first deployment step: Before Cloud Deployment when Deploying via Azure portal can take from 45 minutes to an hour to complete.
Deployment
Deployments via Azure Resource Manager time out after 2 hours. Deployments that exceed 2 hours show up as failed in the resource group though the cluster is successfully created.
To monitor the deployment in the Azure portal, go to the Azure Local resource and then go to new Deployments entry.
Azure Site Recovery
Azure Site Recovery can't be installed on an Azure Local instance in this release.
There's no known workaround in this release.
Update
When updating the Azure Local instance via the Azure Update Manager, the update progress and results may not be visible in the Azure portal.
To work around this issue, on each cluster node, add the following registry key (no value needed):
Then on one of the cluster nodes, restart the Cloud Management cluster group.
Stop-ClusterGroup "Cloud Management"
Start-ClusterGroup "Cloud Management"
This won't fully remediate the issue as the progress details may still not be displayed for a duration of the update process. To get the latest update details, you can Retrieve the update progress with PowerShell.
Issues for version 2311
This software release maps to software version number 10.2311.0.26. This release supports new deployments and updates from 2310.
Release notes for this version include the issues fixed in this release, known issues in this release, and release noted issues carried over from previous versions.
Fixed issues
Here are the issues fixed in this release:
Feature
Issue
Networking
Use of proxy isn't supported in this release.
Security
When using the Get-AzsSyslogForwarder cmdlet with -PerNode parameter, an exception is thrown. You aren't able to retrieve the SyslogForwarder configuration information of multiple nodes.
Deployment
During the deployment, Microsoft Open Cloud (MOC) Arc Resource Bridge installation fails with this error: Unable to find a resource that satisfies the requirement Size [0] Location [MocLocation].: OutOfCapacity"\n".
Deployment
Entering an incorrect DNS updates the DNS configuration in hosts during the validation and the hosts can lose internet connectivity.
Deployment
Password for deployment user (also referred to as AzureStackLCMUserCredential during Active Directory prep) and local administrator can't include a :(colon).
Arc VM management
Detaching a disk via the Azure CLI results in an error in this release.
Arc VM management
A resource group with multiple clusters only shows storage paths of one cluster.
Arc VM management
When you create the Azure Marketplace image on Azure Local , sometimes the download provisioning state doesn't match the download percentage on Azure Local instance. The provisioning state is returned as succeeded while the download percentage is reported as less than 100.
Arc VM management
In this release, depending on your environment, the VM deployments on Azure Local system can take 30 to 45 minutes.
Arc VM management
While creating Arc VMs via the Azure CLI on Azure Local , if you provide the friendly name of marketplace image, an incorrect Azure Resource Manager ID is built and the VM creation errors out.
Known issues in this release
Here are the known issues in this release:
Feature
Issue
Workaround/Comments
Arc VM management
Deployment or update of Arc Resource Bridge could fail when the automatically generated SPN secret during this operation, starts with a hyphen.
Retry the deployment/update. The retry should regenerate the SPN secret and the operation will likely succeed.
Arc VM management
Arc Extensions on Arc VMs stay in "Creating" state indefinitely.
Sign into the VM, open a command prompt, and type the following: Windows: notepad C:\ProgramData\AzureConnectedMachineAgent\Config\agentconfig.json Linux: sudo vi /var/opt/azcmagent/agentconfig.json Next, find the resourcename property. Delete the GUID that is appended to the end of the resource name, so this property matches the name of the VM. Then restart the VM.
Arc VM management
When a new server is added to an Azure Local instance, storage path isn't created automatically for the newly created volume.
You can manually create a storage path for any new volumes. For more information, see Create a storage path.
Arc VM management
Restart of Arc VM operation completes after approximately 20 minutes although the VM itself restarts in about a minute.
There's no known workaround in this release.
Arc VM management
In some instances, the status of the logical network shows as Failed in Azure portal. This occurs when you try to delete the logical network without first deleting any resources such as network interfaces associated with that logical network. You should still be able to create resources on this logical network. The status is misleading in this instance.
If the status of this logical network was Succeeded at the time when this network was provisioned, then you can continue to create resources on this network.
Arc VM management
In this release, when you update a VM with a data disk attached to it using the Azure CLI, the operation fails with the following error message: Couldn't find a virtual hard disk with the name.
Before you update 2310 to 2311 software, make sure to run the following cmdlets on one of your Azure Local nodes: Import-module C:\CloudDeployment\CloudDeployment.psd1 Import-module C:\CloudDeployment\ECEngine\EnterpriseCloudEngine.psd1
There's a sporadic heartbeat reliability issue in this release due to which the registration encounters the error: HCI registration failed. Error: Arc integration failed.
This issue is intermittent. Try rerunning the deployment. For more information, see Rerun the deployment.
Deployment
There's an intermittent issue in this release where the Arc integration validation fails with this error: Validator failed. Can't retrieve the dynamic parameters for the cmdlet. PowerShell Gallery is currently unavailable. Please try again later.
This issue is intermittent. Try rerunning the deployment. For more information, see Rerun the deployment.
Deployment
The AzureEdgeRemoteSupport extension shows as "Failed" in the cluster view and "Succeeded" in the node view. Additionally, the node view displays an incorrect extension name, "AzureEdgeEdgeRemoteSupport".
This issue is cosmetic and doesn't impact extension functionality. You may want to follow these steps to manually mitigate the issue:
1. In the Azure portal, navigate to the resource group for your nodes.
2. Go to each Arc node and uninstall the Remote Support extension.
3. Allow up to 12 hours for the Azure Local Resource Provider to update the extensions.
This procedure enables the reinstallation of the extension, ensuring it displays the correct name, AzureEdgeRemoteSupport, and resolves any failures observed in the cluster view.
Optionally, you can use the cmdlet sync-azurestackhci to force a sync on any of the cluster nodes.
Update
In rare instances, you may encounter this error while updating your Azure Local : Type 'UpdateArbAndExtensions' of Role 'MocArb' raised an exception: Exception Upgrading ARB and Extension in step [UpgradeArbAndExtensions :Get-ArcHciConfig] UpgradeArb: Invalid applianceyaml = [C:\AksHci\hci-appliance.yaml].
If you see this issue, contact Microsoft Support to assist you with the next steps.
Update
When you try to change your AzureStackLCMUserPassword using command: Set-AzureStackLCMUserPassword, you might encounter this error:
Cannot find an object with identity: 'object id'.
There's no known workaround in this release.
Update
In this release, there's a health check issue preventing a single server running Azure Local from being updated via the Azure portal.
When you update from the 2311 build to Azure Local, the update health checks stop reporting in the Azure portal after the update reaches the Install step.
Microsoft is actively working to resolve this issue, and there's no action required on your part. Although the health checks aren't visible in the portal, they're still running and completing as expected.
Add server and repair server
In this release, add server and repair server scenarios might fail with the following error:
CloudEngine.Actions.InterfaceInvocationFailedException: Type 'AddNewNodeConfiguration' of Role 'BareMetal' raised an exception: The term 'Trace-Execution' isn't recognized as the name of a cmdlet, function, script file, or operable program.
Follow these steps to work around this error:
1. Create a copy of the required PowerShell modules on the new node.
There's an infrequent DNS client issue in this release that causes the deployment to fail on a two-node cluster with a DNS resolution error: A WebException occurred while sending a RestRequest. WebException.Status: NameResolutionFailure. As a result of the bug, the DNS record of the second node is deleted soon after it's created resulting in a DNS error.
Restart the server. This operation registers the DNS record, which prevents it from getting deleted.
Azure portal
In some instances, the Azure portal might take a while to update and the view might not be current.
You might need to wait for 30 minutes or more to see the updated view.
Arc VM management
Deleting a network interface on an Arc VM from Azure portal doesn't work in this release.
When you create a disk or a network interface in this release with underscore in the name, the operation fails.
Make sure to not use underscore in the names for disks or network interfaces.
Deployment
Providing the OU name in an incorrect syntax isn't detected in the Azure portal. The incorrect syntax is however detected at a later step during cluster validation.
There's no known workaround in this release.
Deployment
On server hardware, a USB network adapter is created to access the Baseboard Management Controller (BMC). This adapter can cause the cluster validation to fail during the deployment.
Make sure to disable the BMC network adapter before you begin cloud deployment.
Deployment
A new storage account is created for each run of the deployment. Existing storage accounts aren't supported in this release.
Deployment
A new key vault is created for each run of the deployment. Existing key vaults aren't supported in this release.
Deployment
In some instances, running the Arc registration script doesn't install the mandatory extensions, Azure Edge device Management or Azure Edge Lifecycle Manager.
The workaround is to run the script again and make sure that all the mandatory extensions are installed before you Deploy via Azure portal.
Deployment
The first deployment step: Before Cloud Deployment when Deploying via Azure portal can take from 45 minutes to an hour to complete.
Deployment
The network direct intent overrides defined on the template aren't working in this release.
Use the Azure Resource Manager template to override this parameter and disable RDMA for the intents.
Deployment
Deployments via Azure Resource Manager time out after 2 hours. Deployments that exceed 2 hours show up as failed in the resource group though the cluster is successfully created.
To monitor the deployment in the Azure portal, go to the Azure Local resource and then go to new Deployments entry.
Deployment
If you select Review + Create and you haven't filled out all the tabs, the deployment begins and then eventually fails.
There's no known workaround in this release.
Deployment
This issue is seen if an incorrect subscription or resource group was used during registration. When you register the server a second time with Arc, the Azure Edge Lifecycle Manager extension fails during the registration, but the extension state is reported as Ready.
Before you run the registration the second time:
Make sure to delete the following folders from your server(s): C:\ecestore, C:\CloudDeployment, and C:\nugetstore. Delete the registry key using the PowerShell cmdlet: Remove-Item HKLM:\Software\Microsoft\LCMAzureStackStampInformation
Azure Site Recovery
Azure Site Recovery can't be installed on an Azure Local instance in this release.
There's no known workaround in this release.
Update
When updating the Azure Local instance via the Azure Update Manager, the update progress and results might not be visible in the Azure portal.
To work around this issue, on each cluster node, add the following registry key (no value needed):
Then on one of the cluster nodes, restart the Cloud Management cluster group.
Stop-ClusterGroup "Cloud Management"
Start-ClusterGroup "Cloud Management"
This won't fully remediate the issue as the progress details might still not be displayed for a duration of the update process. To get the latest update details, you can Retrieve the update progress with PowerShell.
Azure HPC es una capacidad en la nube creada a propósito para la carga de trabajo de IA y de HPC, mediante procesadores de vanguardia e interconexión InfiniBand de clase HPC, con el fin de ofrecer el mejor rendimiento, escalabilidad y valor de la aplicación. Azure HPC permite a los usuarios desbloquear la innovación, la productividad y la agilidad empresarial, mediante una gama de tecnologías de inteligencia artificial y de HPC de alta disponibilidad que se pueden asignar dinámicamente a medida que cambian
Como administrador híbrido de Windows Server, integra los entornos de Windows Server con servicios de Azure y administra Windows Server en redes locales.