How to Resolve Azure Local Deployment Stuck in Cleanup After OS Update

Question

How to Resolve Azure Local Deployment Stuck in Cleanup After OS Update

Ahmad, Andri 5

Azure Local Deployment Stuck in Clean Up after the OS update. Looking at the logs it seems it stuck when installing versioned components:

0.7 InProgress Clean up after update 2025-03-26T10:40:38.1240126Z

0.7.1 Success Parallel per-node operation top step 2025-03-26T10:40:38.1552645Z 2025-03-26T10:40:56.889865Z 00:00:18.7346005

0.7.1.1 Success Apply KIR 2025-03-26T10:40:38.1708862Z 2025-03-26T10:40:56.889865Z 00:00:18.7189788

0.7.1.1 Success Apply KIR 2025-03-26T10:40:38.1708862Z 2025-03-26T10:40:56.7804839Z 00:00:18.6095977

0.7.2 Success PrepareRoleNugets 2025-03-26T10:40:56.889865Z 2025-03-26T10:42:09.7237031Z 00:01:12.8338381

0.7.2.1 Skipped Download CustomCloud Packages 2025-03-26T10:40:56.905486Z 2025-03-26T10:41:08.4061288Z 00:00:11.5006428

0.7.2.2 Success Parallel per-node operation top step 2025-03-26T10:41:08.4061288Z 2025-03-26T10:41:35.4395648Z 00:00:27.0334360

0.7.2.2.1 Success Expand Live Update Content 2025-03-26T10:41:08.4686286Z 2025-03-26T10:41:35.4395648Z 00:00:26.9709362

0.7.2.2.1 Success Expand Live Update Content 2025-03-26T10:41:08.4373758Z 2025-03-26T10:41:34.6583181Z 00:00:26.2209423

0.7.2.3 Success Parallel per-node operation top step 2025-03-26T10:41:35.4395648Z 2025-03-26T10:41:59.1591957Z 00:00:23.7196309

0.7.2.3.1 Success Install Role Nugets 2025-03-26T10:41:35.4552106Z 2025-03-26T10:41:59.1591957Z 00:00:23.7039851

0.7.2.3.1 Success Install Role Nugets 2025-03-26T10:41:35.4708264Z 2025-03-26T10:41:57.3623139Z 00:00:21.8914875

0.7.2.4 Success Parallel per-node operation top step 2025-03-26T10:41:59.1591957Z 2025-03-26T10:42:09.7237031Z 00:00:10.5645074

0.7.2.4.1 Skipped Install Role Nugets for Azure.local cloud 2025-03-26T10:41:59.2060696Z 2025-03-26T10:42:09.7237031Z 00:00:10.5176335

0.7.2.4.1 Skipped Install Role Nugets for Azure.local cloud 2025-03-26T10:41:59.1748216Z 2025-03-26T10:42:09.6924512Z 00:00:10.5176296

0.7.3 Success InstallBootstrapPowerShellModules 2025-03-26T10:42:09.7237031Z 2025-03-26T10:43:00.7763585Z 00:00:51.0526554

0.7.3.1 Success Parallel per-node operation top step 2025-03-26T10:42:09.7393203Z 2025-03-26T10:43:00.7763585Z 00:00:51.0370382

0.7.3.1.1 Success InstallBootstrapPowerShellModules 2025-03-26T10:42:09.7705756Z 2025-03-26T10:42:57.7739915Z 00:00:48.0034159

0.7.3.1.1 Success InstallBootstrapPowerShellModules 2025-03-26T10:42:09.7549495Z 2025-03-26T10:43:00.7763585Z 00:00:51.0214090

0.7.4 Success Setup Observability Volume 2025-03-26T10:43:00.7763585Z 2025-03-26T10:43:00.791986Z 00:00:00.0156275

0.7.5 Success Installs Bootstrap Observability 2025-03-26T10:43:00.791986Z 2025-03-26T10:43:00.8076093Z 00:00:00.0156233

0.7.6 Success InstallRemainingPowerShellModules 2025-03-26T10:43:00.8076093Z 2025-03-26T10:43:41.777391Z 00:00:40.9697817

0.7.6.1 Success Parallel per-node operation top step 2025-03-26T10:43:00.8232321Z 2025-03-26T10:43:41.777391Z 00:00:40.9541589

0.7.6.1.1 Success InstallRemainingPowerShellModules 2025-03-26T10:43:00.8701071Z 2025-03-26T10:43:41.6211332Z 00:00:40.7510261

0.7.6.1.1 Success InstallRemainingPowerShellModules 2025-03-26T10:43:00.8388564Z 2025-03-26T10:43:41.777391Z 00:00:40.9385346

0.7.7 InProgress Parallel per-node operation top step 2025-03-26T10:43:41.777391Z

0.7.7.1 InProgress Install versioned components 2025-03-26T10:43:41.8242641Z

0.7.7.1 InProgress Install versioned components 2025-03-26T10:43:41.7930143Z

0.7.8 EnableFirewallPortsOnAllHosts

0.7.9 ResizeSystemDriveOnAllHosts

0.7.10 AddDSCCertificateOnHost

0.7.11 UpdateStampInformation

0.8 Evaluate proxy configuration

0.9 Validate network settings for servers

0.10 Configure settings on servers

0.11 Adjust the number of infrastructure VMs

0.12 Prepare servers for security policies

0.13 Apply security settings on servers

0.14 Join servers to a domain

0.15 Install software dependencies

0.16 Set Azure cloud for AzureLocal

0.17 Configure DNS if identity provider is Local with Key Vault

Environment Information:

Running on Nested Virtualization on top of Hyper-V using MSLAB
2 x Machines (virtual) running Azure Stack HCI OS version 10.2411.3.2 (25398.1425)
Azure Local instance is still being deployed from Portal now on the step clean up after update

Troubleshooting steps attempt

wait until something happened, it has been around 10 hours until time-out error
resume the deployment
next step would probably restart the nodes before resuming the deployment

what would be the cause and how to solve this? is there any cache or temporary file that I should removed/cleaned before attempting to resume?

Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
Ahmad, Andri 5 Reputation points

2025-03-28T03:10:03.6166667+00:00

I have waited until time-out error (about 10 hours), tried to restart the nodes manually, and resume the deployment again. let's see what will happened.

Ahmad, Andri 5

still stuck, now my plan is:

Clean temporary file as mentioned here: https://learn.microsoft.com/en-us/azure/azure-local/known-issues?view=azloc-24112
- Delete C:\EceStore.
- Delete C:\CloudDeployment.
- Delete C:\nugetstore.
- Remove-Item HKLM:\Software\Microsoft\LCMAzureStackStampInformation.
Reinstall the Extensions:

Run this from management machine:

$Servers="machine-1","machine-2"
Foreach ($Server in $Servers) {
	$extensionName="AzureEdgeLifecycleManager" 
    Get-AzConnectedMachineExtension -Name $extensionName -SubscriptionID $SubscriptionID -ResourceGroupName $ResourceGroupName -MachineName $Server
    New-AzConnectedMachineExtension -Name $extensionName -ResourceGroupName $ResourceGroupName -MachineName $Server -Location $Location -Publisher "Microsoft.AzureStack.Orchestration" -ExtensionType "LcmController" -NoWait
}

Restart the machines

Restart-Computer -Force -ComputerName $Servers -Credential $Credentials -WsmanAuthentication Negotiate -Wait -For PowerShell
    Start-Sleep 20 
    Foreach ($Server in $Servers) {
        do {$Test= Test-NetConnection -ComputerName $Server -CommonTCPPort WINRM} while ($test.TcpTestSucceeded -eq $False)
}

Rerun the Deployment from Portal

Ahmad, Andri 5 Reputation points

2025-04-07T02:31:08.4466667+00:00
I'm still stuck; my plan is to:

uninstall the extensions

clean up file:

Delete C:\EceStore.

Delete C:\CloudDeployment.

Delete C:\nugetstore.

Remove-Item HKLM:\Software\Microsoft\LCMAzureStackStampInformation.

restart the machines

install the extensions

resume the deployment
Ahmad, Andri 5 Reputation points

2025-04-07T03:41:44.37+00:00

It seems I found new error when resuming the deployment:

ExceptionType 'InitializeEceLite' of Role 'DeploymentService' raised an exception: Failed to initialize Ece Lite with Exception: Exception message: Exception calling "Wait" with "0" argument(s): "One or more errors occurred." at at System.Management.Automation.ExceptionHandlingOps.CheckActionPreference(FunctionContext funcContext, Exception exception) at System.Management.Automation.Interpreter.ActionCallInstruction`2.Run(InterpretedFrame frame) at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame) at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame) with base exception message: Item 'CustomerConfiguration.xml' not found in store 'Default'. The file path is 'C:\ECEStore\efb61d70-47ed-8f44-5d63-bed6adc0fb0f\0b8ae09e-3e34-bd06-78a3-28c0d0d425ca'. at at CloudEngine.Store.FileStore.LoadT at Microsoft.AzureStack.Solution.Deploy.ECEControllerService.ECELiteConnection.GetCustomerConfiguration(String storeName, String storeRoot) at Microsoft.AzureStack.Solution.Deploy.ECEControllerService.ECELiteConnection.<SaveSecretsToConfig>d__14.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.AzureStack.Solution.Deploy.ECEControllerService.ECEActionPlanService.<SaveSecretsToConfig>d__5.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.AzureStack.Solution.Deploy.LCMController.ArcCommunication.ECELiteConfigurationService.<InitECELiteConfig>d__5.MoveNext() at InitializeEceLite, C:\NugetStore\Microsoft.AzureStack.Role.Deployment.Service.10.2503.0.907\content\Classes\DeploymentService\DeploymentService.psm1: line 1624 at <ScriptBlock>, C:\NugetStore\Microsoft.AzureStack.Solution.LCMControllerWinService.10.2503.0.907\content\LCMControllerWinService\InvokeInterfaceInternal.psm1: line 139 at Invoke-EceInterfaceInternal, C:\NugetStore\Microsoft.AzureStack.Solution.LCMControllerWinService.10.2503.0.907\content\LCMControllerWinService\InvokeInterfaceInternal.psm1: line 134 at <ScriptBlock>, <No file>: line 36
Anusree Nashetty 6,475 Reputation points Microsoft External Staff Moderator

2025-05-16T19:21:32.0833333+00:00

Hi Ahmad, Andri,

Could you let us know if the issue is resolved for you. If you are still looking for any help, let me know. Thank you.
Anusree Nashetty 6,475 Reputation points Microsoft External Staff Moderator

2025-05-19T17:09:28.3066667+00:00

Hi Ahmad, Andri,

Could you let us know if the issue is resolved for you. If you are still looking for any help, let me know. Thank you.
Jamie O'Connell 0 Reputation points

2025-09-09T07:29:45.4933333+00:00

Hi @Andri Ahmad

Running in to the same issue with Failed to initialize Ece Lite. What was your fix?
Ahmad, Andri 5 Reputation points

2025-09-09T07:58:01.8266667+00:00

Cleaning up file is a bad idea. I had to redeploy the cluster
Jamie O'Connell 0 Reputation points

2025-09-10T06:25:34.3066667+00:00

For anyone else that stumbles across this thread. For us it was a networking issue from the cluster node to the keyvault. Specifically with TLS inspection.