ASDK installation failed at step 40.51 Invoke-EceAction InvalidOperation

Question

We use asdk-installation.ps1 to deploy the ASDK on a physical machine, the installation stop due to Invoke-EceAction InvalidOperation error (detail below)

VERBOSE: Step: Status of step '40.51 - Configure NC service fabric manifest cluster' is 'Error'. -1/3/2022 4:43:35 PM
VERBOSE: Checking if any of the in progress steps are complete. The following steps are currently in progress: '40.51'. - 1/3/2022 4:43:35 PM
VERBOSE: Action: Action plan reached terminal state due to one of the steps. Finish running all remaining steps that may still be currently in progress before exiting. - 1/3/2022 4:43:35 PM
VERBOSE: Draining all steps that are still in progress. The following steps are still in progress or just completed: '40.51'. - 1/3/2022 4:43:35 PM
VERBOSE: Action: Action plan 'Deployment-Phase1-CreateVMs' failed. - 1/3/2022 4:43:35 PM
Invoke-EceAction : Action: Invocation of step 40.51 failed. Stopping invocation of action plan.
At line:5 char:2

• Invoke-EceAction -RolePath Cloud -ActionType Deployment -Rerun -Retr ...
• ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
• CategoryInfo : InvalidOperation: (:) [Invoke-EceAction], ActionExecutionException
• FullyQualifiedErrorId : Unspecified error,CloudEngine.Cmdlets.InvokeCmdlet

VERBOSE: Action: Status of 'Deployment-Phase1-CreateVMs' is 'Error'. - 1/3/2022 4:43:35 PM

We met the same error after trying to re-run the installation,

cd C:\CloudDeployment\Setup
.\InstallAzureStackPOC.ps1 -Rerun

Any suggestions or guidance to fix the issue will be appreciated, thanks.

Answer 1

Based on below finding, it seems something wrong on AzS-NC01, we reboot this server, the installer run and move to next step.

Step: Running step 40.51 - Configure NC service fabric manifest cluster
Task: Invocation of interface 'ConfigureServiceFabricCluster' of role 'Cloud\Fabric\NC' failed

Type 'ConfigureServiceFabricCluster' of Role 'NC' raised an exception:

Exception calling "Invoke" with "1" argument(s): "The object is closed..Exception.Message

...
2022-01-05 08:22:54 Verbose [NC:ConfigureServiceFabricCluster] Using azurestack\FA-w9b4t328 to communicate with NC
2022-01-05 08:22:54 Verbose [NC:ConfigureServiceFabricCluster] NC nodes found: 'AzS-NC01'
2022-01-05 08:22:54 Verbose [NC:ConfigureServiceFabricCluster] NC node found: 'AzS-NC01'
2022-01-05 08:22:54 Verbose [NC:ConfigureServiceFabricCluster] Purge kerberos tickets of network service account
2022-01-05 08:22:54 Verbose [NC:ConfigureServiceFabricCluster] Purging kerberos tickets of network service account at AzS-NC01
2022-01-05 08:22:54 Verbose [NC:ConfigureServiceFabricCluster] Using Invoke-Command for purging Kerberos tickets of network service account
2022-01-05 08:22:57 Verbose [NC:ConfigureServiceFabricCluster] Successfully purging Kerberos tickets of network service account
2022-01-05 08:22:57 Verbose [NC:ConfigureServiceFabricCluster] Configure service fabric cluster manifest at NC.
2022-01-05 08:22:58 Verbose [NC:ConfigureServiceFabricCluster] Using Invoke-Command for service fabric cluster manifest
2022-01-05 08:26:21 Verbose [NC:ConfigureServiceFabricCluster] Setting Network Controller Diagnostics to use local log location on AzS-NC01
2022-01-05 08:32:15 Verbose Action: Cancel notification received. Cancelling pending step.

Other support info for future reference

Finding from Hyper-V manager
AzS-CA01 not found

Finding from DNS Manager:
Forward Lookup Zone: (no items)
Reverse Lookup Zone: (no items)
Trust Points: (no items)
Conditional Forwarders: (no iems)

Finding from System event log:

• Error on Schannel
  A fatal error occurred while creating a TLS client credential. The internal error state is 10013.
• Error on Failover Clustering-Client
  LogExtendedErrorInformation (974): Extended RPC error information:
  ProcessID is 2528
  System time is: 32748/472/64851 502:0:2672:30150
  Generating component is 2
  Status is 1753
  Detection location is 501
  Flags is 0
  NumberOfParameters is 4
  Unicode string: ncacn_ip_tcp
  Unicode string: AzS-Sql01.azurestack.local
  Long val: -1182943054
  Long val: 382312662
• Error on Failover Clustering (Network Name Resource)

Cluster network name resource failed registration of one or more associated DNS names(s) because the access to update the secure DNS Zone was denied.

Cluster Network name: 'Cluster Name'
DNS Zone: 'azurestack.local'

Ensure that cluster name object (CNO) is granted permissions to the Secure DNS Zone.

• Error on GroupPolicy
  The processing of Group Policy failed. Windows attempted to retrieve new Group Policy settings for this user or computer. Look in the details tab for error code and description. Windows will automatically retry this operation at the next refresh cycle. Computers joined to the domain must have proper name resolution and network connectivity to a domain controller for discovery of new Group Policy objects and settings. An event will be logged when Group Policy is successful.