replicating hyper-v VM to azure suddenly started failing

Question

Hyper-V running on windows 2019 (fully patched)
Trying to replicate a fully patched Windows server 2016 to Azure via ASR. This was working up until about a week ago, and then stopped.
After much irritation I decided to just disable the replication and delete the recovery services vault and start over.
Updated to latest MARS and provider agent(s), updated all guest and host OS to latest, and errors to initiate/complete the initial replication fail.

Logs: (in order of occurrence)

[X:\bt\1014568\repo\src\SRS\Service\ProviderFramework\ReplicationProvider\DraPlugins\HyperVReplicaCommon\Helpers\WmiHelper.cs:348 ExecuteMethod] Job 'Enabling Replication' with instance ID 'XXXXXXXX-36CF-4937-XXXX-XXXXXXXXXXXX' failed. Job State: 'Exception'. Error Code: '32768'. ErrorSummaryDescription: 'Hyper-V failed to enable replication.'. ErrorDescription: 'Hyper-V failed to enable replication for virtual machine 'VMGUESTNAME': Unspecified error (0x80004005). (Virtual machine ID XXXXXXX-14A4-4760-XXXX-XXXXXXXXXXXX)'.

[X:\bt\1014568\repo\src\SRS\Service\ProviderFramework\ReplicationProvider\DraPlugins\HyperVReplicaCommon\Wmi\ReplicationService.cs:571 GetMsvmErrors] Detailed errors:
Error details - <INSTANCE CLASSNAME="Msvm_Error"><PROPERTY NAME="CIMStatusCode" TYPE="uint32"><VALUE>1</VALUE></PROPERTY><PROPERTY NAME="CIMStatusCodeDescription" TYPE="string"></PROPERTY><PROPERTY NAME="ErrorSource" TYPE="string"></PROPERTY><PROPERTY NAME="ErrorSourceFormat" TYPE="uint16"><VALUE>0</VALUE></PROPERTY><PROPERTY NAME="ErrorType" TYPE="uint16"><VALUE>4</VALUE></PROPERTY><PROPERTY NAME="Message" TYPE="string"><VALUE>Hyper-V failed to enable replication for virtual machine 'VMGUESTNAME': Unspecified error (0x80004005). (Virtual machine ID XXXXXXX-14A4-4760-XXXX-XXXXXXXXXXXX)</VALUE></PROPERTY><PROPERTY.ARRAY NAME="MessageArguments" TYPE="string"><VALUE.ARRAY><VALUE>VMGUESTNAME</VALUE><VALUE>7165CF13-14A4-4760-99BD-5E44FD50E7FF</VALUE><VALUE>Unspecified error</VALUE><VALUE>0x80004005</VALUE></VALUE.ARRAY></PROPERTY.ARRAY><PROPERTY NAME="MessageID" TYPE="string"><VALUE>32000</VALUE></PROPERTY><PROPERTY NAME="OtherErrorSourceFormat" TYPE="string"></PROPERTY><PROPERTY NAME="OtherErrorType" TYPE="string"></PROPERTY><PROPERTY NAME="OwningEntity" TYPE="string"><VALUE>Microsoft-Windows-Hyper-V-VMMS</VALUE></PROPERTY><PROPERTY NAME="PerceivedSeverity" TYPE="uint16"><VALUE>5</VALUE></PROPERTY><PROPERTY NAME="ProbableCause" TYPE="uint16"><VALUE>0</VALUE></PROPERTY><PROPERTY NAME="ProbableCauseDescription" TYPE="string"></PROPERTY><PROPERTY.ARRAY NAME="RecommendedActions" TYPE="string"></PROPERTY.ARRAY></INSTANCE>

[X:\bt\1014568\repo\src\SRS\Service\ProviderFramework\ReplicationProvider\DraPlugins\HyperVReplicaCommon\PluginTaskBase.cs:93 Execute] Hit exception while executing task EnableProtection. Consuming exception and failing DRA job. Exception details:
Dra.DraException: Job 'Enabling Replication' (FC4128BD-XXXX-4937-XXXX-XXXXXXXXXXXX) failed. Job State Exception. Error Code: 32768. ErrorSummaryDescription: Hyper-V failed to enable replication.. ErrorDescription: Hyper-V failed to enable replication for virtual machine 'VMNAME': Unspecified error (0x80004005). (Virtual machine ID XXXXXXX-14A4-4760-XXXX-XXXXXXXXXXXX).
at Dra.Plugin.WmiHelper.ExecuteMethod(IInstance instance, Object scope, String method, Dictionary2 args, String contextDescription, PluginTaskBase taskContext, Dra srsAgent, Boolean runAsync) at Dra.Plugin.ReplicationService.ExecuteMethod(Object scope, IInstance instance, String method, Dictionary2 args, IVirtualMachineBase context, Boolean runAsync)
at Dra.Plugin.ReplicationService.ManageReplicationRelationship(ReplicationRelationshipOperation operation, IVirtualMachine computerSystem, ReplicationSettingData vmReplicationSettings)
at Dra.Plugin.EnableProtection.ProcessAction(Object taskContext, List`1& warningList)
at Dra.Plugin.ReplicationActionTask.ExecuteInternal()
at Dra.Plugin.PluginTaskBase.Execute()

[X:\bt\1014568\repo\src\SRS\Service\ProviderFramework\ReplicationProvider\DraPlugins\HyperVReplicaAzurePlugin\Helpers\ExtensionMethods.cs:255 GetExtendedHealthIssues] Errors occured in fectching health issues for VM 'VMGUESTNAME' (XXXXXXX-14A4-4760-XXXX-XXXXXXXXXXXX). Exception details:
Dra.DraException: PowerShell script (MarsAgentError) execution failed with the following error message: System.Management.Automation.Remoting.PSRemotingTransportException: Connecting to remote server VMHOSTNAME failed with the following error message : Access is denied. For more information, see the about_Remote_Troubleshooting Help topic..

[X:\bt\1014568\repo\src\SRS\DRA\DRA\Tasks\DraPluginAction.cs:180 PerformPluginAction] Plugin 'HvrAzure' (371f87f9-53e5-429f-b10b-f14700a4b137) failed action for job '5374e8e2-55da-49d0-9a7e-347c78c69134' (Name: EnableHvrWorkflow, Type: ReplicationAction, Action: EnableProtection). Job will be failed.

Regarding PSremoting error, I can PSRemote to the VMHOST. It just seems that even though the MARS agent is set up and both the MARS and the provider are already registered (or I couldn't have gotten this far) that PSRmoting to the Hyper-V HOST is failing.

Please guide me, or let me know if you need other logs.

Thanks!

Answer 1

@mfassbin Thank you for contacting us and apologies for delayed response!

Please perform the following steps in respective order

Disable replication for the VM in question from azure portal
Remove hyper V host and Hyper V site from Azure portal
Uninstall ‘Microsoft Azure Site Recovery Provider’ and ‘Microsoft Azure Recovery services agent’ from the Hyper V host
Download the latest ASR provider from azure portal and then install the same on the Hyper V host
https://learn.microsoft.com/en-us/azure/site-recovery/hyper-v-azure-tutorial#set-up-the-source-environment

Try to enable replication again

----------------------------------------------------------------------------------------------------------------------

If the response helped, do "Accept Answer" and up-vote it

Answer 2

@mfassbin That's a great new. Thanks for confirming.