Hyper-V Virtual Machines disappear after rebooting

Question

Hi. I have two servers, one is a Windows Server 2019 with the Hyper-V role and another is a Windows 2016 Server with the Hyper-V role installed. All of my virtual machines are configuration version 8.0. I setup Hyper-V replication to replicate the Virtual Machines on the 2019 host to the 2016 host. This works fine and replicates the VM's I have selected. When I reboot the Windows 2016 server I notice that in the Hyper-V Manager console all the Virtual Machines that had been replicated are gone and do not show up. I found other articles to check the folder C:\ProgramData\Microsoft\Windows\Hyper-V\Virtual Machines for symbolic links however this folder is completely empty. I checked the drive location where I setup the replication and the virtual machines and virtual hard disks are there, so I know the data still exists but why are all the VM's gone? I don't find anything in the event log to indicate an issue, and I have reformatted the 2016 Hyper-V host with a clean install and run into the same issue. Any ideas on why and how to fix this?

Answer 1

Hi John,
I just closed out my case with Microsoft last week on this. Their determination was that the .VMCX files that are replicated over from our 2019 server are corrupt in some way. We did a lot of testing and creating new VM's on the 2019 server and then replicating them over to the 2016 server worked, but not my existing machines. The reason that they go and disappear is when the Hyper-V Virtual Machine Management Service starts up it processes the .VMCX and .VMRS files as it loads, and when it hit's a bad one, it stops loading additional VM's.

The only way of fixing the issue for me was I needed to create a new Virtual Machine, attach the existing .VHDX files to it, and then setup replication. It was not the answer I had hoped for, and I am still stumped as to why the .VMCX files work fine on the 2019 server and it does not do the same behavior with stopping the Hyper-V Virtual Machine Management Service on the same VM's. That is the part Microsoft could not give me an answer to which I still would like to understand.

Lyle

Answer 2

Hi Jacob. I opened up a phone based support case with Microsoft. What they determined (and I disagree) is that the VM's that we originally made on the 2016 server and replicated to the 2019 servers were corrupt when they replicated back to the 2016 Hyper-V machine. They had me make new VM's on 2016 and copy the .vhdx files and attach. That certainly works but doesn't do much good for replication! After the support call I setup another Hyper-V server running 2016 and replicated between two 2016 Hyper-V hosts and no corruption, no problems, just worked as expected. I suspect it has to do with 2016 to 2019 or 2019 to 2016 in an internal issue with MS but they seem to not want to admit that at all.

Answer 3

hi NVGoldendog,

Thanks for your update. I'm afraid you are right. We reproduced the issue yesterday with several VM's from 2019 to 2016.
After internal discussion and consulting with the customer we decided not to waste any time waiting for MS Support because chances of this being picked up by MS are to slimm. (not basing on MS now, but let's face it...)

So we ended up reinstalling our 2016 server with 2019 and replication is working perfectly.
This to me confirmes the source files on the 2019 server are definitely not corrupted and it must be a compatibility issue between 2019 and 2016 even whilst using VM version 8.0

for the next virtum of this issue who finds this email thread... just only replicate between exactly the same server versions;)

Answer 4

Hi,

1. On the Server 2016 host, please test if all VMs will disappear after the host reboot. We may create a test VM on Server 2016, then reboot it, check if only replicated VM disappear, or all VMs disappear from the Hyper V manager.
2. If all VMs disappear from the Hyper V manager, check if restart the VMMS service could fix the issue:
   
3. Besides, please check if the Server 2016 host is updated, if not, please install the latest windows update on the Server 2016 host. Thanks for your time!
   Best Regards,
   Anne

---

If the Answer is helpful, please click "Accept Answer" and upvote it.

Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.

Answer 5

Hi. Thank you for getting back to me. It appears that only the replicated VM's disappear. If I create a brand new VM on the 2016 host and reboot or restart the Hyper-V Virtual Machine Management it does not disappear, only the ones that are replicated from the 2019 host. I do have the latest updates installed as well.

I thought that maybe the VM's on the 2019 Server had some sort of corruption, so what I did to test this is create a brand new VM on the 2016 server, then relicated it to the 2019 server. Then I stopped replication on both servers and deleted the original VM on the 2016 server. I then started a replication task from the 2019 server to the 2016 server. It replicates fine but with reboot or restart of the HVVMM service it disappears.

As a second test I created a new VM on on the Windows 2019 server of configuration version 8.0 and replicated it to the 2016 host. I get the same effect, it disappears with reboot or restarting HVVMM service.

Both servers are updated to the latest Windows updates and do not have AV or anything else installed on them, very plain and clean setup. I should mention I am using SMB to replicate if that has any difference?

Thanks,

Lyle Epstein
Kortek Solutions