Hello! I have a customer with a three node Windows Server 2019 Hyper-V Failover Cluster. When I backup VM's w/ Veeam, I sometimes get left-over .avhdx files that don't automatically merge. With additional backups, sometimes the VM keeps adding additional .avhdx files - other times there remains only a single .avhdx file. Hyper-V Manager doesn't show any Checkpoints. And when I try to look at the Hyper-V Settings in Failover Cluster manager, the MMC crashes with the error "MMC has detected an error in a snap-in". I have a process to manually fix these VM's: Remove VM from Cluster [this is necessary because the Failover Cluster MMC crashes and I can't edit the VM disks in Hyper-V Manager when the VM is part of a cluster] Shutdown VM [this is a pain] Manually merge .avhdx files in chain back to parent [Merge-VHD or AZSBTools] Fix disk in Hyper-V Manager Settings back to .vhdx Rejoin Cluster The really interesting thing is that [Get-WinEvent Microsoft-Windows-Hyper-V-* | Where-Object{$_.Message -like " <VMName> "} ] shows "background disk merge has been started" and "background disk merge has been finished successfully" for the VM in question. Yet the .avhdx had been created at some point and not cleaned up automatically. I'd love to be able to fix these VM's without having to shutdown the VM. I'd really love to be able to not have these .avhdx files remain and have the system automatically merge them. Any suggestions? I'd certainly be glad to hear them! Dave

It's been a while since I checked in and I wanted to update the status of this issue. While I don't have the problem resolved, it is much more manageable now. I ended up making smaller batches of VM backups in Veeam. It seems like this has helped mitigate most of the problems that I was having with leftover .avhdx disks. I still have an occasional VM that I have to take down and remove from the Cluster & Hyper-V at which point the single .avhdx merges automatically. This is much better than the several VM's per week that I was having earlier and also having multiple .avhdx files per VM disk volume which had be be manually merged.

Hi Dave, Our issue ended up being incorrect/missing permissions on the folder where the VMs were stored. I created a new VMs folder with correct permissions to hold VMs. I then moved the VMs to the fresh VM storage folder to regenerate permissions/ownership. Since completing this, I have had no issues, either with Veeam or creating checkpoints. Shaun

Hey Dave, I am also getting the same issue of extra .avhdx files with no checkpoints showing. My Hyper-V cluster is Win Server 2022. And I've just added the cluster to Veeam. My Failover Cluster manager MMC also fails when looking at VMs Settings. Running Get-VMSnapshot -ComputerName "host server" -VMName "vm name" doesn't show any checkpoints. And if I manually make a checkpoint, it doesn't show up in Hyper-V Manager. When you have run your fix, has the issue returned? Shaun

Hello Shaun, Can you help to send the type of permissions you setup for your vm folders? Note: We have joined all systems to central domain server

Here you go Santosh. We have a Cluster with 3 hosts, with a cluster shared volume for VM storage. Our hosts and VMs are domain joined.

Windows Server 2019 Hyper-V Failover Cluster .avhdx files remaining after backup

Dave Baddorf 101

Hello!

I have a customer with a three node Windows Server 2019 Hyper-V Failover Cluster. When I backup VM's w/ Veeam, I sometimes get left-over .avhdx files that don't automatically merge. With additional backups, sometimes the VM keeps adding additional .avhdx files - other times there remains only a single .avhdx file. Hyper-V Manager doesn't show any Checkpoints. And when I try to look at the Hyper-V Settings in Failover Cluster manager, the MMC crashes with the error "MMC has detected an error in a snap-in".

I have a process to manually fix these VM's:

Remove VM from Cluster [this is necessary because the Failover Cluster MMC crashes and I can't edit the VM disks in Hyper-V Manager when the VM is part of a cluster]
Shutdown VM [this is a pain]
Manually merge .avhdx files in chain back to parent [Merge-VHD or AZSBTools]
Fix disk in Hyper-V Manager Settings back to .vhdx
Rejoin Cluster

The really interesting thing is that [Get-WinEvent Microsoft-Windows-Hyper-V-* | Where-Object{$_.Message -like "<VMName>"} ] shows "background disk merge has been started" and "background disk merge has been finished successfully" for the VM in question. Yet the .avhdx had been created at some point and not cleaned up automatically.

I'd love to be able to fix these VM's without having to shutdown the VM. I'd really love to be able to not have these .avhdx files remain and have the system automatically merge them.

Any suggestions? I'd certainly be glad to hear them!

Dave

Dave Baddorf 101 Reputation points

2022-08-24T19:28:24.433+00:00

Some additional information:

I did some experimenting with Veeam and even with "Guest Processing - Enable application-aware processing" disabled and with "Enable Hyper-V Guest Quiescense" disabled, Veeam still triggers the creation of a Recovery Checkpoint (apparently different than a Standard or Production Checkpoint). So it doesn't appear like I can prevent the creation of this Recovery Checkpoint .avhdx file - I need to find a way to make sure that it gets merged back in automatically.

The Get-VMSnapshot command shows this .avhdx file with a CheckpointType and SnapshotType of "Recovery" instead of "Standard".
Dave Baddorf 101 Reputation points

2022-10-04T17:16:55.783+00:00

For all of those who are interested (which may just be me):

I wrote a script to show me any .avhdx files which exist on the CSV volumes. It runs twice a day after my Veeam Backups. This way I know exactly when these .avhdx files haven't been cleaned up. For example, I had an .avhdx for VM1 show up after the 7:00PM Veeam backup on Oct 1st.

The really intriguing thing is that I can go to that Hyper-V Server's Event Log (Applications and Services Logs - Microsoft - Windows - Hyper-V-VMMS - Admin) and I see the following events:
* Event ID 19070, 7:09:30PM "VM1 Background disk merge has been started"
* Event ID 19090: 7:09:31PM "VM1 Background disk merge has been interrupted"
* Event ID 19070, 7:09:32PM "VM1 Background disk merge has been started"
* Event ID 19080, 7:09:35PM "VM1 Background disk merge has been finished successfully"

The VM's Checkpoint looks like it was properly merged. Yet the .avhdx remained.

I had another VM which left a .avhdx file after Oct 2nd's backup. It also had the disk merge started and finished without the interrupted step. You'd think that this would certainly not leave the .avhdx file behind. Yet it did.

If these VM's Checkpoints were merged successfully, why does the .avhdx files still exist and why does Hyper-V Manager Settings still show the the .avhdx drive for the VM setups?

If anyone (besides myself) has any good ideas, I am certainly willing to entertain them.

8 answers

Dave Baddorf 101 Reputation points

2023-01-10T14:45:41.537+00:00

It's been a while since I checked in and I wanted to update the status of this issue.

While I don't have the problem resolved, it is much more manageable now. I ended up making smaller batches of VM backups in Veeam. It seems like this has helped mitigate most of the problems that I was having with leftover .avhdx disks. I still have an occasional VM that I have to take down and remove from the Cluster & Hyper-V at which point the single .avhdx merges automatically. This is much better than the several VM's per week that I was having earlier and also having multiple .avhdx files per VM disk volume which had be be manually merged.
Please sign in to rate this answer.

1 person found this answer helpful.
Dave Baddorf 20 Reputation points

2024-02-05T15:51:39.9366667+00:00

I am still periodically having this issue with the Checkpoints not merging. Today I tried to do a bit of further research and found that the vmms.exe process on the owner Hyper-V server has a .vmcx and a .vmrs file opened in the Snapshot folder of the offending VM (while the VM is shutdown). These files being opened prevent a Storage Migration of the VM to a different folder. In trying to manually move folders, etc., somehow Hyper-V lost the configuration for the VM, so I ended up doing a restore. But next time this happens, I'd like to see if these same two files (the .vmcx and the .vmrs) are open on another Hyper-V host in the cluster after a migration between hosts. I don't see any problems with permissions - it just seems like a bug in Hyper-V for the vmms.exe to not be able to automatically merge the Checkpoint as it should, but it somehow keeps these files open.
My goal isn't to prevent this Checkpoint merge issue, but just to try to find a way to quickly recover from it without having to remove the VM from Failover Cluster & from Hyper-V and having to manually add it back as a "new" VM. (I could backup and restore the VM or export & import, but that would take a while).
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.
Shaun 6 Reputation points

2023-01-14T02:15:00.91+00:00

Hi Dave,

Our issue ended up being incorrect/missing permissions on the folder where the VMs were stored.

I created a new VMs folder with correct permissions to hold VMs. I then moved the VMs to the fresh VM storage folder to regenerate permissions/ownership.

Since completing this, I have had no issues, either with Veeam or creating checkpoints.

Shaun
Please sign in to rate this answer.

1 person found this answer helpful.

0 comments No comments
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.
Shaun 6 Reputation points

2022-08-19T00:38:16.947+00:00

Hey Dave,

I am also getting the same issue of extra .avhdx files with no checkpoints showing. My Hyper-V cluster is Win Server 2022. And I've just added the cluster to Veeam. My Failover Cluster manager MMC also fails when looking at VMs Settings.

Running
Get-VMSnapshot -ComputerName "host server" -VMName "vm name"
doesn't show any checkpoints.
And if I manually make a checkpoint, it doesn't show up in Hyper-V Manager.

When you have run your fix, has the issue returned?

Shaun
Please sign in to rate this answer.
Dave Baddorf 101 Reputation points

2022-08-24T14:10:33.457+00:00

Hi Shaun,

The short answer is "I don't think so".

It seems like once I fix a VM it is fixed until it isn't. It seems like a game of whack-a-mole, where I fix one VM then another one fails. I have had the same VM fail multiple times, although I have been fixing other issues as I go and I can't guarantee that some of the system-related issues that I fixed weren't the cause of some of the prior checkpoint problems (Windows Updates, CSV issues on one host, etc). I don't see any evidence of any system or network issues at this point - yet I still have new VM's with checkpoints that won't automatically merge

What I do know is that I continue to have new VM's to fix on a weekly basis and I'd really like to get the root problem solved.

Dave
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.
Santosh Dangle 0 Reputation points

2023-01-25T11:40:05.36+00:00

Hello Shaun, Can you help to send the type of permissions you setup for your vm folders?

Note: We have joined all systems to central domain server
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.
Shaun 6 Reputation points

2023-01-26T22:49:22.2366667+00:00

Here you go Santosh.

We have a Cluster with 3 hosts, with a cluster shared volume for VM storage. Our hosts and VMs are domain joined.
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Windows Server 2019 Hyper-V Failover Cluster .avhdx files remaining after backup

8 answers

Your answer