Server 2019 Windows Server Backup is totally unreliable

Question

I ran WK12 (not R2) for eight years or so and never had a problem backing up or restoring. I backed up to a VHDX file on a backup drive. This was really convenient as I could move it around if I had to, even move it to a different hard drive when I had to upgrade disks.

Eventually, I had to upgrade the whole server and W2K12 didn't have the required drivers. Before purchasing a license for Server 2019, I tried the eval version and am glad I did not pay for it. WSB is totally unreliable. I have now lost five sets of backups.

I do perform an external backup once per month and critical files are stored on a laptop as well - but not always. The internal drive is reliable. No SMART issues with anything.

On Server 2019, I created a VHDX file in a partition on the internal backup hard drive.

I basically have a C drive taking up about 140 GB with the OS on it and a couple of apps that won't run in VMs, and a V drive with about 1.3 TB of VMs and files. The Backup VHDX is 3 TB in size with a 2.8 TB partition and 271 GB of unpartitioned space. All hard drive partitions are encrypted with Bitlocker. The V Drive is deduped.

When it gets to about ten days in, backup runs out of space but does not report an error. The logs say the backup is fine. But they aren't. The V Drive does not show up in wbadmin.msc. Worse, wbadmin.msc cannot be closed or killed. It locks up the file system somehow. Virtual machines cannot be shutdown. The OS itself will not shutdown. It will hang for hours on waiting for Hyper-V Manager or if I'm lucky enough to remember to shutdown the VMs first, it will hang on System Event Notifications. I have to turn off power. That's risky and I run ChkDsk on all drives and any drives of VMs that were running at the time. I have corruption in the component store on the server C drive that cannot be repaired - I will have to reinstall the OS eventually.

If I try mounting the backup vhdx files, they can be assigned a drive letter and I can access the last backup, but not previous days' backups.

I imagine there is a problem with the automatic sizing of shadow files. The backup VHDX reports Shadow Copies as disabled and 941 GB used:

Shadow Copy Storage association
For volume: (O:)\?\Volume{35044491-d15c-4715-b4d4-b3206210225f}\
Shadow Copy Storage volume: (O:)\?\Volume{35044491-d15c-4715-b4d4-b3206210225f}\
Used Shadow Copy Storage space: 891 GB (31%)
Allocated Shadow Copy Storage space: 941 GB (33%)
Maximum Shadow Copy Storage space: UNBOUNDED (613566757%)

I count about 35 shadows on the drive.

WSB should make sure this is all setup automatically without any operator intervention or warn if it isn't set up right. I don't know if UNBOUNDED is correct or not.

Please Microsoft, make this reliable again. In the meantime, any tips would be appreciated.

Thanks,
Bob.

Answer 1

@Xiaowei He

Actually, it seems like the entire file system hangs. File Explorer hangs. VMs can access files in their own VHDs/VHDXs, but will hang if you try to access files shared from the server within them.

VMs cannot be shutdown. Services cannot be closed. Some apps cannot be killed. The server cannot shutdown.

The WSB reports do not report a problem (C:\Windows\Logs\WindowsServerBackup):
73048-backup-26-02-2021-07-00-09-xxxx.log
73049-backup-error-26-02-2021-07-00-09.log

The only notification is:

It’s as if the backup completes okay but something goes wrong with the shadows when they try to close. I have tried both VHDX and VHD files and both dynamically expanding and static. I haven't noticed any difference so far. (I do have unpartitioned space at the end of the VHD/VHDX backup file - in case anyone wants to try to duplicate.)

Answer 2

There's more:
74338-backup-error-04-03-2021-07-00-27-xxxx.log

There is definitely a problem here. Yet, it is not reported in my Backup-04-03-2021_07-00-27.log file. It shows:

Backup of volume \?\GLOBALROOT\Device\HarddiskVolume6\ succeeded.
Backup of volume \?\Volume{a8z2507a-a91b-4968-a9c3-9bb9176e7956}\ succeeded.
Backup of volume C: succeeded.
Backup of volume V: succeeded.v

And it lists all my VMs except the VM listed in the above error file. There is no error or warning in this file. Unless one checks for all the VMs to be listed, one would not realize that one was missing.

The Backup Event log does have a warning though:

I'll try to figure out something for this.

Answer 3

Just to update a bit.
I have tried several other options and finally switched to a dedicated partition on a backup hard drive. It has been running fine now for about three months. It keeps backups available for the past week or so depending upon churn.
Bob.