Hi Iris,
When backups consistently fail at the 10GB mark across both Veeam and native Windows Server Backup, the issue is almost certainly tied to the underlying VSS snapshot process for that VM rather than the backup software or destination. The “Incorrect function” error is a generic VSS failure, usually triggered when the VM has a corrupt checkpoint chain or a problematic integration service.
You can start by checking the Hyper-V host’s Event Viewer under Applications and Services Logs > Microsoft > Windows > Hyper-V-VMMS > Admin for VSS-related errors at the time of failure. On the affected VM, verify that Hyper-V Integration Services are installed and up to date, and confirm that the Volume Shadow Copy service is enabled and running. Also inspect the VM’s .avhdx chain in C:\ProgramData\Microsoft\Windows\Hyper-V\Virtual Hard Disks for inconsistencies; a broken differencing disk will cause snapshot creation to fail once the backup hits a certain size. If corruption is suspected, merge checkpoints manually and recreate a clean baseline. If everything checks out but the error persists, the VM’s VSS writers may be failing: run vssadmin list writers inside the guest and look for any in a failed state. A failed writer will block backups beyond a certain threshold, and repairing that service or application is required before backups succeed.
If you find it's helpful, please give it a thump up or accept the answer, very much appreciate it :)
Harry.