Hi,
Thanks for posting on our forum!
Based on your descriptions, I suppose the issue is in accordance with VSS, as you guessed. Windows Backup can run into a allocation problem for Volume Shadow Copy Service (VSS). It may try to prevent the deletion of all your backups to benefit the newest one only. Firstly, check if you have set the status of the VSS service to Automatic.
1) Press Windows key + R to open up a Run dialog box. Once you’re inside the Run box, type “service.msc” and press Enter to open up the Services window.
2) Once you’re inside the Services screen, scroll down through the list of Services (Local) and locate Volume Shadow Copy. Once you see it, right-click on it and choose Properties from the newly appeared context menu.
3) Once you’re inside the Properties screen of the Volume Shadow Copy Service, select the General tab from the list of menu options and change the Startup type to Automatic. Next, click on Apply to save the changes.
4) Restart your computer and repeat the action that was triggering the 0x80780166 error code at the next startup.
Then, you can follow the articles below to set limit for shadow copy size (instead of using the default "No Limit"):
http://desk.plus/windows-backup-encountered-error-writing-data-backup-target/
(Please note: Information posted in the given link is hosted by a third party. Microsoft does not guarantee the accuracy and effectiveness of information.)
Finally, I found a blog which discusses a similar issue with yours and I think you may be interested in it:
Thanks for your support! Have a wonderful day! : )
Best regards
Joan
If the Answer is helpful, please click "Accept Answer" and upvote it.
Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.