Shadow Copy snapshot file contents silently corrupted on Windows 8.1

Question

Historic overview:

One user had problems with Folder Redirection sync resulting in data loss (details are irrelevant) and we had to recover missing data from PC's VSS snapshot of CSC cache (regular VSS snapshot by System Restore).

In short steps:

1. Elevate to SYSTEM with PsExec (CSC folder is heavily protected by ACL so we have to use SYSTEM to access it)

psexec -i -s -d CMD 2. Get relevant shadow copy with

vssadmin list shadows 3. Mount shadow copy with

mklink /d C:\ShadowMount \?\GLOBALROOT\Device\HarddiskVolumeShadowCopy1
4. Recover data with

robocopy C:\ShadowMount\Windows\CSC...\data D:\Backup /E

No errors were seen and a few test files were consistent. About a month goes by and user reported that most of the data was corrupted. We determined that data had been corrupted already during recovery from Shadow Copy.

We have been able to reproduce this on 3 Windows 8.1 systems so far (not tested on 7 yet).

We also tried to over Network provider just in case, but we still see corruption

1. net use vss$=\?\Globalroot...
2. popd \localhost\vss$
3. Access data

Symptoms:

Snapshot age seems to be largely irrelevant (corruption in both snapshots taken a month ago and yesterday).

Parts of the file or whole files are filled with NULLs. It seems that NULLs occur at cluster borders (4kB clusters).

It does not matter if file exists on live volume or has been deleted/moved since snapshot.

File is OK on live volume.

For example one text log file (for easy content analysis), size ~244kB analyzed in Hex Editor

1. Beginning of file has 12kB of data (last data char is position 2FFF)
2. After that, only 00 characters (NULL)
3. Data continues at position 3B000 until end of file

ChkDsk /scan shows no errors. The system has been patched up-to-date with WU patches. No LDR hotfixes deployed to my knowledge.

Systems are running regular HDD (so TRIM hasn't cleared up clusters).

Background defragment is enabled.

No VSS errors in Event Log.

The files were very unlikely to be in use (old documents, images, beforementioned old log file) during snapshot.

We're aware that Previous Versions has been dropped in Windows 8, but underlying technology still exists and should continue to work.

We are considering that this might be a bug in VSS. Shadow copy is not consistent and parts of it are either dropped, overwritten or... something.

Might it be a bug or am I missing something? Should we get a MS support case?

We didn't find any relevant KB articles for Windows 8.1 but found one similar case: http://superuser.com/questions/888383/shadow-copy-recovered-files-contain-lots-of-null-blocks

Windows 7 and previous seem to have had a similar case in the past: https://support.microsoft.com/en-us/kb/2748349

Answer 1

Hi

I created a MS Support Case back in summer. My support incident 115060812822144 was closed as confirmed bug and my contact said that there would be a hotfix (but no ETA). There haven't been any updates since July...

I suggest you also create an official support case to increase visibility, you'll get your money back if the issue is confirmed to be a product fault.

Answer 2

While we wait for Microsoft to fix this, i have found a workaround. Unfortunately it wont repair previous shadow copies, but those  created with this  seems to work.

Here how to do it

You should download the vshadow command line  utility from microsoft site or here "http://edgylogic.com/blog/vshadow-exe-versions/" (I use the server 2008 R2 version)

and copy in a working directory or in the system32 directory

then create a Task in task scheduler to automate the shadow creation (need administrative rights).

You can see the option (quite a few) in the terminial windows with vshadow /? or the microsoft site (i use "vshadow -p -scsf c:")

Sorry for the bad english

Answer 3

Exactly the same problem here. Windows 8.1 and Windows 10.

Some files (a lot !) are corrupt in the same way.

Do you have some news of Microsoft ? A fix number ?

I want to restore my files :'(

I discovered the same on Windows 10 Enterprise LTSB.

1. Make a Bunch of files (images are very good) with cheksums
2. create snapshot
3. mount snapshot
4. discover corrupted files and bite your desk

http://imgur.com/EJdnVRp

And this is from a snapshot created several seconds ago. Its the same with Windows 10 Pro 1511 , although it works with Server 2016 TP4 which should have the same codebase.

Answer 4

I'm having the same problem:

Over time Shadow Copy files without extension become corrupted..

Last time I opened that shadow copy file it was just corrupted with random characters, but when I opened it today, 8 days later - it's overwritten with movie subtitles and some xml screen gamma configurations (when I scrolled to middle) which were on my computer...

Answer 5

One could learn this the hard way. Given the fact the System restore and File protection (Volume shadow copy for restoring previous versions of files) is announced as safety switch in case something happens on your PC, and it has been the most "popular resolution" for all ransomware infected and affected users, it is really not fair that this issue has not been either pointed out, announced, or resolved. Instead, giving the users insurance that they are safe for such cases, only to find out the hard way that they've actually not been protected and that they lost it all, is really wrong and actions should be taken for this!

We've just been running extended tests on our company, with me being the alfa IT specialist, and found out that almost none of the shadow copies on 6 different types and sizes of SSD and 14 HDD, on Windows 7, 8 and 8.1, together tested 63 machines/systems, (yes, that's what we've done to make as tight conclusions and resolutions as possible) were restored without issues. Most of the systems were clean installed, while few of them older installations therefore having some older restore points as well. Timeframe 3 weeks, giving us enough VSS to try to recover some files as per @drew010 suggestion, we've had some luck, but found it difficult to make it an automatic routine, as it's almost impossible to determine what part of the file has changed since or not, and since non-changed files are not being shadowed therefore the shadow copy usually contains only one copy of file, it being the one with null data in it.

We've made these tests after dozen of our clients reported that our ransomware recovery resolutions were not successful (victims not having backups elsewere and having VSS on were restored using that one, but files were never checked).

We've also found out that there is probably something about spacing as well. I'll try to describe one example:

HDD: 240GB;

Host OS: Windows 7 Home Premium

Partitions: 2, C:\ 190GB, D:\ 30GB,

Shadow copy enabled: C:\ only

Space used on C:\  : 160GB,  70GB user files

Dedicated maximum space for VSS: 9,5GB

Space in used on the day of restoration, reported by the VSS settings form: 4,2GB

Files affected: all user files on hard drive, ransomware infected test host OS

Previous versions of files available on pretty much all folders

Trying to restore several folders with 200+ MB size: all "successful", all included corrupted/empty .txt, .jpg, .rar files (rar archives being reported with "corrupted header" error, containing only parts of archive, able to extract those...)

Trying to restore folder with 12GB .avi encrypted files: recovery collapsed during process, 2 files being falsely recovered, one being corrupted, and other being partially recovered.

Available previous versions of files: 3 points, (as per 3 restore points stored in host OS at the time).

Main question: Why and how did the OS try to restore files that were bigger than his whole used VSS copy at the time?

Why did the VSS only used half of available space (aka it did not run out of space for this to occur), when it should use all he has available, if even more files were changed?

Why it is reported that shadow copy of the files are available, if they are not really there, as they can't be due to space limit?

Why even though it didn't have enough space for even one single shadow copy, it yet suggested to go as far as 3 versions back with it?

This sound like a seriously unstable situation and feature bug. And being completely M$ oriented company as well as their representative, it's giving me serious concerns as per what we're distributing, as this is the second serious thing we've discovered that affected lots of our clients.

First being the software RAID (mirror) function inside the disk manager itself, where the mirrored disk gets synced - but in a way that it's completely whole mirrored/overwritten again from the source, after every system crash (aka not proper shutdown), giving the actual point of mirroring exactly the opposite of what it's been made for, and also giving headaches with worn out hard drives due to constant writing that shouldn't been made (I'm totally aware that system crashes shouldn't be happening either, but then again, that's not the answer to the main issue.)

I'm really looking forward into someone explaining the VSS issue and space allocation!

Best regards,

MarcS.