Problems with Large Files (multi-gigabyte)

First, thanks to anyone who reads this. My issue is not trivial and is explained in depth.

I'm using Windows 7 Ultimate 64-bit, and my problem applies to that system.

This problem is stumping me but I've been able to gather evidence of the onset of the problem, an experiment which is repeatable. It seems either Windows Update and/or MSE delivers a payload that interferes with proper access to large files the first time they access the Internet and transfer any data.

(1) First: Signs of a problem

I had problems with a program that stores data in its program directory using the zip format. Integrity checks of the files would fail (the program internally computes some sort of hash or checksum). Since the files were large I decided to test this issue.

(2) Second: Test preparation on a different computer (Windows XP 32-bit)

I made a large zip file on another computer (about 3.8GB) and computed its MD5 hash. I tested the integrity of the zip file and everything extracted okay. I burned the file to a DVD, and computed the hash of the file on the DVD (it matched the previous hash of the file on the disk drive).

(3) Third: Testing MD5 file hashes on Windows 7

I took the DVD to the Windows 7 computer and computed the MD5 hash of the file on the DVD. It was different. Interestingly, on each reboot of the computer, the computed MD5 would change. When trying to extract the zip file, Windows own zip handler would encounter problems, WinZip would indicate CRC errors, and WinRar would error as well.

(4) Fourth: Safe mode and diagnostic Startup

When booting into safe mode, MD5 hashes on large files compare exactly to the computed hashes on the Windows XP 32-bit machine. In Diagnostic startup, the problem returns. (I have boot logs of the two modes and can post them if that helps).I can't see any difference that stands out and I'm thinking maybe, just as a matter of fact, more is being loaded than what's reported in the boot logs.

(5) Fifth: The onset of the problem on a clean installation

I reinstalled the operating system several times, each time zeroing out my disks before installation. I have successfully repeated the onset of the problem to occur when Windows Update and/or MSE access the Internet for the first time. The problem occurred after Windows Update alerted me to pending updates for the first time (notification but no download) and when MSE downloaded its first set of new definitions. These things happened concurrently. I'm in the process of trying to isolate this issue further.

Note: Before I connected to the Internet for the first time, I installed MSE and download the "standalone" definitions manually on another computer and transferred them to the Windows 7 machine and installed the definitions. This after this, MD5 hashes computed fine, it was only after Windows Update and MSE access the Internet and downloaded data that the problem occurred.

 (6) Sixth: Other Info

Using the Command Prompt to copy a large file from a DVD to the hard disk of the affected computer the line "copy largfile.ext c: /v" returns with "ERROR Verify - c:largefile.ext", and "1 file(s) copied." Through Windows Explorer, a copy operation throws no errors. THIS PROBLEM HAPPENS WITH ANY LARGE FILE I TEST NOT JUST ZIP FILES. I'm not sure where the boundary is in file size that this problem occurs. It's not clear to me that the files are actually corrupted as when I copied large zip files files from my Windows 7 machine to an external drive for backup, and checked those files on another computer, the zip files were fine. I think something is intercepting "correct" access to the large files.

(7) Seventh: When did the problem first happen

I started noticing issues about 3 to 4 weeks ago.

(8) Eight: Another test

This testing has consumed a lot of time. Now that I have a repeatable issue, I'm going to disable the Windows Update Service before I connect to the Internet for the first time, after reinstalling, and see if I can narrow the behavior specifically to either Windows Update or MSE. It may be that KB2533552 could help, I’m not sure. Testing takes a lot of time. Something like “Windows7FirewallControl” may help isolate MSE and Windows Update until this issue is resolved.

I think this is a serious issue as any program that uses integrity checks on large files would be affected. And, anyone wishing to use large archives that perform integrity checks, like zip files, would be affected too -- assuming my experience is not an isolated incident. I believe it is not, as a search over the Internet shows problems with large zip files in an ongoing issue.

Any help would be appreciated.

Best Regards,

Davis