Windows is leaking memory when reading random chunks of a large file

Question

Windows is leaking memory when reading random chunks of a large file

Nazar Mokrynskyi 0

I'd like Windows developers to look at the issue reported here: https://internals.rust-lang.org/t/high-memory-usage-on-random-file-reads-using-seek-read-windows/20396 Tl;DR: A small Windows app (in the thread linked above) reads tiny chunks of a large 500G file at random and uses 0.5M of RAM, while total memory used by Windows increases by ~1G instead and I have no idea where it goes. This is MUCH worse when even larger files are involved, similar app easily makes Windows use over 100G of RAM for no good reason. Reproducible on Windows 10 and 11. Originally asked at https://answers.microsoft.com/en-us/windows/forum/windows_10-performance/windows-is-leaking-memory-when-reading-random/965c422c-cb71-4dbb-9d3e-49dfa7b9df76, but was recommended to ask here instead.

2 answers

Your answer

Answer 1

MotoX80 36,291

I have no idea where it goes.

Use the Vmmap and Rammap tools to analyze memory.

https://learn.microsoft.com/en-us/sysinternals/downloads/vmmap

https://learn.microsoft.com/en-us/sysinternals/downloads/rammap

Start by running your program against a small file. If there is a memory leak bug in your code, or as a result of some Windows API call, you should see memory gradually increase the longer your program runs. And if you let it run long enough it should eventually consume all available memory.

In the process explorer and task manager images that you posted the private bytes graph shows a consistent value. To me that does not indicate a leak. If your memory usage increases only when size of the file it reads increases, then I would suspect that Windows or Rust is just caching the file contents.

Look at the File Summary and File Details tabs in Rammap. Vmmap will also show Mapped File details.

FYI, I am not familiar with the Rust programming language.

Nazar Mokrynskyi 0 Reputation points

2024-02-28T20:00:55.79+00:00

I don't think there is a leak in the app as such, if you look at the code, it literally reads into pre-allocated piece of memory (no dynamic heap allocations as far as I can see) using NtReadFile. I strongly suspect there is a bug in Windows somewhere because what I see makes no sense.

Memory usage grows gradually over time, but it seems to level-off at some point. The amount of memory used at which it seems to stop growing appears to be a function of a file size.

I will check the apps you've linked, but I'm really hoping to catch attention of someone who knows Windows internals, ideally one of the developers, but I'm having a hard time to find a good place to reach out to someone like that.
Nazar Mokrynskyi 0 Reputation points

2024-03-01T00:27:19.9066667+00:00

I tried both apps. RamMap shows that "Mapped File" uses that "invisible" memory, the diff before and after is ~1G that I expect:

However, I'm not using memory mapping! Looking at VMMap of the process shows that it is not my app that is mapping those files, but I can't find what does anywhere:

Is it something secretly buggy in the kernel? How do I see what exactly? Can I even?I do see in RamMap that it is 100% mapping my file though.
MotoX80 36,291 Reputation points

2024-03-01T01:48:45.2533333+00:00

Its the system cache.

See the image in the link.

https://learn.microsoft.com/en-us/windows-server/administration/performance-tuning/subsystem/cache-memory-management/

Rammap explanation.

https://techcommunity.microsoft.com/t5/ask-the-performance-team/introduction-to-the-new-sysinternals-tool-rammap/ba-p/374717

Mystery solved. The outlook offline storage file is taking up the most RAM, followed by my local search index and several thousand smaller files. However, it is important to remember that in this case the memory is almost all on the Standby list, so the OS is simply using the “extra” RAM to cache data for faster access.
Nazar Mokrynskyi 0 Reputation points

2024-03-01T22:49:00.99+00:00

That is interesting, thanks for sharing @MotoX80 !
But doesn't explain the behavior or at least it doesn't make logical sense to me.

I tried opening a file both with FILE_FLAG_RANDOM_ACCESS and without, in either case I (very surprisingly) see that it is actively used memory (not cache) that is being occupied, which doesn't make sense. And note that the program doesn't write anything, file is open as read-only.

I also see memory mapping for a file I didn't request to create memory mapping for. I specifically refactored an app to move away from memory-mapped files because they caused more issues than they helped (and looks like they cause issues again here).

And caching for a file that was open with random access hint makes zero logical sense to me, there is literally no point in caching anything there, all reads are literally unpredictable. I don't have to suffer doing unbuffered (direct) I/O just to make kernel not cache something that I already told it can't help.

Am I misunderstanding it or are those 3 actual bugs in Windows? I'm convinced those are bugs, but I don't know how/where to report them.
MotoX80 36,291 Reputation points

2024-03-02T14:51:29.6066667+00:00

See https://learn.microsoft.com/en-us/windows/win32/fileio/file-caching

Try setting FILE_FLAG_NO_BUFFERING when you open the file and see if that does want you want.
Nazar Mokrynskyi 0 Reputation points

2024-03-02T16:32:52.7933333+00:00

I know about FILE_FLAG_NO_BUFFERING and want to avoid it because it is a huge PITA and comes with a bunch of restrictions that would require me to re-design an app or write custom adapters and refactor things to use those adapters.

I shouldn't have to do that for a trivial use case that I have, don't you think? Windows literally does something I already told it NOT to do.
MotoX80 36,291 Reputation points

2024-03-02T17:49:31.7833333+00:00

I'm sorry, I don't know what to tell you. The first sentence in that link: By default, Windows caches file data that is read from disks and written to disks.

It is my understanding that if your system has plenty of free memory, then the caching mechanism will try to use it (because nothing else is using it) by caching file data to speed up performance. A page fault to map the I/O request to cache memory is faster than a physical disk I/O.

The alternative is to tell Windows to not cache the file data, but that apparently won't work for you either.

If the cache's use of memory is causing "no memory available" or "out of memory" errors in other applications, then I would agree that there is a bug somewhere.

Otherwise, it looks like this is just how windows works.

Any comments from other forum users?
Nazar Mokrynskyi 0 Reputation points

2024-03-02T20:29:49.6666667+00:00

It absolutely does run out of memory and users complained about this numerous times.

I have no issues with Linux saying I have 10MiB of FREE memory on my 128GiB system because I know I have lots of AVAILABLE memory and most of non-FREE is just cache that can be gone any second.

On Windows however that is actively used memory, not cache. So while the rest of Windows is trying to do something about it like actively swap things to disk and slow down to a crawl, eventually it does crash for some users and some users live with 99% used memory and sluggish experience for no reason whatsoever.

I appreciate your help and I definitely learned a few things over last few days, but as I said in the very first line originally, I really need to get to Windows developers somehow to fix their arguably broken OS.

P.S. And this is not the only bug, I am also struggling with https://stackoverflow.com/q/78058306/3806795, which seems to be another bug in either OS or file system driver specifically.
MotoX80 36,291 Reputation points

2024-03-03T16:25:20.51+00:00
Just a thought, have you tried periodically closing and reopening the file?

Also, Rammap can empty the standby list. Try periodically running doing that.

rammap -Et
Nazar Mokrynskyi 0 Reputation points

2024-03-03T17:46:43.9366667+00:00

Reopening file periodically is one of the first things I tried, made zero visible difference
MotoX80 36,291 Reputation points

2024-03-03T19:34:41.2866667+00:00

Try emptying the standby list.
Nazar Mokrynskyi 0 Reputation points

2024-03-03T20:02:01.3966667+00:00

Some users did it and it seemed to help, but for a very short period of time, memory usage quickly grows back where it was right after that.

Answer 2

Nazar Mokrynskyi 0

Live chat support representative suggested to post in yet another place, mentioning here for continuity: https://techcommunity.microsoft.com/t5/microsoft-learn/windows-is-leaking-memory-when-reading-random-chunks-of-a-large/m-p/4074012

Share via

Windows is leaking memory when reading random chunks of a large file

2 answers

Your answer