More control over memory mapped files and working set size

Question

I'm having a lot of question regarding memory mapped files, but currently no one was able to answer specific question. Reading Mark Russinovichs' book didn't help to clarify the details.

My problem is that I'm completely relying on memory mapped files since my data is very large. Originally I did my own - lets call if file-based GC and paging - to offload currently unused memory regions into individual user-defined files. But I had to write my own mechanism to manage modified and unmodified regions and map them back and forth manually. This approach worked well, since it started to page in good memory conditions and became more aggressive when the physical RAM was decreasing more and more, also this mechanism had ther ability to stop other threads so that the paging manager was able to do his work and no one else was producing additional memory.

Since this is what memory mapped files also do I wanted to switch to those. The only problem is that it all works well until the commited memory and the working set reach the size of the physical RAM. Then the memory manager starts to page so heavily that the system freezes and sometimes freezes so much that windows had to be restarted. The problem is that I'm obviously producing so much memory in a very short period of time that the timing conditions in the KeBalanceSetManager and the ModifiedPageWriters are not meet. What happens is that pages are written, but I'm still allocating a lot of memory. This leads to the point where Windows is trying to page but doesn't get out of this condition because I'm allocating too much.

The situation would be much easier if I could throttle allocating memory or even stop my application from allocating until good physical memory conditions are met again, but I have no clue if there are methods how to handle this. I cannot ask the current comitted memory size, because event if everything is pages the working set keeps its size. The only remedy is to call EmptyWorkingSet which really trims the working set. Using QueryMemoryResourceNotification didn't help as well since it is obviously bound to the workingset which is only trimmed when calling EmptyWorkingSet.

The question is if I can have either more control of memory mapped files or at least if there is a method of how ask the system about its memory conditions?

Answer 1

Hi,
Maybe this (Memory Performance Information) is what you are looking for.

Alon.