How to simulate an Out-Of-Memory (OOM) condition in a Win32 desktop application

Question

Hi,

I’m working on a Win32 desktop application and trying to simulate an out-of-memory (OOM) condition within my application.

To make the test more realistic, I disabled paging by setting “No paging file” in the Windows virtual memory advanced settings.

In my application, I continuously allocate memory using VirtualAlloc with MEM_RESERVE | MEM_COMMIT, storing all returned pointers so the memory is not released. The allocation strategy is as follows:

• First, allocate 1 GB chunks in a loop until allocation fails.
• Then allocate 10 MB chunks in a loop until allocation fails.
• Finally, allocate 50 KB chunks in a loop until allocation fails.

At this point, VirtualAlloc fails consistently, indicating that user-mode memory is exhausted.

However, after reaching this state, I am still able to:

• Create a Win32 window
• Display a Win32 MessageBox

Even with paging disabled and no further user-mode memory available, these UI elements continue to work. I understand that Win32 windows and message boxes rely on kernel-mode memory (USER/GDI subsystems), which appears to remain available.

My questions are:

Is it possible to simulate an OOM condition by exhausting kernel memory (e.g., USER/GDI or non-paged pool) from a user-mode Win32 application?

If so, what is the safest and recommended way to do this without crashing or destabilizing Windows?

If not, what is the correct or industry-standard approach to testing OOM handling in Win32 desktop applications?

Thanks in advance for any insights.

Answer 1

Hi @Harshithraj1871 ,

Is it possible to simulate an OOM condition by exhausting kernel memory (e.g., USER/GDI or non-paged pool) from a user-mode Win32 application?

It's possible but I strongly advise against this approach. Exhausting kernel resources like the non-paged pool can destabilize or crash the entire system, not just your application. Windows has built-in protections, but pushing these limits is risky.

If not, what is the correct or industry-standard approach to testing OOM handling in Win32 desktop applications?

Consider these industry-standard practices:

• Use Windows Application Verifier with Low Resource Simulation - this tool can simulate memory allocation failures without actually exhausting memory
• Implement fault injection in your code using preprocessor directives or a test mode that makes allocations fail after certain thresholds
• Use memory limit tools like Job Objects (SetProcessMemoryLimit) to constrain your process to a specific memory budget
• Virtual machine testing - run your app in a VM with very limited RAM (e.g., 512MB) to naturally trigger memory pressure

---

You've correctly identified that USER/GDI subsystems use kernel memory. This is actually by design - Windows tries to keep critical UI functions alive even under memory pressure so users can at least see error dialogs and potentially save work.

It's better to focus on testing your application's own allocation failures rather than trying to exhaust all system memory. Most real-world OOM scenarios in production will fail your app's allocations while the system remains functional.

Answer 2

Maybe these tools can be useful: Testlimit, NotMyFault, CpuStress:

• https://learn.microsoft.com/en-us/sysinternals/

Answer 3

Simulating an Out-Of-Memory (OOM) condition in a Win32 desktop application can be complex, especially when considering the distinction between user-mode and kernel-mode memory. As you've observed, exhausting user-mode memory with VirtualAlloc can still allow certain UI operations because they rely on kernel-mode memory, which may not be exhausted in the same way.

To address your questions:

1. Simulating OOM by exhausting kernel memory: It is generally not feasible to directly exhaust kernel memory (like USER/GDI or non-paged pool) from a user-mode application in a controlled manner. The kernel manages these resources, and they are typically allocated based on system needs and limits. Attempting to exhaust kernel memory can lead to system instability or crashes, which is not recommended.
2. Safest and recommended way to test OOM handling: The industry-standard approach for testing OOM conditions usually involves:
   • Using tools: Consider using specialized testing tools or frameworks that can simulate OOM conditions without directly manipulating system resources. These tools can help you test how your application handles memory allocation failures gracefully.
   • Mocking memory allocation: Implementing a mock memory allocator in your application that simulates allocation failures can provide a controlled environment for testing OOM handling without risking system stability.
   • Monitoring and logging: Ensure that your application has robust logging and monitoring in place to capture how it behaves under low-memory conditions, allowing you to analyze its response without needing to exhaust kernel resources.

In summary, while you can simulate user-mode OOM conditions effectively, directly exhausting kernel memory is not advisable. Instead, leverage testing tools and mock implementations to safely evaluate your application's memory handling capabilities.

---

References:

• Preventing Memory Leaks in Windows Applications
• RAM, virtual memory, pagefile, and memory management in Windows
• Reserving and Committing Memory
• Memory use in high-level applications (visual-studio)