Out of memory? Easy ways to increase the memory available to your program
When you run your VB or C# application, you might get an OutOfMemoryException thrown, even if your machine has lots of memory.
Every 32 bit process has a 2^32 bit (4 Gig) address space. That means every pointer has a size of 32 bits (4 bytes) and thus is limited to 4 Billion.
That’s the equivalent of saying a vehicle license plate number consists of 6 digits and thus there are 1 million possible numbers.
That 4 Gigs is divided into half: the user application gets the lower half and the OS gets the upper. (This boundary can be changed: see below).
Start VS 2010. File->New->Project->VB or C# Windows WPF Application.
Paste the VB or C# code below. It creates a heap then allocates 100Meg of memory in a loop continuously until an exception is thrown.
On my 64 bit Windows 7 machine with 8 Gigs of RAM (your digital camera or phone might have more memory!), I get about 1.4Gig allocated before it dies.
Iter #10 1,048,576,000
Iter #11 1,153,433,600
Iter #12 1,258,291,200
Iter #13 1,363,148,800
Exception Exception of type 'System.OutOfMemoryException' was thrown.
Now choose Project->Properties->Compile->Build Events->PostBuildEvent Command and added these 2 lines
call "$(DevEnvDir)..\..\vc\vcvarsall.bat" x86
"$(DevEnvDir)..\..\vc\bin\EditBin.exe" "$(TargetPath)" /LARGEADDRESSAWARE
Note: the positions of the quotes are critical
The first line calls a BAT file that makes various tools available on the path.
The second runs EditBin on the target binary, using the LARGEADDRESSAWARE flag (that’s almost all left hand keys on the keyboard!)
Also uncheck the option: Project->Properties->Debug->Enable the Visual Studio Hosting Process
The only effect of these 2 lines is to call EditBin to toggle a bit in the EXE. When the EXE starts a process, that entire process is flagged as able to work with pointers above 2G.
With such pointers, the high bit is a 1, which, in 2’s complement notation, is a negative number, and some applications may not be designed to work with “negative” pointers.
Now when I run the code I get 3.5 Gigs: More than twice as much memory!
Iter #30 3,145,728,000
Iter #31 3,250,585,600
Iter #32 3,355,443,200
Iter #33 3,460,300,800
Exception Exception of type 'System.OutOfMemoryException' was thrown.
This Editbin “trick” works fine on a 64bit OS. For a 32bit OS it works too, but you need to "bcdedit /set IncreaseUserVA 3072" (reboot) and you won’t get as much extra memory.
Want even more memory? If you’re on a 64 bit OS, try compiling to 64 bit:
VB: Project->Properties->Compile->Advanced->Target CPU->Any CPU (or x64)
C#: Project->Properies->Build->Platform Target->Any CPU (or x64)
Iter #110 11,534,336,000
Iter #111 11,639,193,600
Iter #112 11,744,051,200
Iter #113 11,848,908,800
Iter #114 11,953,766,400
Exception Exception of type 'System.OutOfMemoryException' was thrown.
Yes, that really is almost 12 gigs on my 8 gig machine! You can also verify via Task Manager.
A 64 bit process has a pointer of size 64 bits (8 bytes). 2 ^ 64= 18,000,000,000,000,000,000. (1.8 x 10^19)
That’s about 18 exabytes, or 900,000 years of DVD quality video.
Still that’s nowhere near the number of possible chess games or even the waaaay smaller number of electrons that will fit in the universe J(Hubble's law, big bang physics)
When I first worked for Microsoft, I developed a thunking mechanism to allow 32 bit applications to call 16 bit DLLs. (https://support.microsoft.com/kb/139070 )
We thought there’s no way anybody could run out of 32 bit address space!
If a license plate number were to double its length from 6 to 12 digits, that would go from 1 Million (population of a medium sized city) to 1Trillion (1x 10^12) , 200 times the world population.
See also
Using multiple heaps efficiently
What is your computer doing with all that memory? Write your own memory browser
Create your own Test Host using XAML to run your unit tests
Undocumented APIs and 16 bit DLLs
<C# Code>
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Windows;
using System.Windows.Controls;
using System.Windows.Data;
using System.Windows.Documents;
using System.Windows.Input;
using System.Windows.Media;
using System.Windows.Media.Imaging;
using System.Windows.Navigation;
using System.Windows.Shapes;
using System.Runtime.InteropServices;
using System.Diagnostics;
namespace Heapcs
{
/// <summary>
/// Interaction logic for MainWindow.xaml
/// </summary>
public partial class MainWindow : Window
{
public MainWindow()
{
InitializeComponent();
}
private void Window_Loaded(object sender, RoutedEventArgs e)
{
var hHeap = Heap.HeapCreate(Heap.HeapFlags.HEAP_GENERATE_EXCEPTIONS, 0, 0);
// if the FriendlyName is "heap.vshost.exe" then it's using the VS Hosting Process and not "Heap.Exe"
Trace.WriteLine(AppDomain.CurrentDomain.FriendlyName + " heap created");
uint nSize = 100 * 1024 * 1024;
ulong nTot = 0;
try
{
for (int i = 0; i < 1000; i++)
{
var ptr = Heap.HeapAlloc(hHeap, 0, nSize);
nTot += nSize;
Trace.WriteLine(String.Format("Iter #{0} {1:n0} ", i, nTot));
}
}
catch (Exception ex)
{
Trace.WriteLine("Exception " + ex.Message);
}
Heap.HeapDestroy(hHeap);
Trace.WriteLine("destroyed");
Application.Current.Shutdown();
}
}
public class Heap
{
[DllImport("kernel32.dll", SetLastError = true)]
public static extern IntPtr HeapCreate(HeapFlags flOptions, uint dwInitialsize, uint dwMaximumSize);
[DllImport("kernel32.dll", SetLastError = true)]
public static extern IntPtr HeapAlloc(IntPtr hHeap, HeapFlags dwFlags, uint dwSize);
[DllImport("kernel32.dll", SetLastError = true)]
public static extern bool HeapFree(IntPtr hHeap, HeapFlags dwFlags, IntPtr lpMem);
[DllImport("kernel32.dll", SetLastError = true)]
public static extern bool HeapDestroy(IntPtr hHeap);
[DllImport("kernel32.dll", SetLastError = true)]
public static extern IntPtr GetProcessHeap();
[Flags()]
public enum HeapFlags
{
HEAP_NO_SERIALIZE = 0x1,
HEAP_GENERATE_EXCEPTIONS = 0x4,
HEAP_ZERO_MEMORY = 0x8
}
}
}
</C# Code>
<VB Code>
Option Strict On
Imports System.Runtime.InteropServices
Class MainWindow
Private Sub Window_Loaded(ByVal sender As System.Object, ByVal e As System.Windows.RoutedEventArgs) Handles MyBase.Loaded
Dim hHeap = Heap.HeapCreate(Heap.HeapFlags.HEAP_GENERATE_EXCEPTIONS, 0, 0)
' if the FriendlyName is "heap.vshost.exe" then it's using the VS Hosting Process and not "Heap.Exe"
Trace.WriteLine(AppDomain.CurrentDomain.FriendlyName + " heap created")
Dim nSize As UInteger = 100 * 1024 * 1024
Dim nTot As ULong = 0
Try
For i = 1 To 1000
Dim ptr = Heap.HeapAlloc(hHeap, 0, nSize)
nTot += nSize
Trace.WriteLine(String.Format("Iter #{0} {1:n0} ", i, nTot))
Next
Catch ex As Exception
Trace.WriteLine("Exception " + ex.Message)
End Try
Heap.HeapDestroy(hHeap)
Trace.WriteLine("destroyed")
End
End Sub
End Class
Public Class Heap
<DllImport("kernel32.dll", SetLastError:=True)> _
Public Shared Function HeapCreate(
ByVal flOptions As HeapFlags,
ByVal dwInitialSize As UInteger,
ByVal dwMaximumSize As UInteger
) As IntPtr
End Function
<DllImport("kernel32.dll", SetLastError:=True)>
Public Shared Function HeapAlloc(
ByVal hHeap As IntPtr,
ByVal dwFlags As HeapFlags,
ByVal dwSize As UInteger
) As IntPtr
End Function
<DllImport("kernel32.dll", SetLastError:=True)>
Public Shared Function HeapFree(
ByVal hHeap As IntPtr,
ByVal dwFlags As HeapFlags,
ByVal lpMem As IntPtr
) As Boolean
End Function
<DllImport("kernel32.dll", SetLastError:=True)>
Public Shared Function HeapDestroy(
ByVal hHeap As IntPtr
) As Boolean
End Function
<DllImport("kernel32.dll", SetLastError:=True)> _
Public Shared Function GetProcessHeap(
) As IntPtr
End Function
<Flags()>
Public Enum HeapFlags
HEAP_NO_SERIALIZE = &H1
HEAP_GENERATE_EXCEPTIONS = &H4
HEAP_ZERO_MEMORY = &H8
End Enum
End Class
</VB Code>
Comments
Anonymous
September 30, 2010
Why the 64bit process OOMs at 12 GB?Anonymous
October 04, 2010
The reason why the 64bit process runs out at 12GB is down to the page file commit limit of Windows, 12GB isn't a hard limit though. In it's default settings, Windows sizes the page file depending on how much physical RAM is in your system, the more RAM, the more it will put into the page file. But you can also change the page file settings manually.Anonymous
October 07, 2010
This should be built into the .NET base classes and not require interop to kernel32.dll.Anonymous
October 09, 2010
Just because this post uses the heap functions doesn't mean you have to use interop to access it. It is easier to show this kind of thing at work using the Windows heap functions rather than using the framework. With the heap functions, you can use one variable, allocate and then forget about it leaking memory until you reach the limit. With the framework you have to keep everything referenced otherwise the gc will start reclaiming memory or do other things to make the gc not clean up, but either way, it will not be as simple as the above code.Anonymous
November 22, 2010
To verify if an EXE file is Large Address Aware or not: Open a Visual Studio Command Prompt (Start->All Programs->Microsoft Visual Studio 2010->Visual Studio Tools->Visual Studio Command Prompt Link /dump /headers <filename> The output will include the line “Application can handle large (>2GB) addresses” Here’s a sample output: Microsoft (R) COFF/PE Dumper Version 10.00.30319.01 Copyright (C) Microsoft Corporation. All rights reserved. Dump of file MemSpect.exe PE signature found File Type: EXECUTABLE IMAGE FILE HEADER VALUES 14C machine (x86) 4 number of sections 4CEA07E2 time date stamp Sun Nov 21 22:04:18 2010 0 file pointer to symbol table 0 number of symbols E0 size of optional header 122 characteristics Executable Application can handle large (>2GB) addresses 32 bit word machineAnonymous
November 29, 2011
I am using visual basic 2010 express. I can't find Project->Properties->Compile->Build Events or Project->Properties->Compile->Advanced->Target CPU (target framework but not target cpu) Any hints please on what I should do?Anonymous
December 11, 2011
Right click the project name in the solution explorer toolbar and you'll see at the bottom the 'properties' menu.Anonymous
July 05, 2012
When using multiple class libraries in a solution with one exe, do all the project need these lines added? call "$(DevEnvDir)....vcvcvarsall.bat" x86 "$(DevEnvDir)....vcbinEditBin.exe" "$(TargetPath)" /LARGEADDRESSAWARE I am assuming, yes. thanksAnonymous
November 07, 2012
Being able to store information for users and for the application in server memory is really awesome when you are developing asp.net web applications. I have in the past abused it a little and at on point I had a web application which would actually throw a OutOfMemory exception every now and then. I started searching for tool which would help me analyze session objects and application state objects. After an extensive search, I came accross the Asp.Net Web Profiler, which allows for viewing session variable values which has been declared in different sessions and their sizes to be particular. It displays variables and objects which has been declared in the application state, the current session state as well as in all user sessions and the asp.net cache. You can download it from <a href="http://www.aspwebprofiler.com">www.aspwebprofiler.com</a> It also alows for drill down into object properties in order to detect large objects and optimizing them for memory usage. This way I could detect large sessions and realized that I had the same large object which runs in every user session. I moved it to the application state and my problem was solved.Anonymous
November 27, 2012
Does the fact that IntPtr is a signed value impact the ability to use this type (e.g. for PInvoke) when LargeAddressAware is enabled? Or does it still marshal cleanly to other native code?Anonymous
December 13, 2012
The comment has been removedAnonymous
June 25, 2013
HI, what about windows service? Is there any way to use this solution there? I am trying to apply this solution to service but i still got nothing... :( pease help meAnonymous
August 25, 2013
Hi, I tried this call "$(DevEnvDir)....vcvcvarsall.bat" x86 "$(DevEnvDir)....vcbinEditBin.exe" "$(TargetPath)" /LARGEADDRE but when I build solution it shows error Error 4 The command "call "C:Program Files (x86)Microsoft Visual Studio 10.0Common7IDE....vcvcvarsall.bat" x86 "C:Program Files (x86)Microsoft Visual Studio 10.0Common7IDE....vcbinEditBin.exe" "C:UsersikrishnaDocumentsRectool With Versionversion-8-22-2013ReconciliationToolReconciliationToolbinReleaseReconciliationConsole.exe" /LARGEADDRE " exited with code -1073741515. ReconciliationConsole Please help me.Anonymous
July 31, 2014
Hi, I am having an out of memory issue with a simulation model I am building in vb.net. I'm using visual studio 2008 and I have added the two lines stated above to the Post BuildEvent Command. I have created a new Heap class using your vb.net code and then I added your 'main window' code to my frmMDI so that it's called when the model loads. This doesn't seem to have made any effect on my model and I still hit an out of memory error. Have I made a mistake in the way I've implemented your code? Would appreciate any help - thanks :)Anonymous
April 20, 2015
Thank you this helped a lot on memory exception.Anonymous
April 28, 2015
Hey this code not working in Windows phone I getting DllImport cannot be used on user-defined methods.Anonymous
September 28, 2015
Thank you. Our 32bit application would crash often very close to the 1.4G mentioned. The task manager would report that the process was using between 1.1G and 1.8G when it crashed. Presumably it was crossing the 1.4G boundary which caused the crash. This problem has now gone. Our application was in visual basic, so I discovered that C++ needs to have been installed for EditBin.exe to exist in the visual studio 2010 path.Anonymous
December 29, 2015
I am posting this comment to let you know that you have literally saved my startup with this single blog. Our's is an Animation heavy smart class for schools. We developed it on 64 bit. Turned out the schools are still struck with 2GB dual core ( luckily W7). We are unable to get big simulations working untill I found your post. Thanks mate.Anonymous
October 06, 2016
Solved my problem! ThankssssssssssssssssAnonymous
November 06, 2016
Great post, this saved me years later.Cheers!http://www.systemoutofmemory.com