Physical Memory Leak by Terminated Processes
Question
Friday, September 23, 2011 2:25 PM | 1 vote
Hi,
I am facing an issue with Windows 2008 x64 Server (any edition). It frequenlty crashes due to lack of Physical Memory and when we check no other programs uses so much Physical Memory to create the issue.
After some investigation with RAMMAP utility, we noticed that Processes (any windows executable) which are terminated does not release some part of Physical Memory (4KB under Private Bytes and 16KB under Page table). So any process terminates, does not release 20K of Physical Memory and gradually, server crashes when complete Memory chews up.
We did some more investigation and noticed this issue Started after installation of one particular application on the server and issue stops after uninstalling this software. We had created a call with the Application Vendor and they confirmed that it is not an issue with their application, it is a Windows issue. Their view "This Application is not running as any Service and will be executed only when Someone starts the Application. As the issue exists, without executing the application it is not application problem. Also, not only their application exe, all windows executable leaves 20K of Physical Memory and as, Windows is responsible for Memory Management it is an issue with Operating System.
Would someone help me to identify why this OS misbehaviour is happening.
Thanks
All replies (19)
Friday, September 30, 2011 5:58 AM âś…Answered | 3 votes
Yes, you are correct. Exactly the same issue.
Problematic application is Netapp Single Mailbox Recovery.
I have identified the driver creating the issue. With the help of Procmon, noticed that it installs two drivers during SMBR installation. Drivers are Aksdf.sys and Hardlock.sys, both are drivers from Aladdin Knowledge Systems. This driver was loaded through CurrentControlset->Services, after stopping Aksdf.sys, issue stopped. This is a filter driver from Aladdin, used for USB Dongle licensing. I also tried to update the driver to the latest version from Aladdin on a test server, that also resolves the issue.
I am planning to uninstall this driver, as I am not using USB licensing key for the application.
I am interested to know what this driver was doing internally on the OS to create the issue. If you have any info to find this out, please share.
Thanks for the support.
Saturday, September 24, 2011 5:14 PM | 1 vote
You might try to scan system files to see if the application changed some files.
Run a cmd prompt as administrator and run "sfc /scannow" you can also add the /"verifyonly" if you dont what it to replace files
Process Explorer can also give a better overview of your memory http://technet.microsoft.com/nb-no/sysinternals/bb795535
/Olav
Monday, September 26, 2011 12:28 PM
Hi Olav,
Thanks for your response.
I tried with sfc /verifyonly and it did not show any issues (all files are OK).
Process explorer is not showing this part of unused (or leaking) Physical Memory. In Task Manager Physical Memory usage is 1.6GB, but sum of WorkingSet size of all processes is only 950MB.
Above image shows the Mem usage of Terminated processes. All these PIDs does not exist (not running on the machine), but still uses Memory.
Thanks
Thursday, September 29, 2011 9:08 AM | 2 votes
Any program will leave 20K of physical memory with or without the application is running, but the issue will not occur if remove the application. Is this correct?
If this is correct, then the application should cause something like a conflition, specifically it occurs on any edition of Windows 2008 x64 server as you mentioned. What kind of application it is? Please provide more information about it so we can check if any one have a similar issue.
Also I would like to confirm, if we reboot the server, without running the appliction, whether the same issue occurs if we start any program such as Notepad, Office program etc?
And if we disable any security program, backup program etc and reboot, without running the appliction, whether the issue still exists?
TechNet Subscriber Support in forum |If you have any feedback on our support, please contact tnmff@microsoft.com.
Wednesday, April 18, 2012 9:33 AM
Sorry to revive an old thread, but I have exactly the same problem but on Windows 7 Prof SP1 64bit and I do not have the Netapp Single Mailbox Recovery installed. Every terminated process leaves a 20k footprint in memory, slowly killing the system. Any ideas beyond the application you mentioned? Help would be really appreciated.
Wednesday, June 6, 2012 7:54 AM
Hi Ulchuchu,
Are you still facing the issue ?
Thanks,
Baiju
Friday, July 20, 2012 5:13 PM
I am seeing this issue as well. Windows 7 x64. I do not have the Netapp application installed. Is there a way to diagnose this to another problem application?
Friday, July 20, 2012 7:28 PM
Process Explorer gives a better overview of your memory usage http://technet.microsoft.com/nb-no/sysinternals/bb795535
Sunday, July 22, 2012 2:37 AM
Process Explorer does not list the processes. They are 'terminated' and only show in rammap. 4K Private and 16K of pagetable per terminated process. The exact symptoms as described earlier in this post. I used msconfig to disable every service and startup item and the behavior continues.
Wednesday, July 25, 2012 10:47 AM
This issue normally happens due to Zombie Processes. Some process does not destory the process handle to another process when it exits. That orphan process handles takes Memory and shown in the RAMMAP with 20KB Total size. Normal way to troubleshoot the issue is take a Memory Dump and analyse the dump in Windbg.
!VM command will show all the Zombie processes (will show as 0KB size). Then, open one particular Zombie process with !Process <processid>, it will show the Parent Process of that Zombie process. Most of the times, that parent process would be the culprit.
Thanks,
Baiju
Tuesday, July 31, 2012 8:07 PM
I got windbg loaded and a zombie process list dumped. There are plenty of processes listed at 0KB when running !VM. They each have different parent id's. Any other suggestions?
1d40 SearchFilterHo 0 ( 0 Kb)
1d08 wscript.exe 0 ( 0 Kb)
1ce4 SMSCliUI.exe 0 ( 0 Kb)
1cac SearchProtocol 0 ( 0 Kb)
1a04 susetsched.exe 0 ( 0 Kb)
19e8 TvsuCommandLau 0 ( 0 Kb)
19bc susetsched.exe 0 ( 0 Kb)
1918 AcWin7Hlpr.exe 0 ( 0 Kb)
18bc sppsvc.exe 0 ( 0 Kb)
1830 GoogleCrashHan 0 ( 0 Kb)
1828 GoogleCrashHan 0 ( 0 Kb)
1810 GoogleUpdate.e 0 ( 0 Kb)
17c4 GoogleCrashHan 0 ( 0 Kb)
1720 BtwLyncIntf.ex 0 ( 0 Kb)
16e0 dllhost.exe 0 ( 0 Kb)
16c4 SvcGuiHlpr.exe 0 ( 0 Kb)
165c BtwLyncIntf.ex 0 ( 0 Kb)
15a0 igfxsrvc.exe 0 ( 0 Kb)
1598 rundll32.exe 0 ( 0 Kb)
14e8 SearchFilterHo 0 ( 0 Kb)
14d4 GoogleCrashHan 0 ( 0 Kb)
14d0 igfxsrvc.exe 0 ( 0 Kb)
14c8 SearchProtocol 0 ( 0 Kb)
14bc mscorsvw.exe 0 ( 0 Kb)
1498 AcFnF5.exe 0 ( 0 Kb)
141c rundll32.exe 0 ( 0 Kb)
1408 raserver.exe 0 ( 0 Kb)
13f4 runonce.exe 0 ( 0 Kb)
13d0 igfxtray.exe 0 ( 0 Kb)
13cc GoogleUpdate.e 0 ( 0 Kb)
1380 SAIICpl.exe 0 ( 0 Kb)
1374 dllhost.exe 0 ( 0 Kb)
1344 AdobeARM.exe 0 ( 0 Kb)
1280 GoogleUpdate.e 0 ( 0 Kb)
1258 taskhost.exe 0 ( 0 Kb)
11fc SearchProtocol 0 ( 0 Kb)
11e0 AcTBenabler.ex 0 ( 0 Kb)
1194 WMIADAP.exe 0 ( 0 Kb)
1174 svchost.exe 0 ( 0 Kb)
10f4 dllhost.exe 0 ( 0 Kb)
10f0 taskeng.exe 0 ( 0 Kb)
10cc WmiPrvSE.exe 0 ( 0 Kb)
10b0 conhost.exe 0 ( 0 Kb)
10a4 cmd.exe 0 ( 0 Kb)
0fcc dllhost.exe 0 ( 0 Kb)
0fb8 dllhost.exe 0 ( 0 Kb)
0ee0 igfxsrvc.exe 0 ( 0 Kb)
0e54 drvinst.exe 0 ( 0 Kb)
0e3c svchost.exe 0 ( 0 Kb)
0df0 taskhost.exe 0 ( 0 Kb)
0d34 GoogleUpdate.e 0 ( 0 Kb)
0d2c mscorsvw.exe 0 ( 0 Kb)
0cb0 rundll32.exe 0 ( 0 Kb)
0c64 AtBroker.exe 0 ( 0 Kb)
0c5c SearchProtocol 0 ( 0 Kb)
0c34 chrome.exe 0 ( 0 Kb)
0b28 SearchIndexer. 0 ( 0 Kb)
0a54 userinit.exe 0 ( 0 Kb)
0968 BdeUISrv.exe 0 ( 0 Kb)
0954 iWrap.exe 0 ( 0 Kb)
0940 dllhost.exe 0 ( 0 Kb)
093c conhost.exe 0 ( 0 Kb)
0934 cacls.exe 0 ( 0 Kb)
08c8 iWrap.exe 0 ( 0 Kb)
0838 tpnumlk.exe 0 ( 0 Kb)
06a4 xkbcomp.exe 0 ( 0 Kb)
039c reader_sl.exe 0 ( 0 Kb)
0358 HyperW7Svc64.e 0 ( 0 Kb)
0268 smss.exe 0 ( 0 Kb)
01e4 LogonUI.exe 0 ( 0 Kb)
01e0 smss.exe 0 ( 0 Kb)
01a8 dllhost.exe 0 ( 0 Kb)
018c autochk.exe 0 ( 0 Kb)
lkd> !process 018c
Searching for Process with Cid == 18c
Cid handle table at fffff8a000004de0 with 1840 entries in use
PROCESS fffffa8009ceb080
SessionId: none Cid: 018c Peb: 7fffffdf000 ParentCid: 0180
DirBase: 1d2bbe000 ObjectTable: 00000000 HandleCount: 0.
Image: autochk.exe
VadRoot 0000000000000000 Vads 0 Clone 0 Private 1. Modified 0. Locked 0.
DeviceMap fffff8a000008c10
Token fffff8a00034ebd0
ElapsedTime 00:09:19.657
UserTime 00:00:00.000
KernelTime 00:00:00.000
QuotaPoolUsage[PagedPool] 0
QuotaPoolUsage[NonPagedPool] 0
Working Set Sizes (now,min,max) (5, 50, 345) (20KB, 200KB, 1380KB)
PeakWorkingSetSize 223
VirtualSize 4 Mb
PeakVirtualSize 4 Mb
PageFaultCount 220
MemoryPriority BACKGROUND
BasePriority 8
CommitCharge 0
No active threads
lkd> !process 01a8
Searching for Process with Cid == 1a8
Cid handle table at fffff8a000004de0 with 1815 entries in use
PROCESS fffffa800e88e080
SessionId: 1 Cid: 01a8 Peb: 7fffffd7000 ParentCid: 0320
DirBase: 157810000 ObjectTable: 00000000 HandleCount: 0.
Image: dllhost.exe
VadRoot 0000000000000000 Vads 0 Clone 0 Private 1. Modified 0. Locked 0.
DeviceMap fffff8a0023757b0
Token fffff8a00d2e7610
ElapsedTime 00:00:53.943
UserTime 00:00:00.000
KernelTime 00:00:00.015
QuotaPoolUsage[PagedPool] 0
QuotaPoolUsage[NonPagedPool] 0
Working Set Sizes (now,min,max) (5, 50, 345) (20KB, 200KB, 1380KB)
PeakWorkingSetSize 1565
VirtualSize 29 Mb
PeakVirtualSize 66 Mb
PageFaultCount 1733
MemoryPriority BACKGROUND
BasePriority 8
CommitCharge 0
No active threads
Friday, August 10, 2012 7:17 AM
Whether the Parent Process details available in the dump (eg ParentCid: 0180 or 0320) or already terminated ?
Also, what are the Filter Drivers installed on your machine ? (with command Fltmc). Some incomatible filter drivers also cause this issue.
Friday, August 10, 2012 9:50 PM
The parent process are terminated (and still show up themselves) with eventual parent being smss.exe Even tried terminating winlogon.exe and it shows up as zombie as well.
Filter Name Num Instances Altitude Frame
MpFilter 5 328000 0
luafv 1 135000 0
FileInfo 5 45000 0
All three of these look to be valid signed microsoft drivers.
Thursday, August 16, 2012 6:11 PM | 2 votes
There is one way (not a straight forward way) to isolate the issue. If you can recreate the same issue on a Test Workstation with the same softwares. Try to uninstall the Non-MS Applications one by one by verifying the issue status with RAMMap after each software uninstall.
- Uninstall application
- Open and Close any program (eg. cmd.exe), note the PID before closing the program
- Verify with RAMMap, if the PID is showing as an Orphan process
This is the only method that comes to my mind to isolate the issue.
Tuesday, November 20, 2012 7:36 PM
Hi Kal,
I have very similar symptoms on my machine. Did you make any progress since you posted this?
Thanks,
Ron
Sample output from !process 0 7:
PROCESS fffffa8012be8600
SessionId: 1 Cid: 35e74 Peb: 7fffffdc000 ParentCid: 1fcc
DirBase: 3d57f000 ObjectTable: 00000000 HandleCount: 0.
Image: svn.exe
VadRoot 0000000000000000 Vads 0 Clone 0 Private 1. Modified 1. Locked 0.
DeviceMap fffff8a0039d6dd0
Token fffff8a014e9b060
ElapsedTime <Invalid>
UserTime 00:00:00.015
KernelTime 00:00:00.015
QuotaPoolUsage[PagedPool] 0
QuotaPoolUsage[NonPagedPool] 0
Working Set Sizes (now,min,max) (5, 50, 345) (20KB, 200KB, 1380KB)
PeakWorkingSetSize 1363
VirtualSize 44 Mb
PeakVirtualSize 64 Mb
PageFaultCount 1385
MemoryPriority BACKGROUND
BasePriority 8
CommitCharge 0
Job fffffa8004301e30
No active threads
Thursday, November 22, 2012 3:25 PM | 1 vote
I followed the steps suggested by Baiju:
- Uninstall a suspicious application.
- Restart the computer.
- Open and close notepad.exe.
- Verify with RAMMAP if notepad.exe still shows up on the Processes tab.
Using this technique and some informed guesswork I discovered the offending application: Lenovo RapidBoot Shield.
Tuesday, February 12, 2013 4:30 AM
Interestingly enough I had the same application installed, however I ended up re-imaging the system to fix it.
Saturday, February 1, 2014 9:48 PM | 2 votes
I had the same problem with Windows 7 64-bit. The issue was solved after removing the Aladdin filter "aksdf".
C:\Fltmc unload aksdf
Tuesday, December 9, 2014 10:11 AM
Lenovo RapidBoot Shield was the culprit for me too.
Uninstalling that and rebooting solved my problem: http://superuser.com/a/850346/79763