Microsoft Windows Dynamic Cache Service
Excessive cached read I/O is a growing problem. For over one year we have been working on this problem with several companies. You can read more about it in the original blog post:
https://blogs.msdn.com/ntdebugging/archive/2007/11/27/too-much-cache.aspx
On 32 bit systems, the kernel could address at most 2GB of virtual memory. This address range is shared and divided up for the many resources that the system needs; one of which is the System File Cache's working set. On 32 bit systems the theoretical limit is almost 1GB for the cache’s working set; however, when a page is removed from the working set it will end up on the standby page list. Therefore the system can cache more than the 1 GB limit if there is available memory. The working set; however, is just limited to what can be allocated within the Kernel's 2GB virtual address range. Since most modern systems have more than 1 GB of physical RAM, the System File Cache's working set's size on a 32 bit system typically isn't a problem.
With 64 bit systems, the kernel virtual address space is very large and is typically larger than physical RAM on most systems. On these systems the System File Cache's working set can be very large and is typically about equal to the size of physical RAM. If applications or file sharing performs a lot of sustained cached read I/O, the System File Cache's working set can grow to take over all of physical RAM. If this happens, then process working sets are paged out and everyone starts fighting for physical pages and performance suffers.
The only way to mitigate this problem is to use the provided APIs of GetSystemFileCacheSize() and SetSystemFileCacheSize(). The previous blog post "Too Much Cache" contains sample code and a compiled utility that can be used to manually set the System File Cache's working set size.
The provided APIs, while offering one mitigation strategy, has a couple of limitations:
1) There is no conflict resolution between multiple applications. If you have two applications trying to set the System File Cache's working set size, the last one to call SetSystemFileCacheSize() will win. There is no centralized control of the System File Cache's working set size.
2) There is no guidance on what to set the System File Cache's working set size to. There is no one size fits all solution. A high cache working set size is good for file servers, but bad for large memory application and a low working set size could hurt everyone's I/O performance. It is essentially up to 3rd party developers or IT administrators to determine what is best for their server and often times, the limits are determined by a best guesstimate backed by some testing.
We fully understand that while we provide one way to mitigate this problem, the solution is not ideal. We spent a considerable amount of time reviewing and testing other options. The problem is that there are so many varied scenarios on how users and applications rely on the System File Cache. Some strategies worked well for the majority of usage scenarios, but ended up negatively impacting others. We could not release any code change that would knowingly hurt several applications.
We also investigated changing some memory manager architecture and algorithms to address these issues with a more elegant solution; however the necessary code changes are too extensive. We are experimenting with these changes in Windows 7 and there is no way that we could back port them to the current operating systems. If we did, we would be changing the underlying infrastructure that everyone has been accustomed to. Such a change would require stress tests of all applications that run on Windows. The test matrix and the chance of regression are far too large.
So that brings us back to the only provided solution - use the provided APIs. While this isn't an ideal solution, it does work, but with the limitations mentioned above. In order to help address these limitations, I've updated the SetCache utility to the Microsoft Windows Dynamic Cache Service. While this service does not completely address the limitations above, it does provide some additional relief.
The Microsoft Windows Dynamic Cache Service uses the provided APIs and centralizes the management of the System File Cache's working set size. With this service, you can define a list of processes that you want to prioritize over the System File Cache by monitoring the working set sizes of your defined processes and back off the System File Cache's working set size accordingly. It is always running in the background monitoring and dynamically adjusting the System File Cache's working set size. The service provides you with many options such as adding additional slack space for each process' working set or to back off during a low memory event.
Please note that this service is experimental and includes sample source code and a compiled binary. Anyone is free to re-use this code in their own solution. Please note that you may experience some performance side effects while using this service as it cannot possibly address all usage scenarios. There may be some edge usage scenarios that are negatively impacted. The service only attempts to improve the situation given the current limitations. Please report any bugs or observations here to this blog post. While we may not be able to fix every usage problem, we will try to offer a best effort support.
Side Effects may include:
Cache page churn - If the System File Cache's working set is too low and there is sustained cached read I/O, the memory manager may not be able to properly age pages. When forced to remove some pages in order to make room for new cache pages, the memory manager may inadvertently remove the wrong pages. This could result in cached page churn and decreased disk performance for all applications.
Version 1.0.0 - Initial Release
https://www.microsoft.com/downloads/details.aspx?FamilyID=e24ade0a-5efe-43c8-b9c3-5d0ecb2f39af
NOTE: The memory management algorithms in Windows 7 and Windows Server 2008 R2 operating systems were updated to address many file caching problems found in previous versions of Windows. There are only certain unique situations when you need to implement the Dynamic Cache service on computers that are running Windows 7 or Windows Server 2008 R2. For more information on how to determine if you are experiencing this issue and how to resolve it, please see the More Information section of Microsoft Knowledge Base article 976618 - You experience performance issues in applications and services when the system file cache consumes most of the physical RAM.
Comments
Anonymous
March 09, 2009
I downloaded DynCache for x64 systems. The ReadMe file indicates "3) Import the DynCache.reg registry file. This registry file contains default settings that you will probably want to modify. " Contents of file reproduced below. Can anyone offer clarification of what I can modify? (Server in questionis x64, 16GB RAM. Sql 2005 x64) . Thanks, Barkingdog Windows Registry Editor Version 5.00 [HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesDynCacheParameters] "BackOffMBytesOnLowMemory"=dword:000001f4 "MaxSystemCacheMBytes"=dword:00000000 "MinSystemCacheMBytes"=dword:00000064 (MIN > MAX ???) "SampleIntervalSecs"=dword:0000003c "CacheUpdateThresholdMBytes"=dword:00000064 [HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesDynCacheParametersNotepad] "AdditionalBackoffMBytes"=dword:0000000a [HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesDynCacheParameterssqlservr] "AdditionalBackOffCounter"="\SQLServer:Memory Manager\Total Server Memory (KB)" [HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesDynCacheParametersSystem] [Great question! The “Configuration” section in the readme file explains what each of these registry keys control. The included registry file is just a base template or starting point. In the included example, the minimum System Cache Mbytes is set to 100 Mbytes. The maximum is set to 0, which means limit the system file cache to 90% of physical RAM (default).]Anonymous
March 12, 2009
Why are there folders for i386 and IA64 in the released package? [On Windows Server 2008 x86 (32 bit) the System File Cache doesn’t have the ~1 GB theoretical limit like it did with Windows Server 2003 x86 (32 bit). The System File Cache can grow to consume nearly all of the kernel address space (2GB). The i386 (32 bit) folder is there for managing the size of the System File Cache for 32 bit servers. The IA64 folder is there for servers running on Intel IA-64 processors (not to be confused with AMD’s x64 processor that Intel has licensed).]Anonymous
March 22, 2009
I'm experiencing system slowness due to slow writes to a SSD. I’m aware of the risks but... ¿is it possible to increase the system write cache and/or reduce the write pace of dirty pages of the cache to the disk? ¿Which is the role of the “Writewatch” on HKLMSYSTEMCurrentControlSetControlSession ManagerMemory Management (no info on Technet about it)? I have been reading your posts about the memory management but I can’t see a way to preserve part of the cache for writes (I can set the maximum amount of memory for cache with your excellent Microsoft Windows Dynamic Cache Service and the amount of dirty pages with SystemCacheDirtyPageThreshold) I’m using the LargeSystemCache configuration and the disk CacheIsPowerProtected and UserWriteCacheSetting options are set. All I want is to cache more writes into memory. It will be fantastic to be able to cache “write through” requests also (more risks but I can handle them) Thanks! [Sounds like you need Windows 7. Windows (before Windows 7) was designed to support conventional hard drives. While SSDs present themselves as conventional hard drives, there are some caveats as you are well aware of. With Windows 7 we added enhancements to improve the usage of SSDs. We will disable defrag, make larger write clusters, reduce read/write cycles, etc. These enhancements should extend SSD lifespan and improve SSD I/O performance over previous versions of Windows. The 'WriteWatch' registry key was used to watch modification of pages in physical memory. After Windows 2000, this registry key is no longer used. For current versions of Windows you should always disable disk defragmenters on SSD drives. If you have enough physical memory, you could disable paging of the Executive with the 'DisablePagingExecutive' registry key. You could even turn off the paging file to reduce write and read I/O to your drive. Please be aware, that if you turn off the paging file, you will not only be unable to collect a memory dump file, but the system could run out of available memory. I can't recommend anything that would prevent "write through" writes. If the system ever hangs, crashes or powers off unexpectedly, then you could lose critical writes (like to the registry). The system may end up being unbootable.]Anonymous
April 01, 2009
Thanks for your answer! Most of your recommendations are generic for most “old SSD” that have too much problems shuttering due to writes. I have not enough memory to disable the page file (only 4 GB) but I have configured the page file on another mechanical disk and I have disabled defrag and paging executive. I know that preventing "write through" writes can be a bad idea in general and the system may end unbootable but it is like having your SO on a RAID 0. Enabling "write caching" and "advanced performance" check on disks without having a real backup power supply is also "risky" but it is not so bad if you know what you are doing. In the end you risk the integrity of the system to gain performance... Like all of life, is about tradeoffs. I don't want you recommend that as a "best practice" but it is interesting to know how it is possible to achieve this behavior or if it is impossible on Windows. For example it can be very useful for demo virtual machines that need to run as fast as possible and you will disregard the changes after the demo anyway.Anonymous
April 08, 2009
The folder has amd64, i386, and ia64. Where is the folder just for 64-bit, NOT ia64 and amd64? Where is the Windows Dynamic Cache service for 64-bit server that is NOT IA or AMD? thanks [All 64 bit installs should use the amd64 bit folder. Unless you’re running an Itanium machine which uses the ia64 folder, you should use the amd64 folder.]Anonymous
April 20, 2009
You mentioned above: "So that brings us back to the only provided solution - use the provided APIs." This solution is only for 64-bit machines right? Based on your explanations above, it sounds like a 32-bit OS wouldn't suffer from this memory issue. Meaning someone could 'downgrade' their OS to 32-bit and basically escape this issue altogether right? LC [Hello LC. Unfortunately 32 bit Vista/Server08 systems could experience worse problems. Prior to Vista, on 32 bit system, kernel resources were limited to certain virtual address ranges. This ended up restricting the system file cache’s working set size to a theoretical limit of about 1 GB. Since most systems have more than 1 GB of physical RAM you were generally not affected by this problem Starting with Vista, the Memory Manager can allocate kernel resources anywhere in the kernel virtual address space. This allows critical resources like paged and non-paged pool to grow larger than they previously could. Unfortunately, this change also allows the system file cache’s working set to grow and consume all of kernel virtual memory. Since the kernel virtual address space is limited to 2GB of virtual memory, the system file cache doesn’t have to consume all of physical RAM. Once it consumes all of the kernel virtual address space, allocations in critical resources like paged and non-paged pool may start failing. If these critical allocations start failing, you’ll get all kinds of generic errors from not enough system resources to access denied to application and system crashes. So for 32 bit Vista/Server 08 systems you have less virtual address range to play in. You can consume it quicker and if you do, you may end up with worse problems than a slow computer.]Anonymous
June 26, 2009
The comment has been removedAnonymous
September 29, 2009
I'm trying to mimic the setcache.exe behavior. If I set MaxSystemCacheMBytes to 4096 and MinSystemCacheMBytes would be 0 (the default which is 100MB), start the dynamic cache service and then run setcache.exe it appears that the settings have not been activated. Is this expected behavior or am I missing something? [What operating system and processor architecture (x86 or x64) are you using? If you are on x64, make sure that you are using the 64 bit version of DynCache.exe. The minimum cache size is 100 Mbytes (unless you change the code and recompile). This is just a safety guard and I highly recommend that you do not set the minimum to below 100 Mbytes. If you are on x86, then use a number less than 4096 as the 32 bit API has this as an upper limit. Also make sure you have over 4096 Mbytes available for the cache to potentially grow into. You can use the debug build of DynCache.exe along with Debug View to collect a log of what the service doing on start up.]Anonymous
September 01, 2010
i am getting this error in the applications event log when I load the service. Shoud i concerned with this? Windows cannot open the 32-bit extensible counter DLL ASP.NET in a 64-bit environment. Contact the file vendor to obtain a 64-bit version. Alternatively, you can open the 32-bit extensible counter DLL by using the 32-bit version of Performance Monitor. To use this tool, open the Windows folder, open the Syswow64 folder, and then start Perfmon.exe [The service uses performance counters if you enable active monitoring. It looks like you have a 32 bit only perf counter. You can ignore this error, or you can disable ASP.NET counters if you aren't using them for anything else.]Anonymous
January 12, 2012
The comment has been removedAnonymous
June 20, 2012
All our X64 file servers seem to have this issue to a greater or lesser extent. It seems a total pain to make those of use who haev upgraded to the latest version jump through the hoops of opening a support to get the fix. [The memory management algorithms in Windows 7 and Windows Server
2008 R2 operating systems were updated to address many file caching problems
found in previous versions of Windows. There are only certain unique situations
when you need to implement this service on computers that are running Windows 7
or Windows Server 2008 R2. We would like to know more about those
situations where the Dynamic Cache Service is required so we can identify areas for additional improvement. For example, the fix in KB 2564236 I/O throughput
is low when large files are read sequentially in Windows 7 or in Windows Server
2008 R2 might be more appropriate than running the Dynamic Cache Service.]Anonymous
January 27, 2014
it's a shame, it should be provided as a hotfix, but when I call the Microsoft Support Service they want me to pay a case, I can't believe that Microsoft sells the R2 patch to fix the issue [The current publicly available version of dyncache will run on R2. We maintained the same download link, so you can get it from the link in this article.]Anonymous
May 25, 2014
The comment has been removedAnonymous
June 11, 2014
I'm running this service in Windows Server 2008 SP2. I'm using the AMD64, debug version. However, running Debug View does not show anything. I've tried running Debug View as administrator as well - still nothing. The service starts alright, but doesn't seem to do anything. The memory consumed by the system cache still exceeds the values set in the registry. [By default debug output is not displayed. You need to set a debug print filter mask. Under HKLMSYSTEMCurrentControlSetControlSession ManagerDebug Print Filter create a value named DEFAULT and set it to 0xF and reboot. After this dbgview should work.]Anonymous
October 17, 2014
I am running a Windows Server 2008 R2 machine and the Dynamic Cache is started. But the Standby memory still goes right pass the set limits I have configure in the DynCache registry. Any ideas? [It is normal and expected for memory to be on the standby list. The system can quickly transition memory from standby back into a working set or to the free list, so it keeps most of the memory on standby. This article deals with a large amount of cache data on the Active list.]