# A script to reset the SharePoint 2010 Disk Based Cache for BLOBs
# Example: .\Reset-Blob-Cache.ps1 http://MySiteCollection

# Loads the SharePoint 2010 PowerShell extensions so we can easily
# get a reference to our dlls.
try
{
    if(((Get-PSSnapin | foreach {$_.Name}) -contains "Microsoft.SharePoint.PowerShell") -eq $false)
    {
      Add-PSSnapIn Microsoft.SharePoint.PowerShell -ErrorAction Stop
    }
}
catch
{
    Write-Host -ForegroundColor "red" "This script only works on machines with SharePoint 2010 installed.`n"
    Exit
}

# Use Reflection to get a reference to our dlls.
try
{
    $spAssembly = [System.Reflection.Assembly]::GetAssembly([Microsoft.SharePoint.SPSite])
    [System.Reflection.Assembly]::Load($spAssembly) | Out-Null
}
catch
{
    Write-Host -ForegroundColor "red" "Could not locate the SharePoint Assemblies.`n"
    Exit
}

# You must provide a site upon which to execute the cache reset.
if ($args.length -ne 1)
{
    Write-Host "Expected 1 arguments:"
    Write-Host "<site_url>"
    exit
}

$site_url = $args[0]

try
{
    $site = Get-SPSite -Identity $site_url -ErrorAction Stop
}
catch
{
    Write-Host -ForeGround 'Red' "$_"
    Write-Host -ForeGround 'Red' "Please correct the arguments and try again.`n`n"
    Exit
}

$warning = "This will clear the blob cache for all sites in this application, on all WFEs in the farm. `nPress 'y' to continue:"

Write-Host -ForeGround 'Yellow' -noNewLine $warning

$key = $host.UI.RawUI.ReadKey("NoEcho,IncludeKeyUp")

if ($key.Character -ine "y")
{
    Write-Host "`nYou chose not to flush the cache."
    exit
}

Write-Host "`nFlushing..."
[Microsoft.SharePoint.Publishing.PublishingCache]::FlushBlobCache($site.WebApplication)

Write-Host "Flushed Cache for:" $site.WebApplication.Name

Simple enough to fix if I can just figure out how to force it to run as a 32bit app. I found the answer here. In a nutshell you can use the CoreFlags.exe utility to force the app to run as a 32-bit application.

The main problem was that it had pretty severe video performance issues. It is also worth noting that this is one of several of these machines we have scattered around and nobody else seemed to be having these problems.

Here is a quick summary of the weird behaviors I was seeing:

1. One of the three monitors attached to this machine would go black and then repaint every time I mapped or unmapped a network drive. No kidding.
2. Unlocking the workstation, even immediately after locking it, resulted in a 50-120 second delay while it repainted my screen and generally sat there looking lame.
3. My VX-600 webcam which I use to chat with my 3 year old son when I am at going be at work late at night was totally unusable. Any time I started the camera the whole machine slowed to a complete crawl.
4. Scrolling anything in windows in the applications I use like IE, Word, Excel, WinDbg, and Visual Studio was jerky and very slow.
5. Dragging a window from one monitor to the others was jerky and sometimes would just stall the machine for 10-20 seconds.
6. Enabling the Aero desktop made the whole thing unusable.

I searched everywhere for problems with Windows Server 2008 and performance problems on the physical console. Why the physical console? Because I can use Remote Desktop to connect to this machine it runs just fine.

Based on all of these symptoms I figured it was the fault of the stock video card I had in the machine which uses a cheap NVidia Quadro GPU so I went out and got a new video card making for a matched set of GEForce 8600 SE cards in this machine. This configuration now matches my neighbors machine which doesn’t have these problems. In the end this change helped somewhat but performance was still pretty poor. I tried updating my video drivers, no help there either. Time to try some other stuff.

The next thing I did was switch my processor scheduling setting from “Background services” to “Programs”. This seemed to produce a slight improvement but not enough to make the machine usable from the console.

Next I shut down my Hyper-V VMs. I considered that perhaps they were chewing up resources that my machine desperately needed, despite the fact that I had 7GB-10GB of RAM free and CPU utilization was averaging 5%. That seemed to have no effect at all but since they just made it take longer each time I rebooted and I wasn’t using them right now I left them turned off. 

Since it wasn’t any of the obvious stuff, next I went to the trusty ole Reliability and Performance Monitor. I created a new counter set to track the Memory, Processor, Processes and Threads objects. Surely if I have a performance problem on the physical console it will show up in one of those.

I started the log, locked my machine, waited 2 minutes and then unlocked it. It took about 55 seconds to repaint all the screens and get control of the apps. When I cracked open the log I could see that there were a few of my helper apps like Communicator, SnagIt and UltraMon taking a pretty big chunk of CPU for most of that unlock time.

I used MSConfig.exe to disable all of the autorun things I had loaded and rebooted. Performance was better. At this point I assumed that the performance I was seeing was as good as it was going to get so I set about determining which of autorun apps was killing what little performance I could get out of this machine.

I proceeded to enable the apps one at a time, reboot, and test the performance of locking and unlocking the machine. In the end I removed the webcam software as it seemed to be the only thing left when everything else was loaded and it still ran better than it had before.

After using the machine for an hour or so I locked it and came back ten minutes later. When I unlocked the machine it took more than a minute for me to get control of the desktop and even then it was pretty sluggish. Back to square one.

During all of this I have been talking with my peers to see what other ideas they might have. One of them suggested that maybe I had a desktop heap problem. Since I have run out of desktop heap in the past and I know what that looks like I didn’t really think that was it but I am desperate now so I doubled the amount of desktop heap for the interactive session to 40MB. No help there either, same behavior.

At this point I had pretty much resigned myself to using this machine in it’s current state until I have time to pave it and reinstall everything. A few minutes later one of my peers who has the exact same configuration was showing me how his machine performed. It was awesome! Unlocking his machine was pretty much instantaneous! What in the @#$%&*%$# is wrong with mine?

It was then that he saved me by commenting that at one point he had installed Hyper-V and had some performance problems, not as bad as mine, but he had some, and he fixed them by uninstalling Hyper-V…

Now I’m thinking “Hyper-V can’t be my problem, I don’t even have any VMs running”…and then it dawned on me. Hyper-V uses a HyperVisor to wedge in between the OS and the hardware. What could possibly go wrong there?

Needless to say, 5 minutes later, after I had uninstalled Hyper-V and rebooted, my machine is as blistering fast as my co-worker’s!

Now that I knew what the solution was was I did some quick searches for problems with Hyper-V video performance. I immediately found KB 961661. For reference here is the cause section of the KB article:

This issue occurs when a device driver or other kernel mode component makes frequent memory allocations by using the PAGE_WRITECOMBINE protection flag set while the hypervisor is running. When the kernel memory manager allocates memory by using the WRITECOMBINE attribute, the kernel memory manager must flush the Translation Lookaside Buffer (TLB) and the cache for the specific page. However, when the Hyper-V role is enabled, the TLB is virtualized by the hypervisor. Therefore, every TLB flush sends an intercept into the hypervisor. This intercept instructs the hypervisor to flush the virtual TLB. This is an expensive operation that introduces a fixed overhead cost to virtualization. Usually, this is an infrequent event in supported virtualization scenarios. However, some video graphics drivers may cause this operation to occur very frequently during certain operations. This significantly magnifies the overhead in the hypervisor.

This is why I say, when you know the solution the problem is a lot easier to find.

I'll try to add a little clarity from the perspective of someone who is often put in the position of telling customers that they have violated the rules laid out by this article. Before I begin let me state for the record that these statements do not establish the official position of Microsoft Corporation. Recommendations and advice are those of the author or people and organizations the author trusts. The contents are provided "AS IS" with no warranties and confer no rights.

Lets take a look at a couple of key points of this article and see if we can make it a little clearer. When the article says:

The products that are listed in the "Applies to" section were tested by using the existing structure and were approved for release based on that structure. Unless Microsoft protocol documentation is followed precisely, Microsoft cannot reliably predict the effect to the typical operation of these products when parties other than Microsoft support change the database or run stored procedures. Parties other than Microsoft support would include, but not be limited to, changes that are made by customers, by third-party vendors, or by consultants.

I read this to say, "When developing custom solutions using our protocol documentation, unless you do things exactly as we do them there is a very, very, good chance that bad things will happen. We may make exceptions for specific updates to specific datasets at specific times for our support personnel. Support personnel are those persons employed by the Microsoft Commercial Office Support Systems group with the following titles: Support Engineer, Support Escalation Engineer, Escalation Engineer. If you do not work in the Microsoft Commercial Office Support Systems group AND posses one of the aforementioned titles then you are not permitted to directly modify our databases."

Having said all of that I can tell you from lots of experience that we in the SharePoint support team very, very, rarely make direct modifications to the database. Even when it does happen it is only when the problem was introduced by the product itself and after all other means of correcting the problem have been exhausted.

The article goes on to describe some, but not all, of the things that are not permitted:

Examples of such database changes include, but are not limited to, the following:

• Adding database triggers
• Adding new indexes or changing existing indexes within tables
• Adding, changing, or deleting any primary or foreign key relationships
• Changing or deleting existing stored procedures
• Adding new stored procedures
• Adding, changing, or deleting any data in any table of any of the databases for the products that are listed in the "Applies to" section unless Microsoft protocol documentation is followed exactly
• Adding, changing, or deleting any columns in any table of any of the databases for the products that are listed in the "Applies to" section
• Making any modification to the database schema
• Adding tables to any of the databases for the products that are listed in the "Applies to" section
• Changing the database collation

I think this list is pretty clear and while we specifically state that this is not an exhaustive list I'm pretty hard pressed to come up with something I might want to do to my database that is not covered by that list. In short, Do not touch the databases for the products associated with this article!

The article goes on to describe in pretty solid detail what happens if you decide to violate these prohibitions so I will not bother with interpreting them here.

Unfortunately this presented a new set of problems because while I knew  where the code was I wanted to debug I still didn't have a good way to catch it when it executed. A quick bit of web searching turned up this blog post that briefly mentioned the "debugger;" instruction. Not being a JavaScript guy I was unaware of this command. It's simply invokes the debugger when it executes. How very handy :)

I then edited my .js file in the \Layouts\1033 folder and added the line "debugger;" at the beginning of the function I wanted to work on. I then cleared the IE temporary internet files folder so I would easily get the new version of the .js file. I went through the steps to reproduce the problem and it opened Windows Script Debugger. This was not what I wanted. I wanted Visual Studio 2008 so I could use all the fancy features like hovering over variables and such.

I speculated that the problem might be that when I installed the script debugger it stole the JIT settings from VS2008 so I uninstalled it and tried again. No joy. OK, it may have stolen them and not put them back when it was removed.

It occurred to me that it may be that VS2008 is just not configured for JIT debugging of scripts. I checked my VS2008 debugging setting in the UI and found that I didn't have the Script setting enabled so I turned off the other two and enabled it:

A quick test verified that this did not fix my problem.

At this point one of my EE cronies (Rob Anderson) mentioned that he gets the JIT debugger popup when he does script debugging, I don't. This reminds me that historically the AEDebug key controlled this behavior. We check his registry settings and compare them to mine and discover that the likely reason is that I am missing the "Auto" key and value. Adding that fixed my problem and I am now happily JIT debugging javascript in Visual Studio 2008!

Here is the proper settings for this to work on an x86 machine:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug
"Auto"="1"
"Debugger"="\"D:\\WINDOWS\\system32\\vsjitdebugger.exe\" -p %ld -e %ld"
"UserDebuggerHotKey"=dword:00000000

In my case these settings needed to be transposed to the x64 registry location:

HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Windows NT\CurrentVersion\AeDebug 

Other than the location, the setting are the same. ]]> https://docs.microsoft.com/archive/blogs/steveshe/how-do-i-tune-the-moss-object-cache-for-performance-and-economy Thu, 12 Mar 2009 15:36:00 GMT https://blogs.msdn.microsoft.com/steveshe/2009/03/12/how-do-i-tune-the-moss-object-cache-for-performance-and-economy/ Tuning the size of the MOSS Object Cache is done via the Max Cache Size settings on the Site Settings > Object Cache Settings page. It is important to recognize that this maximum cache size setting is a limit and not a static value. For example; Just because the maximum cache size setting for the Object Cache is configured to 100MB does not mean that it will consume 100MB of memory at startup. It simply means that if the cache were to exceed 100MB it will be compacted to reduce its memory consumption to a level below that maximum value.

For purposes of this discussion we will categorize cache compaction rates into three categories, unacceptable, acceptable and optimal.   A cache compaction rate of more than 6 per hour should be considered unacceptable. A rate of between 2 and 6 compactions per hour is acceptable and between 0 and 1 cache compactions per hour is optimal. You should only target the optimal level of cache compactions if you have sufficient amount physical memory installed on your servers to achieve this goal and still have sufficient free physical memory remaining on the server to support other critical system operations.

You should regularly monitor for cache flushes because they are extremely expensive in terms of cache performance impact. A cache flush results in the ejection of all cache contents. Cache flushes are triggered by the creation, deletion or moving of a web. They can be monitored via the "SharePoint Publishing Cache/Total number of cache flushes" counter in Performance Monitor.  If you are seeing cache flushes throughout the production day you should reconsider how you manage webs during those hours.

Based on these considerations we have developed the following guidance for how to tune the Object Cache size for acceptable performance using the "SharePoint Publishing Cache / Total number of cache compactions" performance counter.

We feel this level of cache performance will meet an economical customers performance needs without unduly sacrificing their limited memory resources. The steps to achieve this are fairly straightforward and must be applied in an iterative fashion until the desired level of performance is achieved. The recommended steps are:

1.       Start with the default cache settings of 100MB on the site collection.

2.       Capture at least 8 hours worth of performance data from the WFEs while the system is under a typical load using the "SharePoint Publishing Cache/Total number of cache compactions" counter. Since we are interested in tracking compactions per hour it is acceptable to  capture this data at 1 minute intervals.

3.       After analysis of the data, if we are exceeding the threshold for acceptable cache compactions we will need to add an additional 50MB to the Maximum Cache Size value and run the test again.

4.       We should continue this process until such time as we have achieved an acceptable cache compaction rate.

SharePoint 2007 and WSS 3.0 Dispose Patterns by Example

While we have quite a few customers already on 64-bit servers I have not seen a lot of problems that would be addressed by these settings but I have seen a couple. This tells me that there is still value in configuring these settings on 64-bit servers. Unfortunately, I don’t have any solid data to make recommendations on what those settings should be.

That means the best I can offer is some thought on how I would figure it out. I would start by enabling a scheduled recycle each day that happens about 30 minutes before my first users start work. This makes good sense as I consider a daily recycle to be good housekeeping. As mentioned in a previous post, recycling the worker process cleans out fragmentation and results in a more orderly workspace.

Next, I would get Performance Monitor data for my servers for a period of two weeks. These two weeks would ideally contain a few days of my highest user loads. It is also important that during these two weeks that I’m not already experiencing problems like crashes or memory allocation related errors in the ULS or Application Event Logs.

While I can’t tell you exactly what days are in your business are the most intensive I can give an example to guide you. Many companies function on a monthly cycle where their peak loads occur at the end of each month. In this type of business I would gather my data for the last week of the current month and the first week on the next month. This would probably give me a solid representation of my heaviest loads and my lightest loads.

In my scenario I’m going to guess 30% growth over the max I see in my Performance Monitor data is about the most to I want to allow before I recycle my worker process. I will be using that number in the calculations below.

Once I have my data I would then determine the maximum size that the worker process ever reached over that two week period in terms of Process/Private Bytes and Process/Virtual Bytes. I would also gather the Memory/Available MBytes value from those same times where I saw the maximum values.

Once I had these values I would first determine if I have enough free physical memory to enable overlapped recycling. To determine this I would compare the maximum Private Bytes value to the available MBytes value and if Available MBytes was not at least equal to ((Private Bytes value * 1.3)+300MB) I would not enable overlapped recycling until I had added more physical memory to the system.

Assuming I have enough free physical memory I would then use those maximum observed values plus my 30% growth factor to come up with the settings for the “Maximum Memory Used” and “Maximum Virtual Bytes” recycle settings. Here is an example:

Using these numbers I would configure the IIS Application Pool using the following settings:

UPDATE 6/19/2008 

I thought it was worth coming back to this post and updating it. As of this date we have not seen any significant problem in this area with customers running on 64-bit platforms. Most of the customers I am aware of on 64-bit platforms are not using memory based recycling settings. I do want to say that I would still reccomend a scheduled nightly recycle because it will help reduce any possiblity of problems caused by fragmentation.

1.     Click Start, click Run, and then type cmd at the command prompt.

2.     Change to the directory where Adsutil is located. The following is the default directory location: %SYSTEMROOT%\Inetpub\AdminScripts

3.     Type the following command:

cscript adsutil.vbs Set w3svc/AppPools/[YourAppPoolName]/LogEventOnRecycle 255

In the command above, replace [YourAppPoolName] with the actual name of the application pool upon which you want to enable the events.

For those among you who are curious as to the details of what this all means, here are some links to explain:

How to modify Application Pool Recycling events in IIS 6.0
LogEventOnRecycle Metabase Property (IIS 6.0)