That's why I'm pleased to announce that we now have a better approach.  Visual Studio Perf Watson is now available for download and use with VS2010 SP1.  If you download and install Perf Watson it does everything for you (and us).  Here's how it works:

1. Perf Watson monitors VS to make sure it is responsive.
2. If VS goes unresponsive for more than 2 seconds, Perf Watson grabs a stack frame.
3. Perf Watson then times how long it takes for VS to become responsive again.
4. Perf Watson then takes the stack information along with the total time of the hang and sends it to Microsoft.
5. We take the data and load it into a database.
6. We analyze the data to see which call stacks are causing the longest hangs that impact the most customers, and then we log bugs on those.
7. Since we have call stacks, the bugs tend to be very actionable.
8. Since the data comes from real customers, with hit counts and severity information, we have no trouble prioritizing these performance hangs appropriately - we know exactly what they're costing you.

We've been using Perf Watson internally for a while now, and we've been able to identify and fix a lot of problems just based on our internal product usage.  We've also seen some nice correlation between issues raised by Perf Watson and issues found through other analysis.  This correlation helps us know that we're on the right track and gives extra weight to getting these issues resolved.

But our usage patterns aren't the same as yours.  Everyone uses VS a little bit differently.  That's why we're excited to put Perf Watson in your hands so that we can get an accurate picture of the performance issues you're running into, with strong metrics and data to back up the work we need to do.

I hope you'll decide to download and install Perf Watson. 

Regards,

David Berg
DDPE

We are especially interested in hearing about your experience with regards to performance, reliability, and quality.  The more details you share with us in this survey, the better we can understand your experience and apply what we learn into future versions.

The survey is very short and should take you no more than a few minutes. You can get started by clicking on Visual Studio 2010 Survey.   

It's important that we here from you whether you love us or hate us.  The more people who respond, the better we can understand how we're doing and whether or not we're doing the right things to make a difference.

Regards and thanks for your time,

David Berg
Developer Division Performance Engineering Team

The issue first came up back during VS 2010 beta in October of 2009.  Since then we've learned that while old, buggy drivers are the usual cause, some newer drivers and GPUs aren't as good at supporting VS's UI as we'd like.  (This is also an issue with VMs and VM hosts as Video Virtualization technology isn't very good and requires CPU level TLB virtualization support for decent performance, which most CPUs don't have.)

Fortunately, the software rendering inside WPF is pretty good, so the easy fix here is to force WPF to ignore the GPU and use software rendering (I've tested this on my own system, and I found that WPF's software rendering was actually slightly faster than GPU based rendering on my high end CPU with a mid-range graphics card - your mileage may vary).

But first, if you're seeing slow / broken screen updates you should verify you have the latest display drivers for your system.  (See "Guidelines for troubleshooting graphics issues in WPF applications" for more information.)

If that doesn't fix it, then there are three ways to force WPF to use software rendering. 

First and preferred, the final RTM version of VS2010 includes a UI for forcing hardware rendering off - for just VS.  With VS2010 open, go to Tools | Options, then select Environment | General (as shown below).  Then uncheck "Automatically adjust visual experience..." and "Use hardware graphics acceleration..." 

That should be sufficient. 

However, if you want to force software rendering mode for ALL WPF applications (not just VS), you have a second option.  Change (or add) one registry key:

 [HKEY_CURRENT_USER\Software\Microsoft\Avalon.Graphics]

"DisableHWAcceleration"=dword:00000001

Note that this key probably won't exist, and you'll probably need to create it.  To turn hardware acceleration back on, just change the "1" to a "0".

A third alternative is to adjust the hardware acceleration options from the display control panel.  However, we don't recommend this option as it impacts the entire machine, the details vary by manufacturer, and the exact impact of all the different options is untested.

If you try any of these - let us know how it works for you (DevPerf@Microsoft.com).  If it does improve performance, be sure to let us know how much and attach a DXDIAG output so we'll know which video driver / hardware configurations aren't working well.

Regards,

David Berg

Developer Division Performance Engineering Team

Applications that use Windows Automation APIs can significantly decrease Microsoft Visual Studio IntelliSense performance if Windows Automation API 3.0 is not installed. For example, the Windows pen and touch services can significantly decrease Visual Studio IntelliSense performance if Windows Automation API 3.0 is not installed. This article describes how to install the Windows Automation API 3.0 update. This update is available as a stand-alone download for 32-bit editions of Windows XP and for Windows Server 2003. This update is not available for 64-bit editions of Windows XP. The Windows Automation API is a component of the platform update for Windows Vista and of the platform update for Windows Server 2008.

Note that if you're running Windows XP SP2 you'll be told the patch isn't applicable, that's because you need to upgrade to XP SP3 first.  If you're running Vista or Windows Server 2008 you may already have the upgrade, since it's part of the automatic updates.  If not you can download the upgrade at the link above.  Windows 7 and Windows Server 2008 R2 ships with with Windows Automation API 3.0, so no upgrade is required.

The problem is that earlier versions of the Windows Automation API try to read the entire contents of list boxes that we post to the screen, which blocks us from virualizing them.  This is particularly a problem with intellisense as the number of items in an intellisense list is HUGE and it pops up on every character you type.

The API is typically activated when using accessibility devices (such as screen readers), pen / tablet computers, or touch devices, but some software activates it anyway (including iPhone synchronization).  Once activated it affects the entire system.

One of topics Brian addresses in his blog was the benefit of running Visual Studio on a 64-bit OS. We’d like to drill into that topic a little deeper in this post.

 Visual Studio’s devenv.exe process still runs as a 32-bit process. Under a 32-bit OS, a 32-bit process can only address up to 2 GB of private virtual memory. (For the sake of simplicity, we are going to ignore the /3 GB boot option for 32-bit Windows machines.) The remaining 2 GB of the 4 GB virtual address space is reserved for system addresses. This 2 GB max is an architectural limit on the size of a 32-bit process. All the code and data that gets loaded has to fit in this 2 GB virtual memory space. This may seem a little strange, but as developers working on Visual Studio, but we consider your code – and forms, XAML, DBML, DGML, PDBs for debugging, TFS Work items, and whatever else your Solution loads – as the data Visual Studio has to load & process. The problem arises when Visual Studio needs to load some combination of our code and your data that exceeds this 2 GB architectural limit.

Actually, memory fragmentation issues prevent a 32-bit process like devenv.exe from ever reaching its 2 GB architectural limit. Due to fragmentation, when a 32-bit process address space starts to grow into the 1.7 – 1.8 GB range, the risk that a virtual memory allocation request will fail starts to increase sharply. When a virtual memory allocation request fails, Visual Studio encounters an End of Memory error and crashes.

Obviously, this is a scenario you want to avoid. The cleanest way to get around the problem is to run Visual Studio on a 64-bit OS. On a 64-bit version of Windows, the private area of a 32-bit process expands to encompass the full 4 GB virtual memory addressing range. Due to fragmentation, you can’t quite get to the 4 GB upper limit either, but the effect of the change is to allow devenv.exe to address twice as much private virtual memory. Please don’t take this as a challenge, but we have yet to see a customer scenario that exhausts the full 4 GB address range.

So, if you think you could be a good candidate for running Visual Studio 2010 under a 64-bit OS, here is what to look for. We will run through a scenario we ran recently on the final RTM version of Visual Studio 2010 Ultimate. The test project is a smallish Web solution with a handful of simple ASP.NET Pages that use LINQ to query the MS SQL Server AdventureWorks demo database. This is a version of an app one of us built last year originally for stress testing some Visual Studio components he was working on. We would characterize this app as “borderline realistic.” For example, the AdventureWorks database has fifty or so inter-related Tables, and its SaleOrderDetail Table contains in excess of 120,000 rows. So it is not a trivial “Hello World” type of app, but the web forms themselves are pretty basic, lacking many of the UI components that you are likely to put into a real-world e-commerce application.

Table 1 shows the main steps in my test scenario and the amount of virtual memory Visual Studio consumed at the end of end of each step. The tool we recommend for measuring virtual memory usage at the process level is VMMap, one of the free Sysinternals utilities. With VMMap, I can see the overall virtual memory usage of the devenv.exe process, broken down according to different types of allocations, which is something we will drill into in a moment.

Here is a summary of the steps of the scenario we ran and the amount of virtual memory allocated by the devenv.exe process at the end of each step.

Step	Scenario	VM Usage
1	Open VS with empty solution	300
2	Open VS with web solution: ·         Solution Explorer, Team Explorer, and Server Explorer ·          Connect to TFS ·         Open one .cs file Opened in the VS Editor ·         Open one simple web form opened in Split mode ·         Open one (.dbml) Data Designer ·         Open one TFS Work Item open	910
3	Step 2, plus F5 to Debug Solution, and run to a breakpoint ·         Intellitrace is active at its low (default) setting	975
4	Step 3, run to Breakpoint; plus step 10x and then run to 2nd Breakpoint	1060
5	Step 4, with Resharper installed	> 1300

Table 1. Virtual memory usage of Visual Studio at the end of each scenario step.

In Step 5, we repeated the full scenario, but this time with Resharper, a popular Visual Studio add-in, installed. As Figure 1 shows, at that point, the virtual memory footprint of Visual Studio 2010 exceeds 1.3 GB. Committed bytes exceeds 1.2 GB. While that is not a particularly dangerous amount of virtual memory for Visual Studio to consume, it is still enough to start to make you wary.

Uri’s note, reprinted in the previous post, refers to an "issue" associated with the current technique for measuring processor utilization in Windows. As my reply mentioned, these are documented and well-understood issues. At the core is the methodology used to calculate processor utilization that was originally designed 20 years ago for Windows NT. Since one of the original goals of Windows NT was to be hardware independent, the measurement methodology was also designed so that it was not dependent on any specific set of hardware measurement features. And therein lies a tale.

This methodology was amply documented in my Windows 2003 Server Performance Guide, published in the Windows 2003 Server Resource Kit. As you know, I wrote that book before I came to work at Microsoft as a full-time employee, but the Windows Server performance team that helped me at the time was quite open to documenting the existing facility, warts and all.

Briefly summarizing what I wrote there (and which still applies today), the % Processor Time counters in Windows are measurements derived using a sampling technique. The OS Scheduler samples the state of the CPU once per system clock tick, driven by a high priority timer-based interrupt. Currently, the clock tick interval the OS Scheduler uses is usually 15.6 ms. (The precise value that the OS uses between timer interrupts is available by calling  the GetSystemTimeAdjustment() function.) If the processor is running the Idle loop when the quantum interrupt occurs, it is recorded as an Idle Time sample. If the processor is running some application thread, that is recorded as a busy sample. Busy samples are accumulated continuously at both the thread and process level. When a clock interrupt occurs, the Scheduler performs a number of other tasks, including adjusting the dispatching priority of threads that are currently executing, stopping the progress of any thread that has exceeded its time slice, as well as performing its CPU accounting.

The System\Processor Queue Length counter in Perfmon is an instantaneous counter that reflects the current number of Ready threads waiting in the OS Scheduler queue. When a performance monitoring application such as Perfmon requests the Processor Queue counter, there is a measurement function that traverses the Scheduler Ready Queue and counts the number of threads waiting for an available processor. Thus, the System\Processor Queue Length counter represents one sampled observation, and needs to be interpreted with that in mind. (If memory serves, the data collection process that Charles’ analysis relies upon gathers samples of this measurement several times per minute, & his servers are not idle by design. Which basically means I think what he is doing is just fine.)

What I believe Uri is referring to with regard to this processor Queue Length metric not being “correct” is anomalies in this measurement that are due to the kind of phased behavior you can sometimes see on an otherwise idle system. Even on an mostly idle system, a sizable number of threads can be waiting on the same clock interrupt (typically, polling the system state once per second), one of which also happens to be the Perfmon measurement thread, also cycling once per second. These sleeping threads tend to clump together so that they get woken up at the exact same time by the timer interrupt. (As I mentioned, this happens mainly when the machine is idling with little or no real work to do.) These awakened threads then flood the OS dispatching queue at exactly the same time. If one of these threads is the Perfmon measurement thread that gathers the Processor Queue Length measurement, you can see how this “clumping” behavior could distort the measurements. The Perfmon measurement thread executes at an elevated priority level of 15, so it is scheduled for execution ahead of any other User mode threads that were also awakened by the same Scheduler clock tick. The effect is that at the precise time when the Processor ready queue length is measured, there are likely to be a fair number of Ready Threads. Compared to the modeling assumption where processor scheduling is subject to random arrivals, one observes a disproportionate number of Ready Threads waiting for service, even (or especially) when the processor itself is not very busy overall.

This anomaly is best characterized as a low-utilization effect that perturbs the measurement when the machine is loafing. It generally ceases to be an issue when processor utilization climbs or there are more available processors on the machine. But this bunching of timer-based interrupts remains a serious concern, for instance, whenever Windows is running as a guest virtual machine under VMware or Hyper-V. (Please don’t get me started on that topic.)  Another interesting side discussion is how this clumping of timer-based interrupts interacts with power management, but I do not intend to go there either.

To summarize, the CPU utilization measurements at the system, process and thread level in Windows are based on a sampling methodology. Similarly, the processor queue length is also sampled. Like any sampling approach, the data gathered is subject to typical sampling errors, including

·         accumulating a sufficient number of sample observations to be able to make a reliable statistical inference about the underlying population, and

·         ensuring that there aren’t systemic sources of sampling error that causes sub-classes of the underlying population to be under or over-sampled markedly

So, these CPU measurements face familiar issues with sampling size and the potential for systematic sampling bias, as well as the usual difficulty in ensuring that the sample data is representative of the underlying population (something known as non-sampling error). For example, the interpretation of the CPU utilization data that Perfmon gathers at the process and thread level is subject to limitations based on a small sample size for collection intervals less than, for example, 15 seconds. The Performance Guide has a more detailed discussion of these issues, if you are interested.

Concerning a recent presentation by Charles Loboz, my colleague at Microsoft, at CMG09 in Texas (DEC 2009), Uriel Carrasquilla, a very knowledgeable and resourceful performance analyst at NCCI in Florida, writes,

“Mr. Loboz indicated that the theoretical calculations were based on Windows reporting of CPU busy and CPU queue length. My results indicate that the CPU queue length reported by Microsoft can't be correct.  I found that other CMG researchers came up with the same conclusion.

“I used similar ideas for my Linux, AIX and Sun Solaris data as reported by SAR, and Mr. Loboz ideas work like a charm.

“Question:  are you aware of this problem with Microsoft performance reporting? Is anybody working on this issue?”

Uri,

The short answer is,

“The derivation and interpretation of the System\Processor Queue Length performance counter is well-documented in the Windows 2003 Server Performance Guide, published back in the Windows 2003 Server Resource Kit. I believe the Processor Queue Length performance counter continues to be a very useful metric to track, as Charles and his team that is responsible for capacity planning for the many of the Microsoft online properties do.”

n  Mark

I will post a more expansive answer soon, allowing me to expound a little on a question that gets asked quite frequently, namely, “How are measures of CPU utilization in Windows derived and how can they be interpreted?”

First, though, I’d like to mention some of the work Charles Loboz and his team have been doing in the context of capacity planning to support some of the massive applications Microsoft provisions and supports. Consider an application like Hotmail that supports something in the neighborhood of 500 million mailboxes (give or take a couple hundred million) and a customer base that is global in scale. That is an order of magnitude larger than the largest corporate entity responsible for a single e-mail or messaging infrastructure. (My guess is that the largest corporate entity responsible for a single e-mail infrastructure is the US Department of Defense. Although, it might be the US Army instead since the different service branches probably operate separate infrastructures.) Performance monitoring and capacity planning on the scale of Hotmail or Search is certainly unprecedented. Do you think performance and capacity planning are important in an application the size of Hotmail. The answer is, “You bet.” The investment in hardware and power consumption alone justifies the capacity planning effort.

I had an opportunity to see some of the material that Charles was working on back in the summer & gave him some feedback on measurements and what valid inferences can be drawn from them. I haven’t read the final published version, but, I am certainly in sympathy with the approach he has adopted. (BTW, people like Uri that attended the recent CMG Conference have access to Charles’ paper, but no one else at the moment. As soon as Charles posts it somewhere publicly, I will point this blog entry to it.)

Although the scale Charles has to deal with is something new, the approach isn’t. I remember I also sent Charles a pointer to Igor Trobin's work, which I believe is very complementary. Igor writes an interesting blog called “System Management by Exception.” In addition, Jeff Buzen and Annie Shum published a very influential paper on this subject called “MASF: Multivariate Adaptive Statistical Filtering” back in 1995. (Igor’s papers on the subject and the original Buzen and Shum paper are all available at www.cmg.org.) My colleague Boris Zibitsker has also made a substantial contribution to what I consider a very useful approach, namely applying statistical process control (SPC) techniques to mine for gold within the enormous amounts of performance data that IT organizations routinely gather.

For perspective, Carnegie Mellon’s Software Engineering Institute (SEI) is usually credited with the original application of SPC techniques to software engineering. Len Bass at SEI wrote an excellent book entitled Software Architecture in Practice that embraces a broader perspective on quality in software development that I share. Len’s work on software quality metrics is close to my current interests here in Developer Division, especially around the potential value of scenario-driven development processes. (More on scenarios in the next blog post. Len’s submitted to a brief interview on Channel 9 recently that is posted here.)

Within the application life-cycle, performance, unfortunately, is considered one of the non-functional requirements associated with a system specification, which often means it is relegated to a secondary role during the much of the application life-cycle. In the specification process, getting the business requirements and translating them into system specifications correctly is the most pressing problem for developers of Line of Business applications. Performance is one of those aspects of software quality that often doesn’t get expressed during the software development life cycle until very late in the process when design flaws that lead to scalability problems are very expensive to fix.

Len Bass’s suggestion is that the requirements definition of a scenario should include a response time specification that can then be monitored throughout the development life cycle, just like any other set of requirements. That is the approach that we advocate using here in the Microsoft Developer Division for the software products that we built here. In developing Visual Studio 2010, for example, we made major commitments to performance requirements and regularly conduct automated acceptance testing against those requirements. However, you can also see from the many recent blogs on VS 2010 performance coming from the Microsoft Developer Division that we have not exactly gotten this down to a science yet.

The Len Bass and SEI approach is informed by experience building real-time control systems to fly airplanes, for example, where performance goals absolutely have to be met or the system cannot function as designed.  The performance requirements for real-time control systems applications are fundamentally easy to specify. If the computer system doesn’t recognize a condition and respond to it in time, the plane is going to crash. Bass makes the case for that system performance being one of those important Quality Attributes that needs to be addressed at the outset of the development life cycle, beginning with the architectural specification and continuing through the design, development and QA processes, to the delivered software’s operational phase, where it finally becomes the focus of performance analysts and capacity planners like Charles Loboz, Uriel Carrasquilla, and Igor Torbin.

What if you need to specify performance requirements for your LOB application, but don’t know where to start? Consider these two approaches:

·         Research in human factors engineering has generated a set of performance requirements for specific types of human-computer interactions in order to promote usability and improve customer satisfaction. Steve Seow, another colleague here in Microsoft, has an excellent, concise book on this topic called “Designing and Engineering Time,” complete with application responsiveness guidelines to help improve customer satisfaction. If you are in a position to design a new application from scratch from First Principles, Steve’s book will be an invaluable guide.

·         If the application currently exists in some form or another, measure its current performance. When you deliver the next version of the application, any significant decrease in performance from one release to the next will be perceived as an irritant and received negatively by existing users. In other words, measure the scenario of interest on the current system & use that as a baseline that you won’t regress in a subsequent version.

If you have to start somewhere, measuring current levels of performance around key scenarios and using them as a baseline gives you a place to start, at least. My experience is that the current level of performance sets expectations that the next version of the application must meet if you want your customers to be satisfied. In this context, Steve Seow's book cites psychological research into how much of a response time difference is necessary to be perceived as a difference. (About 20% in either direction makes a difference.) This reminds me of Gregory Bateson’s adage that “information is a difference that makes a difference.”

I do think that over time, humans adapt themselves to the response times they experience, such that, eventually, the response times of the new version become the new baseline. In other words, our positive or negative perception tends to atrophy over time. For example, consider the last time you acquired a new desktop or portable computer that was noticeably faster than its predecessor. How long was it before that rush of enthusiasm for the fast, new machine started to diminish? About 30 days, in my experience.

Twenty-five years ago, I was in a similar position to Charles, responsible for performance and capacity planning at a large telecommunications company for maybe 20 IBM mainframe computers, which was considered a whole lot of machines to keep track of back in those days. We used a product called MICS (full disclosure, I was a developer on MICS for a brief period in the mid-80s) to warehouse the performance data we were gathering from these machines and the SAS language for statistical reporting. Subsequently, at Landmark Systems, I designed a “management by exception” feature for our monitoring products that our customers loved based on very simple statistical process control techniques. Today, for Charles’ team that needs to monitor performance on 100,000s of servers, these statistical techniques are the only viable approach.

But, of course, Uri is correct. You do have to choose the right metrics. I believe Charles has. I will discuss the CPU utilization metrics in Windows in my next post.

-- Mark Friedman