Back in the day before the Visual Studio Team Systems profiler (and even before other sampling profilers were available), there were already performance problems – go figure.  My personal experience with an extreme example of this was in the early nineties when I was working on CodeView.  CodeView was the standalone COW (Character Oriented Windows) debugger that we shipped up until most of its functionality was incorporated into Developer Studio. Some of our internal customers were breathing down our necks because it took a half hour to load their application into CodeView to debug it when they had symbols available for all components.  As a stop-gap they were creating hybrid builds that only had debug information for certain modules, but they were definitely not happy with that solution!

So I knew up front that there was some problem with symbol load time, but how do I figure out what?  I used what was a pretty common technique in the day which was to run the scenario under the debugger, break a number of times and take a look at the call stack.  This doesn’t always work, but in this extreme it lead me right to the main culprit.  We were reading the entire symbol table, doing individual allocations for each symbol and doing a transformation from the file-formatted symbol into an internal format for the symbol.  So my random call stacks were consistently showing that we were spending lots of time in the code that did this allocation and transformation.  Reworking the code so that the entire debug section could be loaded as a single block and doing a quick run for some fixups gave me a 10x performance gain with not too much work.

Now while this may be a somewhat interesting historical note, you may ask what this has to do with sampling based profiling. With two pretty important differences, the above technique with the debugger is a poor man’s version of a sampling profiler.  In the sampling profiler, every time a specific event occurs we walk the call stack of the current thread and put this information in our log file.  When we later analyze the file, we aggregate the results to give you a picture of what your program was doing the most when these events occurred. 

Voila – we’ve just re-invented an ancient technique for tracking down performance problems!  But now let’s take a closer look at how the two techniques are different.

The first major difference is that we can gather and aggregate a lot more data than you can when breaking your program randomly in the debugger, so issues that are less extreme than the example I cited above can be detected.  Along with that comes the ability to do more sophisticated analysis than the flat set of call stacks that you would see in the debugger.  This is the justification for having a sampling based profiler in the first place.  I recommend that you take a look at some of the other VSTS profiler team’s blogs for loads of additional info. 

The second major difference is something that was pointed out by the performance engineers the other day, and puts a different spin on a recurring issue with sampling profiling.  When you break in the debugger it stops all threads and you can take a look at all of their call stacks.  With our implementation of a sampling profiler, we only report the call stacks of the thread that was active at the time our event occurred.  Why is this a significant difference, you may ask?  Because a thread that is sitting in WaitForSingleObject or any other system level synchronization call (this counts managed locks as well) won’t ever be sampled.  In our sampling profiler, we do a good job of showing you where you use the most of a particular resource (for instance processor usage in the default case), but we don’t do anything to show you when the problem is multiple threads contending over the same resource.  In the debugger you can look at the call stacks of the inactive threads and see that some particular thread is sitting in WaitForSingleObject much more often that you expect. 

Now before you run out and start using the debugger as your primary tool for identifying resource contention issues, let me tell you that we do have a completely different solution to help you identify resource contention issues.  This is one of the things that the instrumentation based flavor of our profiler does pretty well, and I’ll follow up with a blog about that sometime in the near future.

--dwg

https://blogs.msdn.com/graycode/articles/AspNetOffRoadProfilingArticle.aspx

May it provide many hours of pleasure for those who enjoy digging around the edges of the 'mainline' experience.

Introduction:

The VSTS profiler team sweated blood for beta2 to get the 'click and go' profiling of ASP.NET application to work smoothly. I believe the effort paid off and for mainline developer scenarios, you should be able to open a website in visual studio, run the peformance wizard and click the launch button. (Okay, maybe that's more than one click, but still..) If you want to try this, first take a look at Eric Jarvi's article ASP.NET Profiling

But there will always be the happy developer out there (especially using a beta product) that has a configuration we didn't consider 'mainline' or has a need to gather data in a way that isn't affected by all the things we have to do to make the user experience work for the 'click and go' solution. One obvious example is having Visual Studio running on the machine while gathering performance data has some effect. Another is that we force a shutdown of IIS right before we start gathering data. In this article I'm going to show you how to go off the beaten track a bit and gather performance data with our tools by hand (so to speak). For now, I'm going to assume you have a full install of VSTS on the machine that you're running the profiler on. But don't worry, you won't have to actually run Visual Studio while you're gathering data.

What Profiling Mode Do I Need? Sampling or Instrumentation?

This is just about always the first question to ask when setting up to gather data using our tools. So there quite a bit of blogging going on about the differences between sampling and instrumented profiling. See https://blogs.msdn.com/profiler for our team blog and related links. If you're interested in understanding the difference in depth, I highly recommend spending the time to go through our team blogs

But for now, here are the CliffsNotes®. Normally, I'd tell you to use sampling profiling to get a quick overview of how your app is behaving and hopefully narrow your problem down to hot-spots. Then use instrumentation profiling to take a closer look at the assembly(s) that contain the hot-spots identified by sampling.

But wait - you're running an ASP.NET application. When sampling you end up with a whole bunch of information about what’s going on in the ASP.NET framework that will most likely obscure what is going on in your own code. So for ASP.NET profiling, I recommend reversing the process. Try doing instrumented profiling on your own code and only if you can’t find anything there, move to sampling profiling to see if interaction with the ASP.NET framework might be your performance culprit.

What exactly do I want to profile?

ASP.NET is a complex system. Your code is only a small part of what is going on even if you are running a pretty big web application. For our purposes, there are two general categories of user code in ASP.NET applications.

1. Pre-built Assemblies - generally placed in the bin directory of the web site
2. Dynamically Built Assemblies - for example from code embedded in web pages and within the App_Code directory.

The following sections are perscriptive guidance on how to gather data for these types of data in each profiling mode. The careful reader will note that there are only three sections rather than the expected four. This is because in sampling mode, we gather data on the entire process, so the type of assembly you're interested in isn't relevant.

All of these sections assume that Visual Studio Team Systems is installed on your machine.

So put on you sturdy hiking boots, make sure you're not wearing anything that you mind getting muddy, and join me off the beaten track in one of these 'off-road' experiences.

--dwg

Instrumented Profiling of Pre-built Assemblies:

These are assemblies that are placed in the bin directory of the web site and so are not dynamically built by ASP.NET.

• vsperfclrenv /globaltraceon
  • This will set up the environment variables that tell .NET to load our profiling helpers.
• reboot (as of this writing reboot is still required)
• vsinstr [my assembly path]
  • vsinstr should be run on any pre-built assemblies that you wish to gather performance data on.
• vsperfmon /TRACE /OUTPUT:[your ouputfile].vsp /USER:"[ASP.NET worker process user]"
  • This will start the performance monitor that will collect the data as your application runs.
  • See Appendix A to find the user name that the ASP.NET worker process uses (depends on version of IIS you have running).
  • vsperfmon will run throughout the collection process, so you must start another command prompt to complete other commands.
• Run your test scenario against the web application
• iisreset /STOP
  • The profiled process must be shut down for collection to complete
• vsperfcmd -SHUTDOWN
• vsperfreport [your outputfile].vsp /SUMMARY:FUNCTION /PACKSYMBOLS
  • PACKSYMBOLS is necessary to get the symbols pinned correctly before the target assemblies are lost from their location in temporary storage, it also preserves symbols so the report can be meaningful if opened on another machine for analysis.
• iisreset /START
  • Only do this if you are going to do more profiling
• vsperfclrenv /globaloff
  • This step and the following step will prevent our profiling helpers from being loaded into ASP.NET in the future. This should definitely be done when you have completed profiling, as loading our helpers can affect performance.
• reboot
• Open [your outputfile].vsp in Visual Studio.
  • Since you packed the symbols in a previous step, you can copy [your outputfile].vsp to any machine that has a full install of VSTS and open in there.

Instrumented Profiling of Dynamically Built Assemblies

These are all of the assemblies that contain code from your web pages and code behind and App_Code directories.

• vsperfclrenv /globaltraceon
  • This will set up the environment variables that tell .NET to load our profiling helpers.
• reboot
• Back up the web.config file
• Fix up the web.config file for your site
  • See Appendix B for instructions on fixing up web.config
• vsperfmon /TRACE /OUTPUT:[your ouputfile].vsp /USER:"[ASP.NET worker process user]"
  • This will start the performance monitor that will collect the data as your application runs.
  • See Appendix A to find the user name that the ASP.NET worker process uses.
  • vsperfmon will run throughout the collection process, so you must start another command prompt to complete other commands.
• Run your test scenario against the web application.
• iisreset /STOP
  • The profiled process must be shut down for collection to complete
• vsperfcmd -SHUTDOWN
• vsperfreport [your outputfile].vsp /SUMMARY:FUNCTION /PACKSYMBOLS
  • PACKSYMBOLS is necessary to get the symbols pinned correctly before the target assemblies are lost from their location in temporary storage, it also preserves symbols so the report can be meaningful if opened on another machine for analysis.
• Restore the web.config file (if you don't intend to do more profiling)
• iisreset /START
• vsperfclrenv /globaloff (if you don't intend to do more profiling)
  • This step and the following step will prevent our profiling helpers from being loaded into ASP.NET in the future. This should definitely be done when you have completed profiling, as loading our helpers degraded performance.
• reboot
• Open [your outputfile].vsp in Visual Studio.
  • Since you packed the symbols in a previous step, you can copy [your outputfile].vsp to any machine that has a full install of VSTS and open in there.

You can profile both pre-built and dynamically generated .assemblies at the same time by following the instructions for dynamically built assemblies but manually running vsinstr against the prebuilts prior to fixing the web.config file.

Sampling Profiling

• vsperfclrenv /globalsampleon
  • This will set up the environment variables that tell .NET to load our profiling helpers.
• reboot
• vsperfmon /SAMPLE /OUTPUT:[your ouputfile].vsp /USER:"[ASP.NET worker process user]"
  • This will start the performance monitor that will collect the data as your application runs. See Appendix A to find the user name that the ASP.NET worker process uses. Vsperfmon will run throughout the collection process, so you must start another command prompt to complete other commands.
• vsperfcmd /ATTACH:[PID of ASP.NET WORKER PROCESS]
  • See Appenendix A to find the process id of the ASP.NET worker process.
• Run your test scenario against the web application.
  • It is important to run a scenario that stresses the CPU in order to get usable results out of sampling profiling. Try to get the CPU to a 100% for several minutes. Using the new Whidbey load test tools can be good for this.
• vsperfcmd /DETACH
  • This stops collection of data
• vsperfcmd /SHUTDOWN
  • This tells the performance monitor that you are done with your scenario and to close up the log file.
• vsperfreport [your outputfile].vsp /SUMMARY:FUNCTION /PACKSYMBOLS
  • This packs the symbols into your output file so that they can be transported to another machine for analysis. This is also necessary to get the symbols pinned correctly before the target assemblies are rebuilt.
• vsperfclrenv /globaloff
  • This step and the following step will prevent our profiling helpers from being loaded into ASP.NET in the future. This should definitely be done when you have completed profiling, as loading our helpers degraded performance.
• reboot
• Open [your outputfile].vsp in Visual Studio.
  • Since you packed the symbols in a previous step, you can copy [your outputfile].vsp to any machine that has a full install of VSTS and open in there.

Appendix A: Finding the User Name and Process Id for the ASP.NET Worker Process

• Make certain that your ASP.NET application is running
• Open the task manager
• Go to the Processes Tab
• Show the PID and User Name columns
  • Do this by going to the view menu and choosing select columns and select PID and User Name
• If aspnet_wp.exe exists use it, otherwise look for w3wp.exe.

 On win2K3Srv there may be multiple worker processes depending on how isolation is set up.  In this case, use %windir% \system32\iisapp.vbs to correlate w3wp PIDs with IIS application pools.

• Read the PID and User Name Columns (defaults listed below)
  • Windows Server 2003/IIS 6: (w3wp.exe, NETWORK SERVICE)
  • Windows XP/IIS 5.1: (aspnet_wp.exe, ASPNET)
  • Windows 2000/IIS 5.0: (aspnet_wp.exe, LOCAL SYSTEM)

Appendix B: Modifying the web.config File

1. Add or modify the compilation tag:

 <system.web>
 <compilation assemblyPostProcessorType="Microsoft.VisualStudio.Enterprise.Common.AspPerformanceInstrumenter,  Microsoft.VisualStudio.Enterprise.ASPNetHelper, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" />
 </system.web>

2. Set up the runtime tag to show the location of <runtime> <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1"> <dependentAssembly> <assemblyIdentity name="Microsoft.VisualStudio.Enterprise.ASPNetHelper" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" /> <codeBase version="8.0.0.0" href="file:///C:/Program%20Files/Microsoft%20Visual%20Studio%208/Common7/IDE/PrivateAssemblies/Microsoft.VisualStudio.Enterprise.ASPNetHelper.DLL" /> </dependentAssembly> </assemblyBinding> </runtime>

3. Set up the location of vsinstr and its dependencies

 <appSettings>
 <add key="Microsoft.VisualStudio.Enterprise.AspNetHelper.VsInstrLocation" value="C:\Program Files\Microsoft Visual Studio 8\Team Tools\Performance Tools\vsinstr.exe" />
 <add key="Microsoft.VisualStudio.Enterprise.AspNetHelper.VsInstrTools" value="C:\Program Files\Microsoft Visual Studio 8\Team Tools\Performance Tools\" />
 </appSettings>

• VsInstrLocation is the full path including fil ename of vsinstr.exe.
• VsInstrTools is the directory containing helper .dlls for vsinstr that aren’t available elsewhere on the path, these are msdis150.dll and mspdb80.dll.

Here is a complete sample web.config

 <?xml version="1.0"?>
 <configuration>
 <runtime>
 <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
 <dependentAssembly>
 <assemblyIdentity name="Microsoft.VisualStudio.Enterprise.ASPNetHelper" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
 <codeBase version="8.0.0.0" href="file:///C:/Program%20Files/Microsoft%20Visual%20Studio%208/Common7/IDE/PrivateAssemblies/Microsoft.VisualStudio.Enterprise.ASPNetHelper.DLL" />
 </dependentAssembly>
 </assemblyBinding>
 </runtime>
 <system.web>
 <compilation assemblyPostProcessorType="Microsoft.VisualStudio.Enterprise.Common.AspPerformanceInstrumenter, Microsoft.VisualStudio.Enterprise.ASPNetHelper, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" />
 </system.web>
 <appSettings>
 <add key="Microsoft.VisualStudio.Enterprise.AspNetHelper.VsInstrLocation" value="C:\Program Files\Microsoft Visual Studio 8\Team Tools\Performance Tools\vsinstr.exe" />
 <add key="Microsoft.VisualStudio.Enterprise.AspNetHelper.VsInstrTools" value="C:\Program Files\Microsoft Visual Studio 8\Team Tools\Performance Tools\" />
 </appSettings>
 </configuration>

Appendix C: Troubleshooting

There are two likely candidates for this.

1. The /USER switch was not correctly set when starting vsperfmon.exe. Check and make certain that you correctly identified the process identity of the ASP.NET worker process (see appendix B) and make certain that you provide it to vsperfmon.exe.
2. No assemblies were instrumented or the instrumented assemblies were never run. The command prompt that hosts vsperfmon.exe will show a message when it starts getting data from an instrumented process. If you never see that message, something went wrong. Make certain that the web.config file was correctly modified. If you manually instrument an assembly, make certain it is the one that is being run.

This is because vsperf80.dll must be on the path for the ASP.NET worker process. Both the Visual Studio setup and the stand-alone performance tools setup should place vsperf80.dll in [Windows]\system32.

locate vsperf80.dll and place in your [Windows]\system32 directory.

There are two likely candidates for this.

1. If you aren't seeing any symbols in your own code, make sure you run vsperfreport with the /PACKSYMBOLS flag BEFORE you restart the server.
2. If you aren't seeing any symbols in ASP.NET, make sure you have the symbol server set up correctly

    Set your _NT_SYMBOL_PATH variable to “symsrv*symsrv.dll*c:\localcache*https://msdl.microsoft.com/download/symbols”  

Need more help with symbols? Check out https://www.microsoft.com/whdc/devtools/debugging/symbolpkg.mspx

After spending a number of years OOM (Outside Of Microsoft) I was seduced by the promise of working on a profiler.  Since I occasionally feel the need for speed, I couldn’t resist and returned to Microsoft in the fall of 2003.  I was a debugger developer during a good part of my initial stint at Microsoft, so this seemed like a good fit.  At some later date I may blog about why the challenges of writing profilers and debuggers are more different that one might assume.

We’ve just shipped Beta 2 of Visual Studio Team Systems so it’s public now and I can start talking about how to use the thing.  We’ve done a lot of work to get an easily usable, “press the button and go” type of experience for the average developer or performance engineer working in our core scenarios.  (Take a look at IanHu’s blog for some good walkthroughs).  But like the debugger, when you really need a profiler, you really need a profiler.  If you happen to be off the beaten track enough that our user friendly ‘easy’ solution doesn’t work, you still need to grab that data and find out why you app is running so dang slow.  You won’t take no for an answer (at least I wouldn’t).  For those of you that find yourself in these situations we’re planning a series of “Off-Road” posts.  John Cunningham has posted the first of this series and I’m busily working on one of these little gems myself.

So stay tuned for some off-road profiling and other fast and fun GrayCode.

David