Vance Morrison's Weblog

PerfView Hard Core CPU Investigations (Using CPU Counters) on Windows 10

Vance Morrison — Thu, 14 Mar 2019 20:50:59 GMT

One of the bread-and-butter investigations that people use the PerfView profiling tool for is a CPU investigation. Indeed when you type 'PerfView collect' it collects the information needed for a CPU sampling information needed for such an investigation. This CPU information consists of a stack taken every millisecond on each processor on the machine. This gives you a very good idea where you are spending your CPU time. This is all 'standard' stuff. It turns out that the CPU chips have the ability do do sampling based on other things besides time (1 msec intervals). The most useful of these are

After N instructions were executed (Instructions Retired)
After N cache misses (CacheMisses, LLCCachMiss, LLCCacheReference (LLC = Last Level Cache. The 'biggest' cache before you get to DRAM memory)
After N branch mispredictions (the processor is always guessing (based on previous experience) which way a branch goes, and this is how often it gets it wrong).

First why would you care? Don't we care about time anyway? And the answer is yes, we mostly care about time, but sometimes it is hard to understand why the time is being spent the way it is, and this extra information helps you figure out why. This is where CPU counters come in. These 'CPU Counters' are most useful when the important part of your program is CPU bound, and you have picked the best algorithms already, so now it is about having the CPU execute these instructions as fast as possible. Basically you are looking for 'slow' instructions to see if you can speed them up. Instructions can be slow because

It guessed wrong on which way a branch would push execution.
It needs data that is not in the first level cache (a cache miss).

These CPU counters can add valuable insight in figuring out why instructions are slow. Thus we still care about time, but this extra information can be helpful. For a while now PerfView could collect these CPU counters. They are accessed via the 'Ctrs' button in the 'Advanced Options' section of the PerfView collection dialog box. However until recently this was not very useful because of two limitations

These CPU counters were not available in a Virtual Machine (thus if you were running in the cloud you could not get them) and
If you had windows 'HyperVisor' on (that is your machine could host another virtual machine). CPU Counters were also unavailable.

Well, as of Windows 10 RS5 release, these restrictions have been lived. Thus it is easier get at them. Hurray! Accessing them works as it always did. Basically the 'Cpu Ctrs' text box is a list of 'CounterName:Number' pairs. Thus putting InstructionRetired:1000000 asks the machine to log a stack every time 1 million instructions have been executed. The helpful 'Ctrs' button opens a drop-down menu that shows all the counters that are available on your machine, and a good count for that particular counter (you want to pick a number so that you get about 1000 of these stacks a second. Sometimes you have to try a value, see what happens for your scenario and then change it so the result is in that range. Too high and the overhead is too high. Too low and you don't get the detail you want. Typically 1000/sec is a good compromise. Looking at the data: Once you added things to the 'Cpu Ctrs' textbox, then when you view the resulting data you should have new events called PMCSample (PMC = Precise Machine Counter). You view them in the 'Advanced Group -> Any Stacks' view. This view shows events of all types so typically you want to filter to just the PMCSample events (right click -> Include Item). This shows you were that counter metric is being 'spent' (e.g. what methods are executing many instructions, or causing many Cache misses). Multiple Counters: You CAN have up to 4 counters logging simultaneously, but I would start off collecting them individually. The reason is that all the 4 counters log the same PMCSample event. There is a 'Profile Source' field on the event (it is a small integer value) that uniquely identifies the particular counter (e.g. 26 == InstructionsRetired), and PerfView DOES show this number, but PerfView is currently not particularly friend about telling you the mapping from the number to the name. (you have to look in PerfViews log (lower right corner) to find the mapping from the name to the number). If you only turn on one, you don't have this problem. Good Uses for the Counters Believe it or not the 'InstructionsRetired' is one of the more useful CPU counters. The reason is that it is CPU bound code can be slow because either

Many instructions are being executed OR
The instructions are slow (Because of cache misses or whatever).

The first issue is very straightforward (there is a very clear relationship between the code the number of instructions it executes), but the second is subtle (it depends typically on how 'local' your data is). Because the the solutions for these two are very different, you want to know which one it is. InstructionsRetired can do this. For example suppose you have a 10% TIME regression on some CPU bound benchmark. Well you could collect InstructionsRetired data before and after, and see if you have 10% more instructions being executed, or whether you have about the same number of instructions (they are just slower). This tells you whether you algorithm is simply running more code, or whether it is likely touching memory in a way that it is inefficient. This is ESPECIALLY helpful for MULTIPROCESSOR algorithms. It is very easy for many processors acting on the same data to 'interfere' with each other (slowing things down). Before you can know that that is the problem, you have to rule out the 'easy' case that something is just running more instructions. Once you determine that indeed you are not executing more instructions but they are slower, the stacks show you which methods. You can then collect other CPU counters like DCache miss rates, and BranchMisprediction data (again before and after) to see what has changed. This lets you figure out how best to improve the situation. So there you have it. If you have a hard core CPU problem to optimize, having CPU counters is very helpful, and now with the improvements in Windows 10 RS5, it is easy to collect the data.

Diagnosing .NET Core ThreadPool Starvation with PerfView (Why my service is not saturating all cores or seems to stall).

Vance Morrison — Tue, 16 Oct 2018 19:04:03 GMT

This article is worth a read if you have

A service written in .NET (and in particular .NET ASP.NET Core)
Its ability to service incoming load has topped out
But the machine's CPUs are not fully utilized (often only using a small fraction of the total available CPU

In extreme cases, the loss of throughput is so bad that the service does not seem to be making progress at all. More commonly, you just are getting the throughout you expect, or you can't handle 'bursts' of load. (thus works great in stead state at 20% CPU used, but when you get a burst, it does not use more CPU to service the burst). What these general symptoms tell you is that you have a 'bottleneck' but it is not CPU. Basically there is some other resource that is needed to service a request, that each request has to wait for, and it is this waiting that is limiting throughput. Now the most common reason for this is that the requests require the resources on another machine (typically a database, but it could be a off-machine cache (redis), or any other 'out of process' resource). As we will see, these common issues are relatively straightforward because we can 'blame' the appropriate component. When we look at a breakdown of what why a request takes so long, we will clearly see that some part (e.g. waiting on a response from a database), takes a long time, so we know that that is the issue. However there is an insidious case, which is that the 'scarce resource' are threads themselves. That is some operation (like a database query), completes, but when it does there is no thread that can run the next step in servicing the request, so it simply stalls, waiting for such a thread to become available. This time shows up as a 'longer' database query, but it also happens any time non-CPU activity happens (e.g. any I/O or delay), so it seems like very I/O operation randomly takes longer than it should. We call this problem ThreadPool starvation, and it is the focus of this article. In theory threadpool starvation was always an potential problem, but as will be explained, before the advent of high scale, asynchronous services, the number of threads needed was more predictable, so the problem was rare. With more high scale async apps, however, the potential for this problem has grown, so I am writing this article to describe how to diagnose if this problem is affecting your service and what to do about it if it is. But before we can do that, some background is helpful What is a ThreadPool? Before we talk about a threadpool, we should back and describe what a Thread is. A Thread is the state needed to execute a sequential program. To a good approximation it is the 'call-stack' of partially executed methods, including all the local variables for each of those methods. The key point is that all code need s thread to 'run-in'. When a program starts it is given a thread, and multi-threaded programs create additional threads which each execute code concurrently with each other. Threads make sense in a world where the amount of concurrency is modest, each thread is doing a complex operation. However some workloads have exactly the opposite characteristics: there are MANY concurrent things happening, and each of them are doing simple things. Since all execution needs a thread, for this workload, it makes sense to reuse the thread, having it execute many small (unrelated) work items. This is a threadpool. It has a very simple API. In .NET it is ThreadPool.QueueUserWorkItem that takes a delegate (method) to run. When you call QueueUserWorkItem, the threadpool promises to run the delegate you passed some time in the future. (Note .NET does not encourage direct use of thread pool. Task.Factory.StartNew(Action) also queues methods to the .NET threadpool but makes it much easier to deal with error conditions and waiting on the result). What is a Asynchronous Programming? In the past, services used a 'multi-threaded' model of execution, where the service would create a thread for each concurrent request being handled. Each such thread would do all the work for that particular request from beginning to end, then move on to the next. This works well for low to medium scale serviced, but because a thread is a relatively expensive item (generally you want less than 1000 of them, and preferably < 100), this multi-threaded model does not work well if you want your service to be able to handle 1000s or 10000s of requests concurrently. To handle such scale, you need an Asynchronous style of programming and ASP.NET Core is built with this architecture. In this model, instead of having a 'thread per concurrent request', you instead register callbacks when you do long operations (typically I/O), and reuse the thread to do other processing while you are waiting. This architecture means that just a handful of threads (ideally about the number of physical processors), can handle a very large number (1000s or more) concurrent requests. You can also see what asynchronous programming needs a threadpool, because as I/Os complete, you need to run the next 'chunk' of code, so you need a thread. Asynchrounous code uses the threadpool to get this thread run this small chunk of code, and then return the thread to the pool so the thread can run some other (unrelated) chunk. Why do High-scale Asynchronous Services have Problems with ThreadPool Starvation? When your whole service uniformly uses the asynchronous style of programing, then your service scales VERY well. Threads NEVER block when running service code because if the operation takes a while, then the code should have called a version that takes a callback (or returns a System.Threading.Tasks.Task) and causes the rest of code to run when the I/O completes (C# has a magic 'await' statement that looks like it blocks, but in fact schedules a callback to run the next statement when the await statement completes). Thus threads never block except in the threadpool waiting for more work, so only a modest number of threads are needed (basically the number of CPUs on the machine). Unfortunately, when your service is running at high scale, it does not take much to ruin your perf. For example imagine you had 1000 concurrent requests being processed by a 16 processor machine. Using async, the threadpool only needs 16 threads to service those 1000 requests. For the sake of simplicity, that the machine has 16 CPUs and the threadpool has only one thread per CPU (thus 16) Now lets imaging that someone places a a Sleep(200) at the start of the request processing. Ideally you would like to believe that that would only cause each request to be delayed by 200 msec and thus having a response time of 400 msec. However the threadpool only has 16 threads (which was enough when everything was async), which, all of a sudden, it is not enough. Only the first 16 threads run, and for 200 msec are just sleeping. Only after 200 msec are those 16 threads available again, so another 16 requests can go etc. Thus the first requests are delayed by 200 msec, the second set for 400msec, the third by 600 msec. The AVERAGE response time to does up to over 6 SECONDS for a load of 1000 simultaneous requests. Now you can see why in some cases throughput can go from fine to more than terrible with just a very small amount of blocking. This example is contrived, but it illustrates the main point. if you add ANY blocking, the number of threads needed in the threadpool jumps DRAMATICALLY (from 16 to 1000). The .NET Threadpool DOES try to inject more threads, but does it at a modest rate (e.g 1 or 2 a second), so it makes little difference in the short term (you need many minutes to get up to 1000). Moreover, you have lost the benefit of being async (because now you need roughly a thread per request, which is what you were trying to avoid). So you can see there is a 'cliff' with asynchronous code. You only get the benefits of asynchronous code if you 'almost never block'. If you DO block, and you are running at high scale, you are very likely to quickly exhaust the Threadpool's threads. The threadpool will try to compensate, but it will take a while, and frankly you lost the benefits of being async. The correct solution is to avoid blocking on the 'hot' paths in an high scale service. What typically causes blocking? The common causes of blocking include

Calling any API that does I/O (thus may block) but is not Async (since it is not an Async API, it HAS to block if the I/O can't complete quickly)
Calling Task.Wait() or Task.GetResult (which calls Task.Wait). Using these APIs are red flags. Ideally you are in an async method and you use 'await' instead.

So in a nutshell, this is why threadpool starvation is becoming more common. High scale async application are becoming more common, and it is very easy to introduce blocking into your service code, which forces you off the 'golden path' for async, requiring many threadpool threads, which cause starvation. How do I know that the ThreadPool is starved for threads? So how do you know this bad thing is happening? You start with the symptoms above, which is that your CPUs are not as saturated as you would want. From here I will show you a number of symptoms that you can check that give ever more definitive answer to the question. The details do change depending in the operating system. I will be showing the windows, case but I will also describe what to do on Linux. As always, whenever you have a problem with a service, it is useful get a detailed performance trace. On windows this means downloading PerfView taking a PerfView Trace

PerfView /threadTime collect

On Linux it currently means taking a trace with perfCollect however in release 2.2 of .NET Core you will be able to collect traces using a 'dotnet profile' command (details are TBD, I will update when that happens). If you are using Application Insights, you can use the Application Insights profiler to capture a trace as well. You only need 60 seconds of trace when the service is under load but not performing well. Look for growing thread count. A key symptom of threadpool starvation is the fact that the threadpool DOES detect that it is starved (there is work but no threads), and is trying to fix it by injecting more threads, but does it (by design) at a slow rate (about 1-2 times a second). Thus In PerfView's 'events view' (on windows), you will see the OS kernel events for new threads showing up at that rate. Notice that it adds about 2 threads a second.

Linux traces don't include OS events in the 'events' view but there are .NET Runtime events that will tell you every time a thread is created. You should look at the following events

Microsoft-Windows-DotNETRuntime/IOThreadCreation/Start - windows only (it has a special queue for threads blocked on certain I/O) for new I/O workers
Microsoft-Windows-DotNETRuntime/ThreadPoolWorkerThread/Start - logged on both Linux and Windows for new workers
Microsoft-Windows-DotNETRuntime/ThreadPoolWorkerThreadAdjustment/Adjustment - indicates normal worker adjustment (will show increasing count

Here is an example of what you might see on Linux. This will continue indefinitely if the load is high enough (and thus the need for more threads is high enough). It may also show up in OS performance metrics that count threads in a given process. (so you can do it without PerfView at all if necessary). Finding the blocking API If you have established that you do have a threadpool starvation problem, as mentioned the likely issue is that you called a blocking API that is consuming a thread for too long. On windows, PerfView can show you where you are blocking with the 'Thread Time (with StartStop Activities) view". This view should show you all the service requests in the 'Activities' node, and if you look at these, you should see a pattern of 'BLOCKED_TIME" being consumed. The API that causes that blocking is the issue. Unfortunately this view is not available on Linux today using 'perfCollect'. However the application insights profiler should work and show you equivalent information. In version 2.2 of the runtime the 'dotnet profile' should also work (TBD) for ad-hoc collection. Be Proactive. Not that you don't need to wait for your service to melt down to find 'bad blocking' in your code. There is nothing stopping you from running the PerfView /Collect on your development box on just about ANY load, and simply look for blocking. That should be fixed. You don't need to actually induce a high-scale environment and see the threadpool starvation, you know that that will happen if you have large scale (1000s of concurrent requests), and you block for any length of time (even 10msec is too much, if it is happening on every request). Thus you can be proactive and solve the problem before it is even a problem. Work-around: Force more threads in the ThreadPool As mentioned already, the real solution to thread-pool starvation is to remove the blocking that consuming threads. However you may not be able to modify the code to do this easily, and need SOMETHING that will help in the very short term. The ThreadPool.SetMinThreads can set the minimum number of threads for the ThreadPool (on windows there is a pool for I/O threads and a pool for all other work, and you have to see from your trace which kind of thread is being created constantly to know which to set). The normal worker thread minimum can also be set using the environment variable COMPlus_ForceMinWorkerThreads, but there is not environment variable for the I/O threadpool (but that only exists on Windows). Generally this is a bad solution because you may need MANY threads (e.g. 1000 or more), and that is inefficient. It should only be used as a stop-gap measure. Summary: So now you know the basics of .NET ThreadPool Starvation.

It is something to look for when you service is not performing well and CPU is not saturated.
The main symptom is a constantly increasing number of threads (as the threadpool tries to fix the starvation)
You can determine things more definitively by looking at .NET runtime events that show the threadpool adding threads.
You can then use the normal 'thread time' view to find out what is blocking during a request.
You can be proactive and look for blocked time BEFORE at-scale deployment, and head off these kinds of scalability issues.
Removing the blocking is best, but if not possible increasing the size of the ThreadPool will at least make the service function in the short term.

There you have it. Sadly this is become more common than we would like. We are likely to add better diagnostic capabilities to make this easier to find, but this blog entry helps in the mean time.

Using PerfView (ETW) in Windows (Docker) Conainers

Vance Morrison — Thu, 16 Aug 2018 20:25:49 GMT

Virtual machines have been around on Windows for many years now, and for the most part you treat the exactly as if it was a physical machine. In particular with respect to a tool like PerfView, we expect it to 'just work' and I don't have to say anything special. (Actually there is one things different, which is that the 'CPU Ctrs' Textbox on the collection dialog doesn't do something useful, see Cpu Ctrs TextBox for details). The trend in virtual machines (VMs) is to make them super light weight, so that they are so cheap you use them more frequently. The main way of doing this is to share more of the VM with its underlying host, so that only what REALLY needs to be distinct for each VM is actually unshared. These light weight VMs are called containers and Windows started supporting these light weight containers (Microsoft calls them Windows Server Containers) a few years back. See Windows Container Quick Start for more. A tool/system called Docker is used to create, configure an run applications in these light weight operating system images. I would have liked to say 'PerfView' 'just works like you would expect in these Docker containers, and I would be almost right. But there are enough caveats that I wrote some documentation on using PerfView in windows Docker containers. Without repeating that documentation much the main points are

ETW (and PerfView) work in Windows Docker Containers
However you need a new enough version of the Windows OS (Win 10 1709 or later)
You only get the basic kernel events (CPU + Context Switch, but that covers most scenarios.
Because the OSes used in containers can't display a GUI, PerfView has to be used with the /logfile option as described in its automation documentation.
But other PerfView 'just works'
You can even collect on Windows Nanoserver OS, however you can't use PerfView but stripped down version called PerfViewCollect.

So if you are using Windows Docker containers for your app, and want to do a performance investigation, know that it is possible to do so, and you should read the PerfView Docs on Windows Containers to learn more (Edge users, you may have to search for 'Containers' to find the right spot). This documentation is also part of the PerfView help, so you can find it there if that is more convenient. Once you have collected the data from the container, you copy it to a windows machine with a GUI and analyze the trace. There is one final caveat: as of right now (8/2016) there seem to be an OS problem in what is called 'merging' where the information needed to look up symbolic information from Microsoft's is correctly added to the data file. The exact conditions where it fails is not really clear (it works for some DLLs but not others in the same trace), but the symptom is that sometimes perfView's 'Lookup Symbols' on a native Microsoft-supplied DLL (typically some OS DLL), will fail saying that that data file may not be 'merged' properly. For .NET applications this s rarely a problem since you are much more interested in the app itself and the .NET Framework, and not the OS DLLs, but for other scenarios it may be an issue. There is a bug filed on this, and we will see what we learn.

Debugging through the .NET Core framework using VSCode (e.g. on Linux)

Vance Morrison — Mon, 02 Jul 2018 17:14:50 GMT

In my blog 'Debugging Through the .NET Core framework' I give specific instructions for setting Visual Studio up so that you can debug into the source code for the .NET Core Runtime. Since Version 2.1 of the .NET Core runtime, it is also possible to do this using the 'Visual Studio Code' editor. Since Visual Studio code runs on Linux (as well as windows, or MacOS), this is what you would be using if you were developing on non-windows platform. Conceptually we are doing the same thing, but the exact details are a bit different, so I document them here. They are mostly a transcription of VSCode doc on the subject. Basically you have to sure you are running V2.1 of .NET Core or beyond, as well as V1.15 or beyond of the OmniSharp C# VSCode extension. With those preliminaries out of the way, you basically need to update the '.vscode\launch.json' that VSCode set up in your project to describe how to launch your app and add these entries inside the 'configurations' array for the configuration that you use.

"justMyCode": false, "symbolOptions": { "searchMicrosoftSymbolServer": true }, "suppressJITOptimizations": true, "env": { "COMPlus_ZapDisable": "1" }

The critical piece is turning off 'justMyCode' This tells the debugger that you want to step into things outside your project. However in order to do this it needs to find the symbols for code outside the project. This is what the 'searchMicrosoftSymbolServer' does. Note that this WILL cause alot of files to be downloaded from the symbol server THE FIRST time you debug something, but after your local caches has been populated it should be fast. The last two entries improve the experience debugging through the code. By default you are debugging the optimized code that was compiled by Microsoft and shipped with the .NET Core framework. This means you don't tend to see local variables or even arguments of methods. Sometimes this is OK, but sometimes it can be VERY frustrating. Setting the 'COMPlus_ZapDisable' environment variable tells the runtime NOT to use the precompiled code, and the 'suppressJITOptimizations' says when you do recompile it (Just in time) don't optimize it. Together this makes the code unoptimized and thus you see all the local variables as you would like. This DOES slow the code down a bit, but for debugger you probably don't care.

Update: Debugging Through the .NET Core Framework V2.0.3+ (Windows)

Vance Morrison — Wed, 20 Dec 2017 17:52:07 GMT

This blog shows you how to debug through the .NET Core source code using Visual Studio. There is a companion blog entry that indicates how to do it for Visual Studio Code (e.g. what you would use on Linux). As you may or may not know, for a couple years in addition to running .NET code on the 'Desktop' runtime that ships with the Windows operating system, the .NET Runtime team also ships a version of the runtime called .NET Core which can be installed on various favors of Linux as well as Windows. In addition to being cross platform, it is also possible to ship your application 'with the runtime' which means that your app runs the same regardless of what version of Windows / Desktop .NET is on the target machine. See 'choosing between .NET Core and .NET Framework' for more on when to use .NET Core. Visual Studio (currently 2017) is the suggested editor/debugger, for making .NET Core applications, and there is a free download for the Community Edition, so there reason not to use this excellent debugger/IDE. One of the nice aspects of the .NET Core runtime is that it is open source. The source code for what ships in the runtime mostly lives in CoreCLR repo (for very fundamental things) CoreFX repo (generally the rest of System.*) or one of the ASP.NET repos (for *AspNet* dlls). With all the source code available, it is possible to have a excellent debugging experience with framework code. Basically when you need source code from the framework it can be fetched by the debugger from GitHub and presented just like local source code. Visual Studio 2017 supports the SourceLink standard that enables this as of V2.0.3 of the .NET Runtime the team has published all the right symbolic information to the Microsoft public symbol server to make this experience 'just work'.

Stepping Into the .NET Framework

Cutting to the chase: if you are running on Windows, Have Visual Studio 2017 or later Installed, and have V2.0.3 ) (or later) of the .NET Core SDK installed (Shipped 12/15/2017). then you only need to do two things in Visual Studio to enable the experience of being able to step into the source code for the .NET Runtime (and ASP.NET)

Uncheck the checkbox: Tools -> Options -> Debugging -> General -> Enable Just My Code. This tells Visual Studio that you WANT to look at framework code.
Check the checkbox: Tools -> Options -> Debugging -> Symbols -> Microsoft Symbol Servers. This tells Visual Studio where to get the symbolic information for framework code.

That is it! you can now step into calls to framework code, or set breakpoints in framework methods and Visual Studio will automatically fetch the appropriate symbolic information and source code from GitHub and present it to you so you can step through it. Note that Visual Studio is pretty aggressive about downloading symbolic information so it can definitely slow down debugging (there will be a dialog box as it does its work), so you might want to turn off uncheck the use of 'Microsoft Symbol Servers' when you don't need it. Note that Visual Studio caches the symbolic information locally, so even after you no longer are probing the Microsoft Symbol Server, you still will be able to step into framework code, as long as the information has already been downloaded (likely if you are debugging the same scenario).

Pro Tip: Speeding up debugging by avoiding pointless symbol lookup.

Checking the 'Tools -> Options -> Debugging -> Symbols -> Microsoft Symbol Servers' checkbox in the instructions above has the disadvantage that whenever a DLL is loaded the debugger will see if there are symbols for it out on this server. If your network connection is less than great (e.g. congested WiFi), it can lead to slowdown and poor responsiveness in the debugger. But what is also true is that the debuggers caches all the symbols it finds in %TEMP%\SymbolCache. What this means is that after you have run your program ONCE in the debugger, it will have downloaded ALL the framework and ASP.NET symbol files loaded in that scenario to SymbolCache. That means that if you only want to look at framework DLLs from that scenario you can turn off the 'lookup on 'Microsoft Symbol Servers' and source debugging CONTINUES to WORK. This will work for all debugging sessions (since they all use %TEMP%\SymbolCache) as long as you don't need NEW DLLs or if you upgrade the framework DLLS. Note that even after you turn off the 'Microsoft Symbol Servers', source code lookup continues to work (even for new source code files), as long as the DLL that you are debugging has its PDB downloaded to the SymbolCache.

Disabling Framework Optimization (Seeing All Local Variables)

Enabling stepping is often enough (you can follow the flow of execution, however, because the framework code was precompiled as OPTIMIZED code, the debugger often will not have enough information to display local variables. Thus Visual Studio will show a 'Cannot obtain value' message in these cases. This can be frustrating if that variable was very important. By doing two more tweeks, you can solve this problem as well.

Right click on the Project File in the solution explorer -> Properties -> Debug. Then press the 'Add' button in the 'Environment Variables', and add the environment variable 'COMPLUS_ZapDisable' with the value of 1. Be sure to type 'Ctrl-S' (save) after you make the edit because changes to the property page will NOT take effect by simply running the program, you have to explicitly save it. When you are done the property page should look like this.

Check the checkbox: Tools -> Options -> Debugging -> General -> Suppress JIT optimization on module load.

The environment variable tells the runtime to disable the use of precompiled images (called 'Zap'ing). This forces the runtime to Just In Time compile framework code. By setting the 'Supress JIT optimization' the debugger then forces the runtime to not optimize this code, which allows the debugger to do a good job viewing local variables. You know you succeeded when all the local variables that are in scope show up properly (by the way technically local variables are not in scope at the very entry of a method, you have to step once to be 'inside' the method where they are in scope). If you are not seeing this you can see use the Disassembly window (right click -> Goto Disassembly) to see if the code looks unoptimized (unoptimized will have many moves to and from the stack [rbp+X]). You can also see if the addresses of the code live inside the DLL you are currently in (Use the call stack to see what DLL you are currently in, and the Debug -> Windows -> Modules to see the limits of each DLL). If the code is outside the DLL than it is JIT compiled (which is what we want). If you are like me and think that this should 'just work' please go and my Visual Studio Suggestion to fix this and vote for it.

More Information

So happy debugging through the framework! Here are some additional resources

Dotnet Core Issue 897 - is a gitub issue that is the 'official' source of all the information presented here. It also talks about how to do this on V2.0 (earlier) version of the runtime. It is related to AspNet Issue 4155 which basically asked the same question about ASP.NET framework code. This later issue has a bunch of screenshots that can be useful if the terse instructions above are not enough.

Sadly, things are not quite so automatic if you are developing on Linux rather than Windows. Among other things Visual Studio does not run on Linux (but VSCode does). We are working on making this experience good as well and I hope to write a blog post about how to step through source on non-Windows platforms relatively soon.

PerfView: How to .NET Core Framework Symbols and Source (On Windows)

Vance Morrison — Thu, 15 Dec 2016 16:14:43 GMT

In a previous blog on using PerfView with .NET Core I pointed out that free PerfView profiling tool (see PerfView tutorial for more) should 'just work' to collect performance data for .NET Core scenarios on windows operating system. (On Linux, only CPU analysis can be done current see CPU Analysis on Linux using PerfView for more). However as I mentioned in that blog near the end, if you need to see what is going on inside the .NET Framework itself (and often this is very useful), you are not likely to get the symbols you need. We have fixes in the pipeline, but it may be several more months (I gave the estimate of March 2017) until things get ironed out. Thus if you collected a trace with PerfView for a .NET Core scenario and you don't get symbols for DLLs like System.* or Microsoft.*, this is likely the problem. I wanted to reiterate the work around since it is not hard, but also not obvious. If you collect the data by

Opening an administrative command prompt (this is important, as the special options get lost if PerfView itself has to elevate)
Run PerfView /ForceNGENRundown /BufferSizeMB=256

Then collect as normal (using Collect -> Collect and hit the 'Start collection' button), should fix the problem. Getting Framework Source Code PerfView has a 'Goto Source' option (select a frame -> right click -> Goto Source), which can be made to work with the framework (as well as your own code). We want this to 'just work' but again, we are not quite there yet. If you try this, it is likely to fail, and you can see exactly why by looking in the PerfView log (button in lower right corner). The typical failures is that it cant find the symbol file (PDB file), or it can't find the source file. For your own code, you have to make sure that your symbol path includes directories where you built your symbol files. You can add these by using the File -> Set Symbol Path. For framework DLLs, the defaults should be fine (which look things up on a Microsoft-standard symbol server). However the next step is that it needs to find the exact source file. Again for your own code, you need to set a 'source path' that includes a base directory that is a parent of where your source code lives. You can add these search locations by using File -> Set Source Path. Again you can check the log to see exactly where PerfView looks and whether it finds the file or not. For framework code, the Framework symbol files actually do have the information needed to find the source code, but there are permissions problems currently. The work-around currently is for you to download the source code yourself, and then point to it (as if it were your own code). The .NET Framework code lives in one of two GitHub repositories.

https://github.com/dotnet/coreclr which contains the runtime (mostly coreclr.dll, and corejit.dll) and that minimal part of the class library that is deeply connected to the runtime (called System.Private.Corelib.dll).
https://github.com/dotnet/corefx which contains the rest of the .NET Core framework (thinks like System.XML* System.Collections.* This is most of the public surface area of the .NET Runtime.

For this blog I am going to assume you know how to clone a hithub repo, if not take a look at these instructions for doing it using Visual Studio. Once you have cloned these two repositories you need to set your source path in PerfView to include directory for each of these clones. Finally, you need to synchronize both of the repositories to the exact state that corresponds to the version of the binaries you are actually using. The key piece of information I want to provide here is to note that DLLs have in their version information the GitHub commit hash (ID) that allows you to fetch exactly the correct code. Thus the basic procedure is

Find the complete path for the .NET Runtime or framework DLL you need.
Look in the version information for that DLL to get the GitHub commit ID. (and determine if the DLL was part of coreclr or the corefx repo)
Sync the local clone of that code to that commit ID

At that point, assuming you have put these repos in your PerfView source path, PerfView will then look up exactly the correct source.

There are numerous different ways of achieving each of the steps above, but I will give you one way here.

To find the complete path for the DLL, open the 'Processes' view under the ETL.ZIP file and click on the 'View Process Modules in Excel' link. This shows for every process exactly which DLL was loaded.
Once you have the path, open its directory in Windows explorer select the file right click -> Properties and select the 'Details' path. You should see a line labeled 'Product version' and it should say something like 'Commit Hash: 9688ddbb...' This is what you are looking for. You only need the first 8 bytes of that hash (The hash is actually very long but Git accepts any prefix that is unique in the repo, and 8 digits is almost certainly unique). If the DLL coreclr, corejit or System.Private.Corelib.* then it lives in coreclr, otherwise it lives in corefx.
Once you have the commit ID you can go to the appropriate repository and sync to it. One way of doing that is the following which created a name (branch) for the commit (in this case called Version1.1), and then uses git checkout to sync to it.
- git branch Version1.1 9688ddbb
- git checkout Version1.1

Tada! At this point you should have exactly the right source code. Now clearly this is a bit of a pain and we believe in a few months PerfView will 'just work' and automatically open the right source code for you, but in the mean time this is a way of working around it.

Performance Analysis on .NET Core applications with PerfView (on Windows)

Vance Morrison — Wed, 02 Nov 2016 19:33:28 GMT

Back in July, the .NET Runtime Team shipped the first official version of .NET Core. .NET Core is a stand-alone down version of the .NET Runtime that is Open Source (it lives in the .NET Core GitHub Repo) and cross platform (runs on Windows, OSX and a variety of Linux distributions). Version 5 of ASP NET Web framework uses .NET runs on .NET Core (as well as the Desktop .NET Framework), and allows those web services to be run on Linux if desired. You can also build 'console' applications that can do all sorts of tasks. The links above describe how to actually build things with this new framework. Once you have done this, inevitably, you will want to do performance analysis on your .NET Core app, and as you would expect, PerfView can help you there. Generally speaking, PerfView should 'just work' with .NET Core (on Windows), and the investigation the normal documentation works just as you would expect. In particular you can use PerfView to collect CPU samples and you should be able to see all your code displayed symbolically as you would expect. Pretty much any investigation on .NET Core works like it would if you were running on the .NET Desktop framework. There is a slight caveat in the short term however: If you want to look inside the .NET Core framework, and see the symbolic names of .NET Framework code, that also works fine (it looks the necessary symbolic information up on standard Microsoft symbol servers) IF you are NOT using pre-release versions of the code. For the next few months, however (10/2016 through 3/2017) if you go to the .NET Core Site for getting started on windows https://www.microsoft.com/net/core#windows, you will find it probably DOES point you at prelease code. What his means is that you may not get symbolic information for .NET Framework code. If you need to see these methods, there is a work-around.

Open an Administrative command prompt
Start PerfView with the command 'PerfView /ForceNGENRundown /BufferSizeMB=256

And use that to collect the data. What this does is force the Runtime to dump symbolic information (NGEN Rundown) for managed code instead of trying to look up that data using Microsoft's symbol servers. This allows PerfView to display symbols for ANY managed code you use (including the framework) and works around this problem. UPDATE 10/2017. IF you are profiling .NET Core please get at least version 1.9.65 of PerfView. The current version of PerfView on the download page is 1.9.0 which is reasonably old (over a year) and is missing some fixes that are useful for the .NET Core case. In particular fixes to simply 'get framework symbols' are not present. We are working on getting the Download page updated, but it may take a while because the way we sign packages has changed since V1.9.0 was release. While the instructions above are still accurate and useful as a work-around, it is strongly recommended that until we get the download page updated, if you are using PerfView for .NET Core applications that you go to the Github releases page and download the 1.9.65 or later version.

Collecting Performance Data on Linux using PerfCollect and PerfView.

Vance Morrison — Fri, 07 Oct 2016 15:45:55 GMT

In a previous post on Linux Tracing I described how it was possible to collect traces of CPU samples on Linux machines and view them with PerfView. Basically there is a Bash script called PerfCollect that turns on the Linux kernel's CPU sampling. You also need to set some environment variables if you are running .NET Core Runtime scenarios so you get the names of the .NET methods. All the information in that previous post is still relevant, but we have added some new capabilities. In particular we have added the ability to get at EventSource Events. Because this is an area of active development, the instructions embedded in PerfView itself can get out of date. Thus we have put the PerfCollect Instructions up on GitHub so we can keep them up to date easily. The main take-away of this blog is that you should use the instruction from the PerfCollect Instructions link if you are trying to collect performance data on Linux. Vance

PerfView is now Open Source On GitHub

Vance Morrison — Sun, 18 Sep 2016 17:26:39 GMT

I am happy to announce, that the PerfView source code is now open source as a GitHub repository. It is available at https://github.com/Microsoft/perfview The readme associated with the GitHub repository has getting started information (how to fetch the repository, how to build, test and deploy the code. We use Visual Studio 2015. You can download a free copy of Visual Studio 2015 Community Edition that has everything you need to clone, build test and deploy PerfView. Thus you can get going with PerfView RIGHT NOW. The instructions on the PerfView repository tell you how to get started even if you know nothing about GIT (although knowing something about GIT and Visual Studio certainly helps). The readme also talks about how you can log issues as well as contribute to PerfView. You should definitely read the documentation about contributing if you intend to do that (probably best however to just start with learning to build / run / debug and get familiar with the code. We could definitely use help in both documentation and testing, so this is a great place to contribute, especially to start with. One of the nice things about the PerfView repository is that it has the latest version of the code. Thus if you need a bug-fix or some recently added feature, you can get this by simply building the repository. It turns out that one of the new features of PerfView (not present in the 1.9 version on the download center) is the ability to generate good symbolic information for .NET Core runtime native code. Thus if you are trying to do performance investigation of a .NET Core scenario you will find that if you use version 1.9 of PerfView that the names of most method in the framework are not resolved. I will be publishing new version to the download center soon, but in the mean time you can build your own PerfView that fixes this and gives a good experience investigating .NET Core Runtime framework issues. The PerfView repository also contains the complete source code for the Microsoft.Diagnostics.Tracing.TraceEvent library. Currently the PerfView repository does not have all the logic for actually generating the nuget package, but that will soon be added. In the mean time you can certainly build the DLL if you need that for your own ETW parsing needs. Enjoy! Vance

Encore Presentation Measure Early and Often For Performance

Vance Morrison — Fri, 13 May 2016 18:26:00 GMT

Back in 2008 I wrote a two-part MSDN article (April 2008 and May 2008) about the basics of performance investigation entitled 'Measure Early And Often for Performance'. At the time I also wrote a blog linking those articles. Well it seems that after about 8 years, MSDN archives articles and so the links got broken. However the content in those articles really holds up nicely over time. The true basics of performance investigation have really not changed.

Step 1, Have a Plan (that is care about performance enough to devote some thought to it (and create a plan)
Step 2 Measure - That is get enough information to execute on your plan. You will have to do this many times hence the early and often).

A lot has changed in the past 8 years. In particular back when these articles were written PerfView did not exist, so now it is much easier to measure early and often. But a lot has stayed the same, and in particular the general guidance that these articles give is still relevant. So I when and plucked these out of the archives and put them here.

I apologize for the questionable formatting. If you really want good formatting you can go to the MSDN Site, find the April and May 2008 issues, save them to disk, right click -> Properites and check the 'Unblock' box, at which point you can simply open the CHM file in windows explorer and you can find the article (if you see the index but not the article you did not unblock it. Vance