I wrote a simple test application in C# WinForms (Visual Studio 2015) called RSWPA to understand how to use the WPA. The New Thread Stack column cannot display the symbols defined in my application (just a simple exe, no dlls). All the PDBs are loaded including RSWPA.pdb: but in the New Thread Stack column, I got: I'm using WPA version 11.1.5.2 The SymCache folder contains the correct version and I can see my symbols in the file I found on the internet several similar questions but without an answer/solution. Can somebody help? Thanks Razvan

Hello Razvan, This picture of a WPA Analysis View shows why there are no symbolic names for your application: RSWPA.exe did not generate any of the DotNET events needed to map native addresses to IL addresses and .NET classes/methods. RSWPA.vshost.exe did produce the right type of events, but that is not your application (RSWPA.exe). So, I would say that WPA is working correctly. The mystery is why no DotNETRuntime events were recorded for the application but that is (potentially) a DotNETRuntime issue and not a WPA problem. I see that the application is running as a 32 bit application under WOW. I am currently trying to reproduce that behaviour (no DotNETRuntime events for a process when DotNETRuntime tracing is enabled). My suspicion is that RSWPA.exe is "suspended" by a debugger at the time that the events would have been generated and is therefore not able to participate. UPDATE: I am now fairly sure that RSWPA.exe is "suspended" by a debugger and is not able to participate in DotNET Runtime Rundown event generation. RSWPA.exe may well have run to completion and is possibly stopped at the final (intrinsic) debugger breakpoint (e.g. ntdll!NtTerminateProcess+0x14 or wow64cpu!CpupSyscallStub+0x13). Gary

WPA - Symbols from my application PDB not shown in the New Thread Stack (?!?)

Razvan Sirbu 0

I wrote a simple test application in C# WinForms (Visual Studio 2015) called RSWPA to understand how to use the WPA.

The New Thread Stack column cannot display the symbols defined in my application (just a simple exe, no dlls).

All the PDBs are loaded including RSWPA.pdb:

User's image

but in the New Thread Stack column, I got:

User's image

I'm using WPA version 11.1.5.2

The SymCache folder contains the correct version and I can see my symbols in the file

User's image

I found on the internet several similar questions but without an answer/solution.

Can somebody help?

Thanks

Razvan

Gary Nebbett 5,721 Reputation points

2023-08-16T19:03:12.55+00:00

Hello Razvan,

Have you included events from the Microsoft-Windows-DotNETRuntime and Microsoft-Windows-DotNETRuntimeRundown providers in your trace?

Gary
Razvan Sirbu 0 Reputation points

2023-08-17T15:20:02.43+00:00

Hello Gary,

I'm using the build-in providers and can see them in the list.

Additionally, I created a new etl using PerfView and I could see my symbols in the PerfView app. For this reason I assume the my PDB was created correctly and it is a "configuration"/app problem with WPRUI/WPA
Gary Nebbett 5,721 Reputation points

2023-08-17T15:54:53.09+00:00

Hello Razvan,

The .pdb file contains information about the Intermediate Language (IL) representation of the executable; the .NET runtime JIT events are needed to relate native addresses to IL addresses (and symbolic names).

If you can share a .etl file with the problem then I would take a look at it.

Gary
Razvan Sirbu 0 Reputation points

2023-08-17T18:22:41.6966667+00:00

Here it is:

https://1drv.ms/f/s!Ak_Vbf_CIg3nmx3hRbjeDsrtVrld?e=Mcadkh

and thanks for your support

1 answer

Gary Nebbett 5,721 Reputation points

2023-08-18T07:30:06.7633333+00:00

Hello Razvan,

This picture of a WPA Analysis View shows why there are no symbolic names for your application:

RSWPA.exe did not generate any of the DotNET events needed to map native addresses to IL addresses and .NET classes/methods. RSWPA.vshost.exe did produce the right type of events, but that is not your application (RSWPA.exe).

So, I would say that WPA is working correctly. The mystery is why no DotNETRuntime events were recorded for the application but that is (potentially) a DotNETRuntime issue and not a WPA problem. I see that the application is running as a 32 bit application under WOW.

I am currently trying to reproduce that behaviour (no DotNETRuntime events for a process when DotNETRuntime tracing is enabled). My suspicion is that RSWPA.exe is "suspended" by a debugger at the time that the events would have been generated and is therefore not able to participate.

UPDATE:

I am now fairly sure that RSWPA.exe is "suspended" by a debugger and is not able to participate in DotNET Runtime Rundown event generation. RSWPA.exe may well have run to completion and is possibly stopped at the final (intrinsic) debugger breakpoint (e.g. ntdll!NtTerminateProcess+0x14 or wow64cpu!CpupSyscallStub+0x13).

Gary
Please sign in to rate this answer.
Razvan Sirbu 0 Reputation points

2023-08-18T12:34:53.3966667+00:00

Hello Gary,

You are correct, I had VS project opened when I traced the RSWPA.exe launched directly from File Explorer. Please check the RSWPA.exe process (the RSWPA.vshost.exe was suspended or whatever - launched by VS). For RSWPA.exe process there are signs of activity.

Razvan

Gary Nebbett 5,721 Reputation points

2023-08-18T12:51:37.0433333+00:00

Hello Razvan,

There are no signs of activity for RSWPA.exe 34 to 35 seconds into the trace - the time at which the attempt to collect rundown events was started, but the process was still "present" until the end of the trace.

Gary

Razvan Sirbu 0 Reputation points

2023-08-21T19:52:16.27+00:00

Hello Gary,

I still have problems, although I made some progress.

I installed the WindowsInsiderPreviewADKen-us25931

I created a new simple C# WinForms application called AAA.exe

using this version when I trace in memory

(selecting only First level triage) the pdb of my application will not be processed

but if I record the trace in File, the pdb is loaded:

The only problem is that now (using File tracing) I lost a lot of events and the symbols for System.windows.Forms.ni cannot be loaded but I'm able (for the first time) to see my function calls in the New thread Stack.

I recorded both traces without changing the AAA.exe or the AAA.pdb

Another noticeable difference is the Generic Events:

(File trace)

(Memory trace)

where are much fewer events.

Here is the link for the files:

https://1drv.ms/f/s!Ak_Vbf_CIg3nmyQEbT9D9w5vpaNy?e=DYFZjZ

The questions are:

a. why the PDB cannot be loaded when profiling in memory

b. what are the differences between profiling in memory/file from loading/processing PDB point of view.

Thanks a lot,

Razvan

Gary Nebbett 5,721 Reputation points

2023-08-22T11:40:34.6866667+00:00

Hello Razvan,

There are not many differences between profiling to memory or to file (apart from the obvious (ensuring that all events from start to end are saved to file)).

WPR(UI) often specifies different buffer counts for the memory and file profiles (40 to 80 times more buffering for the GeneralProfile) but, by editing profiles, one can control the amount of buffering.

Profiling to memory is a circular operation, so the first events written to a memory trace will be the first to be overwritten. Startup "rundown" events are amongst the first to be written and the first to be overwritten.

There are at least four types of event that can be used to map native addresses to .NET symbolic names: startup rundown method loads, live method loads, live method unloads and rundown method unloads.

For some (unknown) reason, there are no .NET events of any type for RSWPA and AAA in your memory traces. When I run a .NET application with "target" "x86" (32 bit) then I still get .NET traces. Can you share the binary executable of AAA.exe?

Gary

Razvan Sirbu 0 Reputation points

2023-08-22T13:08:49.5566667+00:00

Hello Gary,

I added the AAA.zip file to the share drive:

https://1drv.ms/f/s!Ak_Vbf_CIg3nmyQEbT9D9w5vpaNy?e=LThxh7

It contains the VS solution.

Thanks,

Razvan

Gary Nebbett 5,721 Reputation points

2023-08-23T08:37:01.2066667+00:00

Hello Razvan,

I always get the expected DotNET events needed to map native addresses to method names, even when running your binary.

I am trying to compare the stack traces in your trace files with similar data from my PC, but the fact that your application is running under WoW (32 bits) is making this difficult.

Could you modify your project to target the platform "anycpu" or "x64" and send a new "problem" trace? I only need the .etl file (none of the additional files/folders).

Gary

Razvan Sirbu 0 Reputation points

2023-08-23T12:53:02.1866667+00:00

Hello Gary,

I uploaded the RAZVAN-PC.08-23-2023.08-23-10_ANYCPU.zip

to the

https://1drv.ms/f/s!Ak_Vbf_CIg3nmyQEbT9D9w5vpaNy?e=9OmBhe

Additionally, I added the WPA - Diagnostic Console Xperf log

This is my development machine, where, over time, I install/validate different tools.

Is it possible for some of these tools to "steal" the events?

I have installed the https://memprofiler.com/

,Concurrency Visualizer for Visual Studio 2015 and

Performance Monitor

Thanks again,

Razvan

Gary Nebbett 5,721 Reputation points

2023-08-23T14:34:19.97+00:00

Hello Razvan,

Nothing is impossible and I think that we are searching for something odd here.

It is possible to request a stack trace whenever a DotNET event is generated. Here is such a stack trace from my system:

0x7FFA2A8CF904=ntdll!NtTraceEvent+0x14 0x7FFA2A85FD45=ntdll!EtwpEventWriteFull+0x14D 0x7FFA2A85FBE9=ntdll!EtwEventWrite+0x29 0x7FFA0B400AA9=clr!CoTemplate_xxqqzzhjqzjqz+0x1DD 0x7FFA0B3FE98E=clr!ETW::LoaderLog::SendModuleEvent+0x48A 0x7FFA0B3FB2F2=clr!ETW::LoaderLog::ModuleLoad+0x1A6 0x7FFA0B0C22B0=clr!DomainAssembly::DeliverSyncEvents+0x20 0x7FFA0B071CB9=clr!DomainFile::DoIncrementalLoad+0xD1 0x7FFA0B0718A4=clr!AppDomain::TryIncrementalLoad+0x104 0x7FFA0B0715E4=clr!AppDomain::LoadDomainFile+0x154 0x7FFA0B0182B9=clr!AppDomain::LoadDomainFile+0xE1 0x7FFA0B111BF8=clr!DomainFile::EnsureLoadLevel+0x38 0x7FFA0AFF7CBB=clr!SystemDomain::Init+0xDB 0x7FFA0B0C30DC=clr!EEStartupHelper+0x78C 0x7FFA0B15D75D=clr!EEStartup+0x15 0x7FFA0B0A1B29=clr!EnsureEEStarted+0x115 0x7FFA0AFF9745=clr!_CorExeMainInternal+0x71 0x7FFA0AFF9B64=clr!CorExeMain+0x14 0x7FFA0BB1D6EA=mscoreei!CorExeMain+0xFA

I chose this event because interval sampling in your trace captured a similar stack:

0x7FFEFDC3EB82=clr!ETW::LoaderLog::SendModuleRange+0xCE 0x7FFEFDC3E9A0=clr!ETW::LoaderLog::SendModuleEvent+0x49C 0x7FFEFDC3B2F2=clr!ETW::LoaderLog::ModuleLoad+0x1A6 0x7FFEFD9022B0=clr!DomainAssembly::DeliverSyncEvents+0x20 0x7FFEFD8B1CB9=clr!DomainFile::DoIncrementalLoad+0xD1 0x7FFEFD8B18A4=clr!AppDomain::TryIncrementalLoad+0x104 0x7FFEFD8B15E4=clr!AppDomain::LoadDomainFile+0x154 0x7FFEFD8582B9=clr!AppDomain::LoadDomainFile+0xE1 0x7FFEFD951BF8=clr!DomainFile::EnsureLoadLevel+0x38 0x7FFEFD837CBB=clr!SystemDomain::Init+0xDB 0x7FFEFD9030DC=clr!EEStartupHelper+0x78C 0x7FFEFD99D75D=clr!EEStartup+0x15 0x7FFEFD8E1B29=clr!EnsureEEStarted+0x115 0x7FFEFD839745=clr!_CorExeMainInternal+0x71 0x7FFEFD839B64=clr!CorExeMain+0x14

This indicates to me that much of the "machinery" of DotNET tracing is working, but somehow no event is being added to the trace (there are no DotNET events of any type for the AAA.exe process).

Something that you could try is to start a WPR trace session and then start AAA.exe under a debugger, setting a breakpoint at clr!CoTemplate_xxqqzzhjqzjqz; once there then set a new breakpoint at ntdll!NtTraceEvent and then examine the call stack if and when you hit the new breakpoint.

Gary

Razvan Sirbu 0 Reputation points

2023-08-23T15:25:40.4333333+00:00

Hello Gary,

I don't know how to set these breakpoints.

Gary Nebbett 5,721 Reputation points

2023-08-23T16:59:34.6333333+00:00

Hello Razvan,

If your are willing, try the following:

Save the attached file as razvan.wprp

Issue the command wpr -start razvan.wprp!Bespoke & AAA & ping -n 2 localhost & wpr -stop why.etl -skipPdbGen making any changes necessary to allow for paths, etc.

Share the file why.etl

The "ping -n 2 localhost" is just to give the program AAA enough time to start; the commands are run quickly in sequence by use of the "&" operator so that the trace is kept as short as practicable.

Gary

razvan.wprp.txt

Razvan Sirbu 0 Reputation points

2023-08-23T18:28:42.26+00:00

Hello Gary,

here it is:

https://1drv.ms/f/s!Ak_Vbf_CIg3nmyQEbT9D9w5vpaNy?e=inRsba

Razvan

Gary Nebbett 5,721 Reputation points

2023-08-23T19:13:33.9833333+00:00

Hello Razvan,

As I feared, when one tries to pin down what is not working then everything works!

I have attached a new .wprp file that should be useable by you to profile AAA with "symbolic" names for the .NET methods in the stack traces.

What I now suggest is the following:

Verify that razvan.wprp gives a usable trace file with symbolic names for AAA methods.

Export the built-in GeneralProfile to a .wprp file (wpr -help profiles, wpr -exportprofile GeneralProfile gp.wprp)

See whether using the GeneralProfile from the command line (with wpr) works.

If it does not work, "simplify" it (step-by-step) and see whether any change enables it to work. razvan.wprp shows which elements of GeneralProfile are essential.

Let me/us know how that goes.

Gary

razvan.wprp.txt

Razvan Sirbu 0 Reputation points

2023-08-23T19:27:53.3533333+00:00

Hello Gary,

I'll try it, but I was looking at the original WHY.etl and I found this:

Can be the version .NET 4.8.1 for these two dlls a problem?

Note: I had installed the development .NET 4.8.1 on my machine, but I uninstalled it. Still, these two are from 4.8.1

Razvan

Gary Nebbett 5,721 Reputation points

2023-08-23T20:17:15.9266667+00:00

Hello Razvan,

I have the same .NET 4.8.1 DLLs on my system and your .NET 4.8.1 DLLs worked in the razvan.wprp test.

We also have your information, from the original message, that DotNET events were produced in the "File" mode but not in the "Memory" mode. The .NET version would have been the same for both tests/traces (I presume).

It would certainly be worth verifying whether there is any correlation between your results (.NET symbolic information available or not) and exact .NET DLL versions.

We need to control for all factors that might differ between success and failure.

Gary

Razvan Sirbu 0 Reputation points

2023-08-23T20:31:19.8833333+00:00

Hello Gary,

I added the new razvan.wprp file to the custom measurements (Add profiles) in WPRUI, and selected the First Level Triage and CPU usage.

I recorded a trace, and I can see my functions in the stack! But:

I cannot see the symbols from System.Windows.Forms.ni

(before I tried to load the symbols, I deleted all related files/versions related to System.Windows.Forms.ni from Symcache and C:\Symbols)

also, related to my previous comment, I found:

the message in the symbol Hub is misleading:

Do you have any ideas why :

a. now (using Razvan.wprp) I'm able to find symbols,

b. cannot load the symbols from System.Windows.Forms.ni

c. Is it okay to use some of the dll from .NET 4.8.1 (when I'm forcing in VS to use 4.6.2)

Thanks a lot,

Razvan

Razvan Sirbu 0 Reputation points

2023-08-23T20:41:05.9066667+00:00

Hello Gary,

Regarding the events being generated when use FILE mode, I discovered yesterday something:

I tried several times to use the file mode and when I did lose "a lot of events" I was able to see the symbols from my app. It happened once or twice to record it without losing events (still in file mode) and I lost by symbols... At that moment I was assuming that the "losing events" somehow triggered the symbols to be loaded/processed. I'm hoping it is not a red herring.

Gary Nebbett 5,721 Reputation points

2023-08-23T21:12:51.53+00:00

Hello Razvan,

As I have said, the .pdb file for AAA is not needed/used to convert native addresses to symbolic names - it does not contain that type of information.

For NGEN DLLs, the symbols are typically not downloaded from the Microsoft symbol server but rather generated locally - see https://learn.microsoft.com/en-us/windows-hardware/test/wpt/using-clr-40-ngen-pdb-support.

Gary

Razvan Sirbu 0 Reputation points

2023-08-24T15:04:24+00:00

Hello Gary,

Thanks for the support. I'm not sure yet if everything is working fine - I finally have the symbols loaded - but I started to read and try to understand on how to use WPA to analyze the critical path for my "real" application.

Updates:

I'm able to load symbols for my real app (multi-threaded app, multi-dll) - although in the Symbols Hub, I still see some of my DLL as unprocessed.

I got it on how the NGEN DLL works - thanks.

I'll continue to use WPA - and if I cannot find the answers on the internet, I'll post a new question here.

Question:

How can I "activate" the UI Delay chart? I cannot find it under the System Activities group.

What is the best documentation/example for Critical Path analysis? I found some videos and WPA documentation (e.g. https://learn.microsoft.com/en-us/shows/defrag-tools/156-critical-path-analysis-windows-performance-analyzer). Do you have other suggestions?

Thanks again,

Razvan

Gary Nebbett 5,721 Reputation points

2023-08-24T15:48:54.05+00:00

Hello Razvan,

The various charts are typically only available in the UI if corresponding events are present in the trace file.

I don't have any general advice on performance analysis.

Gary
Sign in to comment