Background and documented behavior

One can switch GCLatencyMode in order to adjust the GC behavior. According to this doc SustainedLowLatency mode should ensure that full blocking collections are only made under low memory pressure:

(...) The collector tries to perform only generation 0, generation 1, and concurrent generation 2 collections.

Full blocking collections may still occur if the system is under memory pressure

One can subscribe to Full Blocking GC notifications by using Garbage Collection Notifications.

According to CLR via C# book (and you can verify that through debugger like demonstrated in this post) GC subscribes to low memory notifications through usage of CreateMemoryResourceNotification API.

Repro Code

I put together code (by getting and adjusting code sample from mentioned Garbage Collection Notifications msdn doc) that lets you select the GCLatencyMode, runs allocations and display number of performed allocations, number of performed Gen 2 Collections (and how many are blocking) + info about low system memory notification:

namespace GCExperiments

{
    using System;
    using System.Collections.Generic;
    using System.Runtime;
    using System.Runtime.InteropServices;
    using System.Threading;

    public static class Win32Imports
    {
        public static int LowMemoryResourceNotification = 0;

        [DllImport("kernel32.dll", SetLastError = true)]
        public static extern IntPtr CreateMemoryResourceNotification(int notificationType);

        [DllImport("kernel32.dll", SetLastError = true)]
        public static extern bool QueryMemoryResourceNotification(IntPtr hResNotification, out bool isResourceStateMet);
    }

    public class GCModesTest
    {
        public static void Main(string[] args)
        {
            GCModesTest t = new GCModesTest();
            t.Run();
        }

        private List<byte[]> _load = new List<byte[]>();
        private IntPtr _hResNotification;

        public void Run()
        {
            Console.WriteLine(
                "Press 'I' for test under (I)nteractive GC mode, press 'S' for test under (S)ustainableLowLatency mode and 'L' for test under (L)owLatency mode");

            GCLatencyMode requestedMode;

            switch (Console.ReadKey(true).Key)
            {
                case ConsoleKey.I:
                    requestedMode = GCLatencyMode.Interactive;
                    break;
                case ConsoleKey.S:
                    requestedMode = GCLatencyMode.SustainedLowLatency;
                    break;
                case ConsoleKey.L:
                    requestedMode = GCLatencyMode.LowLatency;
                    break;
                default:
                    Log("Unknown option, exiting");
                    return;
            }

            GCSettings.LatencyMode = requestedMode;
            Console.WriteLine("Current GC mode: {0}, Is server GC: {1}",
                              GCSettings.LatencyMode, GCSettings.IsServerGC);

            _hResNotification = Win32Imports.CreateMemoryResourceNotification(Win32Imports.LowMemoryResourceNotification);

            GC.RegisterForFullGCNotification(1, 1);
            ThreadPool.QueueUserWorkItem(o => WaitForFullGCProc());

            RunAllocations();
        }

        private void RunAllocations()
        {
            try
            {
                int lastCollCount = 0;
                int newCollCount = 0;
                int allocations = 0;

                while (true)
                {
                    _load.Add(new byte[500]);
                    allocations++;
                    newCollCount = GC.CollectionCount(2);
                    if (newCollCount != lastCollCount)
                    {
                        // Show collection count when it increases:
                        Log("Allocations: {0}. Gen 2 collection count: {1}, from that blocking collections: {2}",
                            allocations, GC.CollectionCount(2), _gcFullollectionNotificationsCount);
                        lastCollCount = newCollCount;
                    }
                }
            }
            catch (OutOfMemoryException)
            {
                Console.WriteLine("Out of memory.");
            }
        }

        private int _gcFullollectionNotificationsCount = 0;

        private void WaitForFullGCProc()
        {
            // CheckForNotify is set to true and false in Main. 
            while (true)
            {
                // Check for a notification of an approaching collection.
                GCNotificationStatus s = GC.WaitForFullGCApproach();
                if (s == GCNotificationStatus.Succeeded)
                {
                    Log("GC Approach Notification raised.");
                }
                else if (s == GCNotificationStatus.Canceled)
                {
                    Log("GC Approach Notification cancelled.");
                    break;
                }
                else
                {
                    // This can occur if a timeout period 
                    // is specified for WaitForFullGCApproach(Timeout)  
                    // or WaitForFullGCComplete(Timeout)   
                    // and the time out period has elapsed. 
                    Log("GC Approach Notification not applicable.");
                    break;
                }

                // Check for a notification of a completed collection.
                s = GC.WaitForFullGCComplete();
                if (s == GCNotificationStatus.Succeeded)
                {
                    bool isResourceStateMet;
                    bool succeeded = Win32Imports.QueryMemoryResourceNotification(_hResNotification,
                                                                                  out isResourceStateMet);

                    if (!succeeded)
                    {
                        Log("Call to QueryMemoryResourceNotification failed!");
                    }

                    _gcFullollectionNotificationsCount++;
                    Log(
                        "GC Completed Notification raised. Count: {0}, Gc gen 2 count: {1}, Is Low memory condition: {2}",
                        _gcFullollectionNotificationsCount, GC.CollectionCount(2), isResourceStateMet);
                }
                else if (s == GCNotificationStatus.Canceled)
                {
                    Log("GC Completed Notification not applicable.");
                    break;
                }
                else
                {
                    // Could be a time out.
                    Log("GC Completed Notification not applicable.");
                    break;
                }
            }
        }

        private void Log(string format, params object[] args)
        {
            Console.WriteLine(format, args);
        }
    }

}

Observed behavior

When running this code you can see for yourself that under SustainedLowLatency mode the full blocking Gen2 collections are being performed despite the system low memory condition notification is not raised. In fact it runs approximately same number of Gen2 collection and with approximately same frequency as under the Interactive mode, with the difference that all those are blocking under SustainedLowLatency mode. That seems to be contrary to the mentioned documentation.

I have reported this issue to Microsoft and I'll update this post once I have some further info.

Sample outputs

Here goes sample output of provided code for Interactive GC mode:

Press 'I' for test under (I)nteractive GC mode, press 'S' for test under (S)ustainableLowLatency mode and 'L' for test under (L)owLatency mode
Current GC mode: Interactive, Is server GC: False
GC Approach Notification raised.
Allocations: 43050. Gen 2 collection count: 1, from that blocking collections: 0
GC Completed Notification raised. Count: 1, Gc gen 2 count: 1, Is Low memory condition: False
Allocations: 102612. Gen 2 collection count: 2, from that blocking collections: 1
Allocations: 209854. Gen 2 collection count: 3, from that blocking collections: 1
Allocations: 412419. Gen 2 collection count: 4, from that blocking collections: 1
Allocations: 769896. Gen 2 collection count: 5, from that blocking collections: 1
Allocations: 1413345. Gen 2 collection count: 6, from that blocking collections: 1
Allocations: 2640671. Gen 2 collection count: 7, from that blocking collections: 1
Allocations: 4940414. Gen 2 collection count: 8, from that blocking collections: 1
Allocations: 9230099. Gen 2 collection count: 9, from that blocking collections: 1
Allocations: 17166009. Gen 2 collection count: 10, from that blocking collections: 1
Allocations: 23969919. Gen 2 collection count: 11, from that blocking collections: 1
Allocations: 26800670. Gen 2 collection count: 12, from that blocking collections: 1
Allocations: 28591795. Gen 2 collection count: 13, from that blocking collections: 1

And here goes sample output for SustainedLowLatencyMode:

Press 'I' for test under (I)nteractive GC mode, press 'S' for test under (S)ustainableLowLatency mode and 'L' for test under (L)owLatency mode
Current GC mode: SustainedLowLatency, Is server GC: False
GC Approach Notification raised.
Allocations: 43038. Gen 2 collection count: 1, from that blocking collections: 0
GC Completed Notification raised. Count: 1, Gc gen 2 count: 1, Is Low memory condition: False
GC Approach Notification raised.
Allocations: 102584. Gen 2 collection count: 2, from that blocking collections: 1
GC Completed Notification raised. Count: 2, Gc gen 2 count: 2, Is Low memory condition: False
GC Approach Notification raised.
Allocations: 209795. Gen 2 collection count: 3, from that blocking collections: 2
GC Completed Notification raised. Count: 3, Gc gen 2 count: 3, Is Low memory condition: False
GC Approach Notification raised.
Allocations: 424242. Gen 2 collection count: 4, from that blocking collections: 3
GC Completed Notification raised. Count: 4, Gc gen 2 count: 4, Is Low memory condition: False
GC Approach Notification raised.
Allocations: 781680. Gen 2 collection count: 5, from that blocking collections: 4
GC Completed Notification raised. Count: 5, Gc gen 2 count: 5, Is Low memory condition: False
GC Approach Notification raised.
Allocations: 1437019. Gen 2 collection count: 6, from that blocking collections: 5
GC Completed Notification raised. Count: 6, Gc gen 2 count: 6, Is Low memory condition: False
GC Approach Notification raised.
Allocations: 2688148. Gen 2 collection count: 7, from that blocking collections: 6
GC Completed Notification raised. Count: 7, Gc gen 2 count: 7, Is Low memory condition: False
GC Approach Notification raised.
Allocations: 5035521. Gen 2 collection count: 8, from that blocking collections: 7
GC Completed Notification raised. Count: 8, Gc gen 2 count: 8, Is Low memory condition: False
GC Approach Notification raised.
Allocations: 9396673. Gen 2 collection count: 9, from that blocking collections: 8
GC Completed Notification raised. Count: 9, Gc gen 2 count: 9, Is Low memory condition: False
GC Approach Notification raised.
Allocations: 17463633. Gen 2 collection count: 10, from that blocking collections: 9
GC Completed Notification raised. Count: 10, Gc gen 2 count: 10, Is Low memory condition: False
GC Approach Notification raised.
Allocations: 19656130. Gen 2 collection count: 11, from that blocking collections: 10
GC Completed Notification raised. Count: 11, Gc gen 2 count: 11, Is Low memory condition: False
GC Approach Notification raised.
Allocations: 21157575. Gen 2 collection count: 12, from that blocking collections: 11
GC Completed Notification raised. Count: 12, Gc gen 2 count: 12, Is Low memory condition: True
Allocations: 22670954. Gen 2 collection count: 13, from that blocking collections: 12
GC Approach Notification raised.
Allocations: 24241691. Gen 2 collection count: 14, from that blocking collections: 12
GC Completed Notification raised. Count: 13, Gc gen 2 count: 14, Is Low memory condition: True
GC Approach Notification raised.
Allocations: 25856669. Gen 2 collection count: 15, from that blocking collections: 13

As you can see SustainedLowLatency mode is causing approximately same number of collections (despite it's far from low os memory pressure), but all of them are blocking.

Our usage of ZipFile.CreateFromDirectory and discovery of a bug

The ZipFile class has a very useful utility methods. We decided to use it for compressing our daily logs (which were running into dozens of GB in cumulative size). We basically zipped the previous days logs and if zipping operation finished successfully, we also deleted the original file:

private static void ArchiveDirectory(string target_dir)
{
	try
	{
		if (Directory.Exists(target_dir) && !File.Exists(target_dir + ".zip"))
		{
			System.IO.Compression.ZipFile.CreateFromDirectory(target_dir, target_dir + ".zip");
			DeleteDirectory(target_dir);
		}
	}
	catch (Exception)
	{ /* This is intentional as there is no place to log during log archival */ } 
}

Where DeleteDirectory just recursively deletes the original directory with all its content.

This worked great, until we started to cross a magic line of about 45GB of a original directory to be zipped and at the same time about 4GB size of a destination zip archive (the size of a destination zip seems to be more significant here). Nothing changed during zipping - ZipFile.CreateFromDirectory created a zip file without throwing any exception. Destination zip files seem to have appropriate sizes and they contain all the source files. Problems started when we wanted to access original log files - not all of them could be accessed. Build in windows zip files support failed (0x80004005 - Unspecified Error). Various third party archiving tools failed (complaining about the zip file being compressed). So we tried to call reciprocal method: ZipFile.ExtractToDirectory - it started to unpack the original zip - calming us down for a minute - but after unpacking those magic some 45GBs (or more probably 4GBs of a zipped folder) it threw:

System.IO.InvalidDataException: A local file header is corrupt.
at System.IO.Compression.ZipArchiveEntry.ThrowIfNotOpenable(Boolean needToUncompress, Boolean needToLoadIntoMemory)
at System.IO.Compression.ZipArchiveEntry.OpenInReadMode(Boolean checkOpenable)
at System.IO.Compression.ZipArchiveEntry.Open()
at System.IO.Compression.ZipFileExtensions.ExtractToFile(ZipArchiveEntry source, String destinationFileName, Boolean overwrite)
at System.IO.Compression.ZipFileExtensions.ExtractToDirectory(ZipArchive source, String destinationDirectoryName)
at System.IO.Compression.ZipFile.ExtractToDirectory(String sourceArchiveFileName, String destinationDirectoryName, Encoding entryNameEncoding)
at System.IO.Compression.ZipFile.ExtractToDirectory(String sourceArchiveFileName, String destinationDirectoryName)

Reproduction of a bug - you can try it yourself

This turned out to be a very significant problem for us (I'm building financial platform where missing logs is an unacceptable situation). So I quickly reported this to Microsoft and got some attention - what helped was probably also a very simple repro code, that can be used without any changes/configuration (you just need about 90GBs of a free space and about 30-60 minutes for running it):

namespace ZipFileTest
{
    using System;
    using System.Diagnostics;
    using System.IO;
    using System.Linq;
    using System.Text;

    class Program
    {
        static void Main(string[] args)
        {
            ZipFileTest zft = new ZipFileTest();
            zft.DoTest();
        }
    }

    public class ZipFileTest
    {
        const string LOGS_DIR = "logs";
        const string LOG_FILE_NAME_PREFIX = "dummyLog";
        private const string LOG_LINES_STRING = @"
Lorem ipsum dolor sit amet, consectetuer adipiscing elit. 
Nam cursus. Morbi ut mi. Nullam enim leo, egestas id, condimentum at, laoreet mattis, massa. 
Sed eleifend nonummy diam. 
Praesent mauris ante, elementum et, bibendum at, posuere sit amet, nibh. Duis tincidunt lectus quis dui viverra vestibulum. Suspendisse vulputate aliquam dui. Nulla elementum dui ut augue. Aliquam vehicula mi at mauris. Maecenas placerat, nisl at consequat rhoncus, sem nunc gravida justo, quis eleifend arcu velit quis lacus. Morbi magna magna, tincidunt a, mattis non, imperdiet vitae, tellus. Sed odio est, auctor ac, sollicitudin in, consequat vitae, orci. 
Fusce id felis. Vivamus sollicitudin metus eget eros.
Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. In posuere felis nec tortor. Pellentesque faucibus. Ut accumsan ultricies elit. 
Maecenas at justo id velit placerat molestie. Donec dictum lectus non odio. Cras a ante vitae enim iaculis aliquam. Mauris nunc quam, venenatis nec, euismod sit amet, egestas placerat, est. 
Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Cras id elit. Integer quis urna. Ut ante enim, dapibus malesuada, fringilla eu, condimentum quis, tellus. Aenean porttitor eros vel dolor. Donec convallis pede venenatis nibh. Duis quam. Nam eget lacus. Aliquam erat volutpat. Quisque dignissim congue leo.
Mauris vel lacus vitae felis vestibulum volutpat. Etiam est nunc, venenatis in, tristique eu, imperdiet ac, nisl. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. In iaculis facilisis massa. Etiam eu urna. Sed porta. Suspendisse quam leo, molestie sed, luctus quis, feugiat in, pede. Fusce tellus. Sed metus augue, convallis et, vehicula ut, pulvinar eu, ante. Integer orci tellus, tristique vitae, consequat nec, porta vel, lectus. Nulla sit amet diam. Duis non nunc. Nulla rhoncus dictum metus. Curabitur tristique mi condimentum orci. Phasellus pellentesque aliquam enim. Proin dui lectus, cursus eu, mattis laoreet, viverra sit amet, quam. 
Curabitur vel dolor ultrices ipsum dictum tristique. Praesent vitae lacus. Ut velit enim, vestibulum non, fermentum nec, hendrerit quis, leo. Pellentesque rutrum malesuada neque.
Nunc tempus felis vitae urna. Vivamus porttitor, neque at volutpat rutrum, purus nisi eleifend libero, a tempus libero lectus feugiat felis. Morbi diam mauris, viverra in, gravida eu, mattis in, ante. 
Morbi eget arcu. Morbi porta, libero id ullamcorper nonummy, nibh ligula pulvinar metus, eget consectetuer augue nisi quis lacus. 
Ut ac mi quis lacus mollis aliquam. Curabitur iaculis tempus eros. 
Curabitur vel mi sit amet magna malesuada ultrices. Ut nisi erat, fermentum vel, congue id, euismod in, elit. Fusce ultricies, orci ac feugiat suscipit, leo massa sodales velit, et scelerisque mi tortor at ipsum. Proin orci odio, commodo ac, gravida non, tristique vel, tellus. Pellentesque nibh libero, ultricies eu, sagittis non, mollis sed, justo. Praesent metus ipsum, pulvinar pulvinar, porta id, fringilla at, est.
Phasellus felis dolor, scelerisque a, tempus eget, lobortis id, libero. Donec scelerisque leo ac risus. Praesent sit amet est. In dictum, dolor eu dictum porttitor, enim felis viverra mi, eget luctus massa purus quis odio. Etiam nulla massa, pharetra facilisis, volutpat in, imperdiet sit amet, sem. 
Aliquam nec erat at purus cursus interdum. 
Vestibulum ligula augue, bibendum accumsan, vestibulum ut, commodo a, mi. Morbi ornare gravida elit. Integer congue, augue et malesuada iaculis, ipsum dui aliquet felis, at cursus magna nisl nec elit. Donec iaculis diam a nisi accumsan viverra. Duis sed tellus et tortor vestibulum gravida. Praesent elementum elit at tellus. Curabitur metus ipsum, luctus eu, malesuada ut, tincidunt sed, diam. Donec quis mi sed magna hendrerit accumsan. Suspendisse risus nibh, ultricies eu, volutpat non, condimentum hendrerit, augue. Etiam eleifend, metus vitae adipiscing semper, mauris ipsum iaculis elit, congue gravida elit mi egestas orci. Curabitur pede.
Maecenas aliquet velit vel turpis. Mauris neque metus, malesuada nec, ultricies sit amet, porttitor mattis, enim. In massa libero, interdum nec, interdum vel, blandit sed, nulla. In ullamcorper, est eget tempor cursus, neque mi consectetuer mi, a ultricies massa est sed nisl. Class aptent taciti sociosqu ad litora torquent per conubia nostra, per inceptos hymenaeos. Proin nulla arcu, nonummy luctus, dictum eget, fermentum et, lorem. Nunc porta convallis pede.";

        private string[] _loglines;
        private int _fileCnt = 0;
        Random _rnd = new Random((int)DateTime.Now.Ticks);

        public ZipFileTest()
        {
            _loglines = LOG_LINES_STRING.Split(new[] { Environment.NewLine }, StringSplitOptions.RemoveEmptyEntries);
        }

        private string CreateLogDirectory()
        {
            string basedir = System.IO.Path.GetDirectoryName(System.Reflection.Assembly.GetExecutingAssembly().Location);
            basedir = basedir ?? string.Empty;

            string targetDir = System.IO.Path.Combine(basedir, LOGS_DIR, DateTime.UtcNow.ToString("yyyy-MM-dd"), Process.GetCurrentProcess().Id.ToString());

            if (!Directory.Exists(targetDir))
            {
                Directory.CreateDirectory(targetDir);
            }

            return targetDir;
        }

        private string RandomString(int size)
        {
            StringBuilder builder = new StringBuilder();

            builder.Append(
                Enumerable.Repeat(0, size)
                          .Select(dummy => Convert.ToChar(Convert.ToInt32(Math.Floor(26*_rnd.NextDouble() + 65))))
                          .ToArray());

            return builder.ToString();
        }

        private void CreateNextLogFile(string logDir)
        {
            FileStream stream = File.Open(
                            System.IO.Path.Combine(logDir, string.Format("{0}_{1}.log", LOG_FILE_NAME_PREFIX, _fileCnt++)),
                            FileMode.Append,
                            FileAccess.Write,
                            FileShare.Read);

            for (int i = 0; i < 500000; i++)
            {
                string logLine = _loglines[_rnd.Next(_loglines.Length)];

                for (int j = 0; j < _rnd.Next(5); j++)
                {
                    logLine = logLine.Insert(_rnd.Next(logLine.Length), RandomString(_rnd.Next(30)));
                }

                byte[] logLineBytes = Encoding.Default.GetBytes(logLine);
                stream.Write(logLineBytes, 0, logLineBytes.Length);
            }

            stream.Close();
        }

        private string ZipLogFiles(string logDir)
        {
            string dirToZip = Directory.GetParent(logDir).FullName;
            System.IO.Compression.ZipFile.CreateFromDirectory(dirToZip, dirToZip + ".zip");
            return dirToZip + ".zip";
        }

        private void UnzipZippedLogs(string zipName, string logDir)
        {
            string dirToUnzip = Path.Combine(Directory.GetParent(logDir).Parent.FullName, "UnzippedLogs");
            System.IO.Compression.ZipFile.ExtractToDirectory(zipName, dirToUnzip);
        }

        public void DoTest()
        {
            string logDir = CreateLogDirectory();
            for (int i = 0; i < 700; i++)
            {
                CreateNextLogFile(logDir);
            }

            string zipName = ZipLogFiles(logDir);
            UnzipZippedLogs(zipName, logDir);
        }
    }
}

You need to reference System.IO.Compression.FileSystem assembly, so your section of csproj file with references will look this way:

  <ItemGroup>
    ...
    <Reference Include="System.IO.Compression.FileSystem" />
    ...
  </ItemGroup>

Do not pay to much attention to all the magic constants - they are tuned in a way so that you can reliably hit the issue with reasonable time and space runtime requirements for the repro.

When running this code the unzipping will throw above mentioned exception. The thing is that handling of this exception won't help you - archive is already corrupted during the zipping operation

Workaround and a fix

I have reported this issue to Microsoft and I'm in contact with owners of this code. They were able to find rootcause of the issue (my wild guess would be something with 32bit checksums or something like that - as this reproes with archives over 4GBs). It seems that there is no remedy for already archived folders (I'll try to find something myself - at least partial remedy, as we simply cannot miss logs), for newly created archives it's highly recommendable to split the original folder into multiple destination archives, each below 4GBs (you'll probably need to do some tuning to find the source sizes - as level of compression highly depends on patterns in your logs).

However it might happen to you that your IntelliTrace Events or IntelliTrace Calls windows are missing and you cannot find the menus for those any more (even if you’re sure that it was working the day ago and that you are really running VS Ultimate). What is causing this?

My IntelliTrace Events and IntelliTrace Calls Windows and Menus for those are missing

During your debugging session you open the Debug –> Windows menu and you see something like this:

No menu for IntelliTrace – despite you double-checked that it’s enabled in Debug –> Options –> IntelliTrace.

There is one more option that you need to check (and that would probably never came to your mind in connection with IntelliTrace) – and it’s SQL integrated debugging (in your project properties, Debug pane):

Checking this option will cause the IntelliTrace functionality to disappear.

Bringing the IntelliTrace functionality back

Once you disable SQL Server debugging (it’s off by default by the way), your IntelliTrace windows and menus will magically reappear!:

I have logged bug for this with Visual Studio team – so let’s hope this behavior will be history soon.

Symptoms

The crash dump did have following confusing stack trace:

0:001> k
Child-SP          RetAddr           Call Site
0000007b`cee4fe60 0000007b`d091e970 <Unloaded_myproductiondll.dll>+0x18361
0000007b`cee4fe68 00000000`00000000 0x7b`d091e970

The exception record didn't contain much more helpful info. The last value of instruction pointer was really pointing to address within our module (as we'll see later):

0:001> .ecxr
rax=0000000000000001 rbx=0000007bd091e970 rcx=000007feed8f33aa
rdx=0000000000000000 rsi=0000000000000001 rdi=0000000000000000
rip=000007fee6528361 rsp=0000007bcee4fe60 rbp=0000000000000000
r8=0000007bcee4fe28  r9=0000000000000000 r10=0000000000000000
r11=0000000000000246 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0         nv up ei pl nz na pe nc
cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010200
<Unloaded_myproductiondll.dll>+0x18361:
000007fe`e6528361 ??              ???

Investigation steps

First I tried to have a look what modules are in the address space and where does the instruction pointer point to:

0:001> lm
(…)

Unloaded modules:
000007fe`e6340000 000007fe`e6506000   msxml3.dll
000007fe`ec700000 000007fe`ec8f3000   WININET.dll
000007fe`ed520000 000007fe`ed65d000   urlmon.dll
000007fe`eb0c0000 000007fe`eb0c5000   Normaliz.dll
000007fe`eae70000 000007fe`eb0bc000   iertutil.dll

000007fe`e6510000 000007fe`e6548000 myproductiondll.dll 
000007fe`ed890000 000007fe`ed8e4000   WS2_32.dll
000007fe`ed370000 000007fe`ed379000   NSI.dll 

We can see that our module (myproductiondll) is in address space, last instruction pointer value is in it’s ranges. Also it’s not currently loaded. So we can try to reload it the standard way:

0:001> !sym noisy
noisy mode - symbol prompts on

0:001> .reload /f myproductiondll*

0:001> lmvm myproductiondll.dll
start             end                 module name

Unloaded modules:
000007fe`e6510000 000007fe`e6548000   myproductiondll.dll
    Timestamp: Fri Oct 14 12:00:00 2011 (4E98E6E2)
    Checksum:  0003E937
    ImageSize:  00038000

That didn’t work (as it doesn’t load modules that were already unloaded from address space). However verbose output of module list (lmvm) gives us little more info about the binary – including the timestamp. This should be enough to locate the actual binary on the release share location. Let’s point executable image path to this location (you can read more on executable image path in my blog post about setting up windows debugging):

0:001> .exepath+ \\foo\bar\Vxx.x\productionbinaries
DBGHELP: Symbol Search Path: srv*c:\mysymcache*http://symbolserver
Executable image search path is: \\foo\bar\Vxx.x\productionbinaries
Expanded Executable image search path is: \\foo\bar\Vxx.x\productionbinaries

The module output gave us the memory ranges where the module was loaded – we can compute size of the module and then force reload the module by supplying base address and size of the module:

0:001> ? 000007fe`e6548000 - 000007fe`e6510000
Evaluate expression: 229376 = 00000000`00038000

0:001> .reload myproductiondll.dll=000007fe`e6510000,00000000`00038000
DBGHELP: \\foo\bar\Vxx.x\productionbinaries\myproductiondll.dll - OK
DBGENG:  Partial symbol load found image \\foo\bar\Vxx.x\productionbinaries\myproductiondll.dll.
SYMSRV:  http://symbolserver/myproductiondll.pdb/F08FC95575DXXYXX9YYY8CDE0EAEBB581/file.ptr
SYMSRV:  myproductiondll.pdb from \\symbols\files: 2165760 bytes - copied         
DBGHELP: myproductiondll - private symbols & lines 
        c:\mysymcache\myproductiondll.pdb\F08FC95575DXXXXYXXF8CDE0EAEBB581\myproductiondll.pdb

Now once the module is fully loaded, we can have debugger to show us the stack trace once again:

0:001> k
SYMSRV:    No longer indexed.
SYMSRV:  c:\mysymcache\kernel32.pdb\492C273DC1724466A04XXYXX91ACC1722\kernel32.pdb not found
SYMSRV:  http://symbolserver/kernel32.pdb/492C273DC1724466A04XXYXX91ACC1722/kernel32.pdb not found
DBGHELP: kernel32.pdb - file not found
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for kernel32.dll - 
DBGHELP: kernel32 - export symbols
Child-SP          RetAddr           Call Site
0000007b`cee4fe60 000007fe`ed241668 myproductiondll!ThreadWriteFunc+0x145 [x:\foo\bar.cxx @ 118]
0000007b`cee4fe90 000007fe`ed906e5d kernel32!BaseThreadInitThunk+0x18
0000007b`cee4fec0 00000000`00000000 ntdll!RtlUserThreadStart+0x21

So now we have a caller (it’s the same dll – so no external fault by the way), also if symbols were properly source indexed (you can read more on source indexing in my blog post about setting up windows debugging), then we also have code location

Root cause

In this particular case, the dll has a worker thread that is being spawned during construction and which exits during destruction. So destructor code needs to synchronize with the worker thread and this was the attempt to do that:

(...)

        fOK = ::GetExitCodeThread
                (
                m_hWorkerThread,
                &dwRet
                );

        //
        //  If we get a STILL_ALIVE code back, we wait for worker to finish
        //

        if (STILL_ACTIVE == dwRet)
        {
            ::WaitForSingleObject
                    (
                    m_hWorkerExitEvent,
                    100000      // time out after 100 seconds
                    );
        }

(...)

Basically destructor is waiting on an event that the worker thread code is setting BEFORE exiting:

}
(...)

    if ((NULL != pLocalPub) &&
        (NULL != pLocalPub->m_hEntryExitEvent))
    {
        ::SetEvent
                (
                pLocalPub->m_hEntryExitEvent
                );
    }

    ::ExitThread(0);  <-------------------- thread can be swapped here and never exit 
                                                       before the destructor finishes!!!

    return 0;         <-------------------- location of the crash
}

This can became problematic when the destructor is automatically being called by loader when unloading the dll (e.g. when member of this dll is being used as global variable on dllmain level of a different dll, which is even more dangerous – I’ll try to create different blog post discussing how global loader lock in such a case prevents from proper synchronization). Worker thread can be swapped right after call to ExitThread and the destructor code might be able to finish and system loader might be able to finish it’s work and unload the dll. Then once the scheduling mechanism swaps the worker thread back it is already executing code from unloaded module.

Resolution

In this particular case the resolution is very easy – when waiting for thread to finish one should always wait on handle of the thread which is automatically signaled by the OS once the thread exits (instead of relying on the thread code to signal some handle during it’s cleanup logic).

More generically – one should carefully synchronize between the worker threads and make sure the reference counting is always properly accomplished so that module is not being unloaded until all outstanding worker threads are done with their work.

Sample code exposing the problem

Let’s use the following test code:

        static void TestStringBuilder(bool reuse)
        {
            StringBuilder sb = new StringBuilder();
            Stopwatch stopWatch = new Stopwatch();
            stopWatch.Start();
            for (int i = 0; i < 16384; i++)
            {
                for (uint j = 0; j < 32; j++)
                {
                    sb.Append("a string");
                }

                string dummy = sb.ToString();

                if (reuse)
                {
                    sb.Length = 0; //sb.Clear() behaves same way
                }
                else
                {
                    sb = new StringBuilder();
                }
            }
            stopWatch.Stop();
            Console.WriteLine("Total time: {0}", stopWatch.Elapsed);
        }

        static void TestStringBuilder2(bool reuse)
        {
            StringBuilder sb = new StringBuilder();
            Stopwatch stopWatch = new Stopwatch();
            stopWatch.Start();
            for (int i = 0; i < 16384; i++)
            {
                for (uint j = 0; j < 32; j++)
                {
                    sb.Insert(0, "a string");
                }

                string dummy = sb.ToString();

                if (reuse)
                {
                    sb.Length = 0; //sb.Clear() behaves same way
                }
                else
                {
                    sb = new StringBuilder();
                }
            }
            stopWatch.Stop();
            Console.WriteLine("Total time: {0}", stopWatch.Elapsed);
        }


        static void Main(string[] args)
        {
            TestStringBuilder(true);
            TestStringBuilder(false);
            TestStringBuilder2(true);
            TestStringBuilder2(false);
        }

Now let’s try to build the code while targeting .NET 3.5 (or older) and then .NET 4.0 (or newer) and see what happens when we run it.

Output for .NET framework 3.5 (and below):

Total time: 00:00:00.0251184
Total time: 00:00:00.0294951
Total time: 00:00:00.0477691
Total time: 00:00:00.0535298

Output for .NET framework 4.0 (and higher):

Total time: 00:00:00.0167945
Total time: 00:00:00.0281705
Total time: 00:00:11.1285809
Total time: 00:00:00.0849541

Profiling results

Once you connect some .NET performance profiler (I used build-in performance profiler in VS 2012), you’ll notice that the vast majority of the time is spent inside the StringBuilder.set_Length property:

By using some .NET decompiler tool you’d find out that using StringBuilder.Clear() wouldn’t make any difference as this method just calls the setter of the Length property (so is effectively identical to the code we wrote).

Where is all that time spent?

That should always be our next question when finding similar behavior. We have few options of how to find this out:

• Run performance profiler in sampling mode (to gain more extensive data) while pointing it to Microsoft symbols server in order to get detailed stack information
• Inspect the code – by decompiling it by some .NET decompiler tool, or by pointing debugger to Microsoft symbol server and turning on source server usage.
• Indirectly profile (AKA trace - meaning in sampling mode) the process and collect various metrics (especially .NET performance and memory ones).

By first two approaches we would find out that StringBuilder.set_Length code that is being executed is actually allocating a memory (and it seems to be mainly allocating memory on Large Objects Heap). By the third approach we would find the same indirectly. Let’s see how extensively is this code using memory and causing garbage collection:

We can see that the offending code is spending enormous amount of time inside the Garbage collection and that almost all memory being allocated is actually being allocated at Large Objects Heap. If we would compare this with behavior in .NET 3.5:

We would see more expectable results – reusing temporary variables between multiple iterations of small cycles causes garbage collection to do more work (as those local variables get promoted to higher generations and as a result garbage collection needs to collect and compact more generations in order to obtain enough memory), however it’s not so prohibitively expensive to do so.

Why is this happening?

If you would inspect code of StringBuilder in .NET framework 3.5 or earlier and compare it to StringBuilder implementation in .NET framework 4.0 or newer you would find out that previously the StringBuilder internal memory storage was implemented as one big array (this was done to optimize ToString operation which basically just gave the ‘pointer’ to the internal buffer), however this was changed to internally use linked list (in order to optimize appending/inserting operations).

The set_Length implementation then in case of shortening the StringBuilder tries to allocate chunk of memory to perform some consolidation of some nodes that should be left in string builder after shortening. However it fails to recognize that we’re trying to completely erase the internal buffer (as the Insert operation prepends the nodes to the list rather then appends – and set_Length logic doesn’t count on that). As a result set_Lenght will do some memory allocation and copying that will be actually thrown away right after that.

What’s the workaround for this?

As already mentioned – we shouldn’t generally reuse temporary local variables between short cycles with multiple iterations. That way we only cause those local variables to be promoted to higher generations and this will lead to more work for Garbage Collector (Unless we know that we will want the reuse the same variable multiple times even between different calls to the same function – then we might consider implementing some sort of Resource Pool design pattern). So we should simply use new StringBuilder in each iteration and this way we wouldn’t hit the problem.

Local live debugging

In the case of local live debugging we don’t have to much obstacles in our way (since we have access to the used .net framework binaries and data access layers – those I’ll discuss later). You should make sure that you are using same bitness of debugger and extension as is the bitness of the target application (.NET application are usually built architecture agnostic and then run as 32 bit processes on 32 bit OS and 64 bit processes on 64 bit OS).

If you would be about to use mismatching bitness of debugger to attach/start the managed process you might see the following error:

---------------------------
WinDbg:6.13.0014.1618 X86 
---------------------------
Could not create process '\\Foo\Bar\Baz.exe', Win32 error 0n50

The request is not supported.
---------------------------
OK   
---------------------------

SOS extension

Once you are in the debugger, broken in into the target process, you will want to load sos extension (name is a acronym for ‘Son Of Strike’ as the extension was developed as the next version of the extension named ‘Strike’). This extension is shipped with .NET framework (and also indexed on Microsoft symbols server which will come in handy for other scenarios). An old version (1.1) of this extension also ships with the debugger package – and this can actually get into your way.

For .NET framework 1.1 the SOS extension has all the information about .NET internals ‘baked-in’ itself (some of those information you would normally expect to be in the symbol files) – so all you needed was loading the SOS. From the .NET version 2.0 the definition of data structures was separated into separate binary – mscordacwks (sometimes called data access layer). When loading the SOS extension it will need to also load correct data access layer (for correct architecture and version of .NET framework targeted by the debugged application/dump). Another important change for our purposes was introduced in .NET framework 4.0 – the mscorwks.dll module was replaced with module clr.dll. I’ll show how all those facts affects the managed code debugging.

Loading SOS extension during live local debugging

During the live local debugging you have the luxury of knowing that you computer has the same version and architecture of .NET framework as what is the debugged target using (because they are on the same machine :-)). Same approach also applies when you are debugging remotely or postmortem but know for sure that you have same architecture and version on your computer as what was on the target computer.

In this case you know that your installation of .NET framework will also have appropriate version SOS extension and mscordacwks.dll binary. They will actually in the same folder as where is the core dll for .NET – mscorwks.dll (for .NET up to 4.0) or clr.dll (for .NET 4.0 and higher) in that case you can simply use loadby command that will load an extension from a location of another binary that is already loaded in the debugger (so you need to make sure those modules are already loaded, or force load the with ‘.load /f <module>’):

.loadby sos.dll mscorwks
.loadby sos.dll clr

Loading SOS extension during postmortem or remote debugging

If you would try previous commands on remote or postmortem debugging, you might happen to see the following error message:

The version of SOS does not match the version of CLR you are debugging.  Please
load the matching version of SOS for the version of CLR you are debugging.
CLR Version: 4.0.30319.19010
SOS Version: 4.0.30319.269
Failed to load data access DLL, 0x80004005
Verify that 1) you have a recent build of the debugger (6.2.14 or newer)
            2) the file mscordacwks.dll that matches your version of clr.dll is 
                in the version directory
            3) or, if you are debugging a dump file, verify that the file 
                mscordacwks___.dll is on your symbol path.
            4) you are debugging on the same architecture as the dump file.
                For example, an IA64 dump file must be debugged on an IA64
                machine.

You can also run the debugger command .cordll to control the debugger's
load of mscordacwks.dll.  .cordll -ve -u -l will do a verbose reload.
If that succeeds, the SOS command should work on retry.

If you are debugging a minidump, you need to make sure that your executable
path is pointing to clr.dll as well.

The message itself already contains few helpful tips. However if you meet this situation you might be able to easily resolve majority of those cases by leveraging the fact that .NET Framework debugging related binaries are being indexed on Microsoft symbol server – internal and also external (for more info or symbol server, setting symbols path and execution image path you can refer to the setup windows debugging blog post). Basically for remote or postmortem debugging of managed application/dump in windows debugger you should follow those simple steps:

1) Set your symbol path to Microsoft symbol server (internal or external)

2) Set your execution image path to Microsoft symbol server

3) Make sure that the core CLR binary (mscorwks.dll or clr.dll) is loaded, reload it (with .reload /f) if it’s not

4) Reload CLR debugging related binaries by using ‘.cordll –ve –u –l’ command (-ve for verbose, –u for unload, –l for load)

5) Load the SOS module from the core CLR binary location (as you’d do that for live debugging)

Here is the example debugging session demonstrating those steps:

Microsoft (R) Windows Debugger Version 6.13.0014.1618 X86
Copyright (c) Microsoft Corporation. All rights reserved.


Loading Dump File [\\XXXXXX\FooBar_121030_075015.dmp]
User Mini Dump File with Full Memory: Only application data is available

Comment: '
*** Procdump   -ma FooBar.exe
*** Manual dump'
Symbol search path is: YYYYYYYYY
Executable search path is: 
Windows 7 Version 7601 (Service Pack 1) MP (32 procs) Free x86 compatible
Product: Server, suite: TerminalServer DataCenter SingleUserTS
Machine Name:
Debug session time: Tue Oct 30 06:50:17.000 2012 (UTC - 8:00)
System Uptime: 16 days 19:11:53.254
Process Uptime: 0 days 23:42:59.000
................................................................
..................................................
Loading unloaded module list
.
Cannot Automatically load SOS
eax=00000000 ebx=00000000 ecx=00000000 edx=00000000 esi=000001ec edi=00000000
eip=7784f8b1 esp=0044f164 ebp=0044f1d0 iopl=0         nv up ei pl zr na pe nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000246
ntdll!ZwWaitForSingleObject+0x15:
7784f8b1 83c404          add     esp,4

0:000> !eeversion
No export eeversion found

0:000> .symfix+
0:000> .sympath
Symbol search path is: SRV*http://msdl.microsoft.com/download/symbols
Expanded Symbol search path is: srv*http://msdl.microsoft.com/download/symbols

0:000> .exepath SRV*http://msdl.microsoft.com/download/symbols
Executable image search path is: SRV*http://msdl.microsoft.com/download/symbols
Expanded Executable image search path is: srv*http://msdl.microsoft.com/download/symbols

0:000> lmvm mscorwks
start    end        module name
709f0000 70f9b000   mscorwks   (deferred)             
    Image path: C:\Windows\Microsoft.NET\Framework\v2.0.50727\mscorwks.dll
    Image name: mscorwks.dll
    Timestamp:        Sat Dec 24 22:48:13 2011 (4EF6C72D)
    CheckSum:         005B474B
    ImageSize:        005AB000
    File version:     2.0.50727.5456
    Product version:  2.0.50727.5456
    File flags:       0 (Mask 3F)
    File OS:          4 Unknown Win32
    File type:        2.0 Dll
    File date:        00000000.00000000
    Translations:     0409.04b0
    CompanyName:      Microsoft Corporation
    ProductName:      Microsoft® .NET Framework
    InternalName:     mscorwks.dll
    OriginalFilename: mscorwks.dll
    ProductVersion:   2.0.50727.5456
    FileVersion:      2.0.50727.5456 (Win7SP1GDR.050727-5400)
    FileDescription:  Microsoft .NET Runtime Common Language Runtime - WorkStation
    LegalCopyright:   © Microsoft Corporation.  All rights reserved.
    Comments:         Flavor=Retail
    
0:000> .reload /f mscorwks.dll

0:000> lmvm mscorwks
start    end        module name
709f0000 70f9b000   mscorwks   (private pdb symbols)  D:\Debuggers_32bit\sym\mscorwks.pdb\7139B75336C24F7CAA1DC4060608770D2\mscorwks.pdb
    Loaded symbol image file: mscorwks.dll
    Image path: C:\Windows\Microsoft.NET\Framework\v2.0.50727\mscorwks.dll
    Image name: mscorwks.dll
    Timestamp:        Sat Dec 24 22:48:13 2011 (4EF6C72D)
    CheckSum:         005B474B
    ImageSize:        005AB000
    File version:     2.0.50727.5456
    Product version:  2.0.50727.5456
    File flags:       0 (Mask 3F)
    File OS:          4 Unknown Win32
    File type:        2.0 Dll
    File date:        00000000.00000000
    Translations:     0409.04b0
    CompanyName:      Microsoft Corporation
    ProductName:      Microsoft® .NET Framework
    InternalName:     mscorwks.dll
    OriginalFilename: mscorwks.dll
    ProductVersion:   2.0.50727.5456
    FileVersion:      2.0.50727.5456 (Win7SP1GDR.050727-5400)
    FileDescription:  Microsoft .NET Runtime Common Language Runtime - WorkStation
    LegalCopyright:   © Microsoft Corporation.  All rights reserved.
    Comments:         Flavor=Retail
    
0:000> .cordll -ve -u -l
CLRDLL: C:\Windows\Microsoft.NET\Framework\v2.0.50727\mscordacwks.dll:2.0.50727.6387 f:0
doesn't match desired version 2.0.50727.5456 f:0
CLRDLL: Unable to find '' on the path
Cannot Automatically load SOS
CLRDLL: Loaded DLL D:\Debuggers_32bit\sym\mscordacwks_x86_x86_2.0.50727.5456.dll\4EF6C72D5ab000\mscordacwks_x86_x86_2.0.50727.5456.dll
CLR DLL status: Loaded DLL D:\Debuggers_32bit\sym\mscordacwks_x86_x86_2.0.50727.5456.dll\4EF6C72D5ab000\mscordacwks_x86_x86_2.0.50727.5456.dll

0:000> .loadby sos mscorwks

0:000> !eeversion
2.0.50727.5456 retail
Workstation mode
SOS Version: 2.0.50727.6387 retail build

This should help you resolve vast majority of issues with loading modules for managed debugging, there still might be some special cases where you can see issues.

Issues with loading SOS extension and related binaries

One of those can happen when the mscordacwks is not properly indexed on the symols server. Then ouw would see similar output after running .cordll:

3: kd> .cordll -ve -u -l
CLRDLL: C:\Windows\Microsoft.NET\Framework64\v4.0.30319\mscordacwks.dll:4.0.30319.269 f:8
doesn't match desired version 4.0.30319.19010 f:8
CLRDLL: Unable to find 'R?' on the path
CLRDLL: Unable to find 'mscordacwks_AMD64_AMD64_4.0.30319.19010.dll' on the path
CLRDLL: Unable to get version info for '\\ntdev.corp.microsoft.com\wincorerelease13\fbl_core1_kernel\9257.0.121021-1715\amd64fre\bin\indexes\..\netfx\core\win\microsoft.net\framework64\v4\mscordacwks_AMD64_AMD64_4.0.30319.19010.dll', Win32 error 0n87
CLRDLL: ERROR: Unable to load DLL mscordacwks_AMD64_AMD64_4.0.30319.19010.dll, Win32 error 0n87
CLR DLL status: ERROR: Unable to load DLL mscordacwks_AMD64_AMD64_4.0.30319.19010.dll, Win32 error 0n87

This can potentially happen to you inside Microsoft when working with specific build of .NET Framework that wasn’t properly indexed (or the index was already purged) – it should never happen outside of Microsoft. If this happens, contact CLR team, as they have their own share for the build binaries that might have longer data retention.

Other – most common – issue can happen when you accidentally load the improper version of sos (e.g. from the debugger package, by simply running ‘.load sos’ and picking some wrong version that is in the path). .chain command can help reveal those issues:

0:000> !eeversion
Doesn't work with 2.x
0:000> .chain
Extension DLL search Path:
    D:\Debuggers_32bit\WINXP;D:\Debuggers_32bit\winext;D:\Debuggers_32bit\winext\arcade;D:\Debuggers_32bit\pri;D:\Debuggers_32bit;D:\Debuggers_32bit\winext\arcade;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\Windows Kits\8.0\Windows Performance Toolkit\;C:\Program Files\Microsoft SQL Server\110\Tools\Binn\;C:\Program Files (x86)\Odd\;C:\Program Files (x86)\Microsoft SQL Server\110\Tools\Binn\ManagementStudio\;C:\Program Files (x86)\Microsoft SQL Server\110\Tools\Binn\;C:\Program Files (x86)\Microsoft SQL Server\110\DTS\Binn\;C:\Program Files\Microsoft SQL Server\110\DTS\Binn\;D:\Debuggers_64bit
Extension DLL chain:
    D:\Debuggers_32bit\clr10\sos: image 6.13.0014.1618, API 1.0.0, built Wed Oct 17 10:41:23 2012
        [path: D:\Debuggers_32bit\clr10\sos.dll]
    dbghelp: image 6.13.0014.1618, API 6.2.6, built Wed Oct 17 10:42:02 2012
        [path: D:\Debuggers_32bit\dbghelp.dll]
    ext: image 6.13.0014.1618, API 1.0.0, built Wed Oct 17 10:41:49 2012
        [path: D:\Debuggers_32bit\winext\ext.dll]
    exts: image 6.13.0014.1618, API 1.0.0, built Wed Oct 17 10:42:19 2012
        [path: D:\Debuggers_32bit\WINXP\exts.dll]
    uext: image 6.13.0014.1618, API 1.0.0, built Wed Oct 17 10:42:18 2012
        [path: D:\Debuggers_32bit\winext\uext.dll]
    ntsdexts: image 6.2.9255.0, API 1.0.0, built Wed Oct 17 10:42:28 2012
        [path: D:\Debuggers_32bit\WINXP\ntsdexts.dll]

Loading SOSEX extension

SOSEX is helpful extension that extends possibilities of SOS. You first need to have SOS loaded – as described above. Then you can simply download latest version of SOSEX from http://www.stevestechspot.com/ and load it into the debugger and use it:

0:000> .load D:\Tools\sosex\32bit\sosex
This dump has no SOSEX heap index.
The heap index makes searching for references and roots much faster.
To create a heap index, run !bhi
0:000> !sosex.help
SOSEX - Copyright 2007-2012 by Steve Johnson - http://www.stevestechspot.com/
To report bugs or offer feedback about SOSEX, please email sjjohnson@pobox.com
Quick Ref:
--------------------------------------------------
bhi       [filename]                                     BuildHeapIndex - Builds an index file for heap objects.
bpsc      (Deprecated.  Use !mbp instead)
chi                                                      ClearHeapIndex - Frees all resources used by the heap index and removes it from memory.
dlk       [-d]                                           Displays deadlocks between SyncBlocks and/or ReaderWriterLocks
dumpgen   <GenNum> [-free] [-stat] [-type <TYPE_NAME>]   Dumps the contents of the specified generation
                   [-nostrings]
finq      [GenNum] [-stat]                               Displays objects in the finalization queue
frq       [-stat]                                        Displays objects in the Freachable queue
gcgen     <ObjectAddr>                                   Displays the GC generation of the specified object
gch       [HandleType]...                                Lists all GCHandles, optionally filtered by specified handle types
help      [CommandName]                                  Display this screen or details about the specified command
lhi       [filename]                                     LoadHeapIndex - load the heap index into memory.
mbc       <SOSEX breakpoint ID | *>                      Clears the specified or all managed breakpoints
mbd       <SOSEX breakpoint ID | *>                      Disables the specified or all managed breakpoints
mbe       <SOSEX breakpoint ID | *>                      Enables the specified or all managed breakpoints
mbl       [SOSEX breakpoint ID]                          Prints the specified or all managed breakpoints
mbm       <Type/MethodFilter> [ILOffset] [Options]       Sets a managed breakpoint on methods matching the specified filter
mbp       <SourceFile> <nLineNum> [ColNum] [Options]     Sets a managed breakpoint at the specified source code location
mdso      [Options]                                      Dumps object references on the stack and in CPU registers in the current context
mdt       [TypeName | VarName | MT] [ADDR] [Options]     Displays the fields of an object or type, optionally recursively
mdv       [nFrameNum]                                    Displays arguments and locals for a managed frame
mfrag     [-stat] [-mt:<MT>]                             Reports free blocks, the type of object following the free block, and fragmentation statistics
mframe    [nFrameNum]                                    Displays or sets the current managed frame for the !mdt and !mdv commands
mgu       // TODO: Document
mk        [FrameCount] [-l] [-p] [-a]                    Prints a stack trace of managed and unmanaged frames
mln       [expression]                                   Displays the type of managed data located at the specified address or the current instruction pointer
mlocks    [-d]                                           Lists all managed lock objects and CriticalSections and their owning threads
mroot     <ObjectAddr> [-all]                            Displays GC roots for the specified object
mt        (no parameters)                                Steps into the managed method at the current position
mu        [address] [-s] [-il] [-n]                      Displays a disassembly around the current instruction with interleaved source, IL and asm code
muf       [MD Address | Code Address] [-s] [-il] [-n]    Displays a disassembly with interleaved source, IL and asm code
mwaits    [-d]                                           Lists all waiting threads and, if known, the locks they are waiting on
mx        <Filter String>                                Displays managed type/field/method names matching the specified filter string
refs      <ObjectAddr> [-target|-source]                 Displays all references from and to the specified object
rwlock    [ObjectAddr | -d]                              Displays all RWLocks or, if provided a RWLock address, details of the specified lock
sosexhelp [CommandName]                                  Display this screen or details about the specified command
strings   [ModuleAddress] [Options]                      Search the managed heap or a module for strings matching the specified criteria

ListGcHandles - See gch

Use !help <command> or !sosexhelp <command> for more details about each command.

That’s it today. Now you are introduced to setting up managed code debugging and next time I’ll try to post some real live interesting case scenario.

Creating a debugging session

First of all one must decide the way how he/she will be debugging:

• Live local user mode debugging
• Live remote user mode debugging
• Post-mortem (dump) debugging
• Kernel mode debugging (live, or dump)
• Live user mode debugging piped through kernel mode session
• Debugging live user mode process in kernel mode context

Lets have a quick look on each of those – how can we setup those, and when to choose which

Live local user mode debugging

Live local user mode debugging is most common in a typical development scenario. Easiest way how to set such a session is by breaking into the existing running process (in WinDbg use File –> Attach to a process; or by running <debugger> –p <PID> or <debugger> –pn <process name>), or by directly starting the process under the debugger (File –> Open Executable; or by running <debugger> <commandline>). But there are also some more interesting ways of creating live user mode session.

Have the OS start the process under the debugger whenever a particular process would be about to start. This can be done by setting the ‘debugger’ Executable Image global OS flag. There are few ways how to set this flag (direct registry edit, in kernel debugging session, command line), but the most intuitive is by using Global Flags UI editor (gflags.exe executable in Debuggers package):

Have the OS attach the debugger to the process with unhandled exception before it would be forcefully terminated. This OS behavior (applied to all native user mode processes) can be controlled by AeDebug registry key:

HKLM\Software\Microsoft\Windows NT\CurrentVersion\AeDebug

First add or edit the ‘Auto’ string value and set it to “1” for automatic attachment of debugger without prompt dialog to user. Then add or edit the ‘Debugger’ string value to the full path to the debugger (relative paths are not recommendable as OS starts the debugger under the context of the crashing app – so the working directory might vary). For more detailed description of this setting visit msdn. Same registry key can be also updated by using the –iae (install AeDebugger) or –iaec (install AeDebugger with commandline) switches of user mode windows debugger of your choice (cdb, ntsd, windbg).
To automatically attach to crashing managed applications edit:

HKLM\Software\Microsoft\.NETFramework

First add or edit DbgJITDebugLaunchSettings string value of “2” and then add or edit DbgManagedDebugger string value to full path to the debugger. For more information about this setting of managed application post mortem debugging visit msdn.

Attaching manually but noninvasively. In this mode the debugger can ‘read’ the state of the process (memory, registry …), however it cannot ‘write’. In order to be able to change process state – and so fully debug the process – Debug port member of the object representation of a process I kernel needs to point to a Debug object (kernel mode structure) which can be associated only with one debugger process at any point of time. Therefore only one debugger can invasively debug a user mode process. Noninvasive debugging doesn’t need to communicate through Debug port (as it only reads the state of the process) – so we can achieve scenario where we debug process with multiple debuggers (e.g. we want to issue few diagnostics commands from windbg, while we are debugging from Visual Studio). Also this way we don’t have to be worry about possibility of corrupting state of the debugged process. For noninvasive debugging use the ‘-pv’ switch:

<debugger> –pv –p <PID>
<debugger> –pv –pn <ProcessName>

Live remote user mode debugging

Once you have live user mode debugging set up you can share this session remotely via the TCP/IP or named pipes protocol (or also 1394 or COM transport protocols – but those are much more common for kernel mode debugging). Some of the techniques (with the exception of smart client debugging) can be used to share dump debugging session. There are several ways of doing this.

Starting debugging server

In your debugging session to be shared start the debugging server:

.server <protocol>:<parameters>[,IcfEnable]

where protocol is either ‘tcp’ or ‘npipe’ and paramaters are either ‘port=<port>’ or ‘pipe=<pipeName>’. IcfEnable is optional convenient parameter which will case debugger to create the needed firewall exceptions on your behalf. 
Alternatively you can set up the debugging server at the time of starting the debugger:

windbg –server tcp:port=55555,IcfEnable notepad.exe

To list the debuggers server in your session use the .servers commands (helpful for easy copy-paste sharing of the remote with the person expected to perform the remote debugging):

0:000> .servers
On the client, use any of these command lines
0: <debugger> -remote tcp:Port=12345,Server=JAKRIVAN-DEV

From the output it should be clear how to attach to the remote session.

Remember that if you are starting the debugger server, the debugging engine (dbgeng.dll) actually runs on the remote machine – and so also all the symbols and sources resolution or other file system operation are relative to the remote system (there are ways how to send the symbols to remote machine through debugging session, or how to have sources loaded on local machine – I’ll discus those later in sections on symbols and sources resolving)

Starting debugging proxy server and smart client debugger

In situations where we would want the debugging engine to run on the local machine (usually due to access to symbols and sources – typical scenario when debugging live issue from customer machine) Debugging Proxy is our friend. Debugging proxy started on the remote machine (needs to have the debug privilege) communicates with the remote debuggers via a low level protocol (memory reads, memory writes etc.) – and all the actual debugging logic is happening on the server side. To start the Debugging Proxy run on the remote machine:

dbgsrv.exe –t <transport>:<parameters>

To connect to the remote Debugger Server run:

<debugger> -premote <transport>:<parameters>,server=<remote server> 

You can connect to the actual process later or at the time of starting the debugger:

<debugger> –premote <transport>:<parameters>,server=<remote server> –p <remote process PID>

Debugging via Debugger Proxy is very similar technique to what the Visual Studio uses for native mode debugging and managed debugging in version 2012. Managed mode debugging in older versions of Visual Studio are more similar to the basic remote debugging (so symbols need to be available from remote machine …).

Piping console input output through remote.exe

Another way of remote debugging is by using the remote.exe executable – since this executable is just remotely redirecting input and output it is then impossible to use the debugger engine API or windbg rich user interface to interact with the debugger. Therefore this way is recommendable in a very few limited scenarios (like e.g. when you need to see the actual output of the debugger when it’s initializing itself and may be failing due to various reasons – like incorrect arguments, insufficient privileges or resources etc. – then the native transport mechanism (the debugger server) wouldn’t even start at all, but the remote.exe can already transfer the output). In those special scenarios it might be helpful to use both remoting mechanisms when starting the debugger:

remote.exe /s "<debugger> -server < transport>:<parameters>,IcfEnable <debugger parameters>" < pipename>

And to connect to such a remote server you can either use either the remote.exe:

remote.exe /c <server name> <pipename>

Or the debugger itself (which is preferable as then you have all API and windbg UI richness available):

<debugger> –remote <transport>:<parameters>,server=<server name>

One beauty of remote.exe is that it can redirect console input/output of an arbitrary application – so even though it’s usually mentioned in connection with debugging, you can effectively use it for task like redirecting diagnostic output of a remote automated task (compiling, installation …).

Piping console input and output through live kd session

This is technically a live user mode remote debugging technique, but due it’s specific behavior I’ll discuss it as a separate debugging technique.

Post-mortem debugging

In order to be able to debug the issue post-mortem, one needs to have a memory dump of process (or system) do debug. There are few ways how to take a memory dump:

Taking memory dump in the Task Manager – this is least recommendable way, since one cannot influence the information that will be included/excluded in the dump and also on 64bit systems Task Manager takes 64bit dump of 32bit processes which will unnecessary complicate the debugging.

Take the memory dump in live debugging session – we can use the .dump command for this. You can consult debuggers help for the detailed options, but I usually recommend taking minidump with all advanced options with overwriting possible existing file:

.dump /ma /o <dump path with extension>

Take the dump with one of the SysInternals tools – Process Explorer, or probably most preferably ProcDump (it can also take dumps based on several conditions like 1st & 2nd chance exceptions, application hangs etc.)

Kernel mode debugging

Kernel mode debugging is a very large topic – so I’ll probably dedicate some post to it in future. Now just let me not that you can either debug real physical machine – over the serial, usb or fire-wire cable or recently also over the TCP/IP (with the introduction of the kdnet) – more on setting up kernel debugging please see the msdn. You can also debug the virtual machine over the synthetic serial port. But probably the most convenient way how to try kernel debugging (as it doesn't require any setup on the target machine – so you mainly don’t need to reboot!) is LiveKd. From very simplistic point of view LiveKd basically opens a memory as a dump file and this way it can trick the kernel debugger to debug the local system. This is especially helpful when we need to debug local user mode process in kernel mode context (e.g. to troubleshoot synchronization issues).

Live user mode debugging piped through kernel mode session

There might be situations where local or remote user mode debugging is impossible – e.g. due to very limited resources (machine under load or stress test) or to debug early boot-up processes where no session is yet available to start the console debugger. For those cases one viable way of debugging is by setting up a post-mortem debugger (as described in Live local user mode debugging) and setting it to ‘ntsd –d’, e.g. by running:

ntsd –iaec -d

That way once the debugger starts it immediately redirects its input and output through kernel mode session (while rest of the OS freezes, unless there is some activity required by debugger – e.g. loading symbols, traversing memory etc.).

Because the kernel mode session needs to be set up and healthy and the user mode debugger just redirects it’s input and output through it – this approach combines disadvantages of kernel mode debugging (problems with maintaining infrastructure for debugging, keeping sessions in sync …) and redirecting input and output (no way of direct interacting with dbgeng.dll API, no advanced UI functionality of windbg) – and so it should be only used in cases where you have no other choice (early bootup processes debugging, processes on machine with limited resources).

Debugging live user mode process in the kernel mode context

In some cases the kernel mode debugging session can offer more information then simple usermode session. Let’s have a quick look on a example how can we debug usermode process (e.g. in livekd):

By !process command, we can query currently running processes (second parameter is a byte flag that influences the number of information spewed out):

  

0: kd> !process 0 0 cmd.exe
PROCESS fffffa8008136740
    SessionId: 1  Cid: 1564    Peb: 7f7d8dd8000  ParentCid: 0c94
    DirBase: 150382000  ObjectTable: fffff8a014733400  HandleCount: 119.
    Image: cmd.exe

PROCESS fffffa800b4c5940
    SessionId: 1  Cid: 12b4    Peb: 7f7d8a26000  ParentCid: 0c94
    DirBase: 3435b000  ObjectTable: fffff8a0123f0e80  HandleCount:  19.
    Image: cmd.exe

PROCESS fffffa8011f9d940
    SessionId: 1  Cid: 1440    Peb: 7f7d835f000  ParentCid: 0c94
    DirBase: c1209000  ObjectTable: 00000000  HandleCount:   0.
    Image: cmd.exe

0: kd> !process 0 0 visio.exe
PROCESS fffffa800d3666c0
    SessionId: 1  Cid: 1558    Peb: 7ea05000  ParentCid: 18cc
    DirBase: 4b4d2000  ObjectTable: fffff8a0133418c0  HandleCount: 222.
    Image: VISIO.EXE

0: kd> !process 0 f visio.exe
PROCESS fffffa800d3666c0    SessionId: 1  Cid: 1558    Peb: 7ea05000  ParentCid: 18cc
    DirBase: 4b4d2000  ObjectTable: fffff8a0133418c0  HandleCount: 222.
    Image: VISIO.EXE
    VadRoot fffffa800c1d30f0 Vads 216 Clone 0 Private 1531. Modified 240. Locked 0.
    DeviceMap fffff8a001c1cf70
    Token                             fffff8a015a59050
    ElapsedTime                       3 Days 01:38:29.603
    UserTime                          00:00:00.187
    KernelTime                        00:00:02.449
    QuotaPoolUsage[PagedPool]         607592
    QuotaPoolUsage[NonPagedPool]      27328
    Working Set Sizes (now,min,max)  (3892, 50, 345) (15568KB, 200KB, 1380KB)
    PeakWorkingSetSize                6260
    VirtualSize                       307 Mb
    PeakVirtualSize                   327 Mb
    PageFaultCount                    61551
    MemoryPriority                    BACKGROUND
    BasePriority                      8
    CommitCharge                      2627

        THREAD fffffa8006e6bb00  Cid 1558.1994  Teb: 000000007e8da000 Win32Thread: 0000000000000000 WAIT: (WrQueue) UserMode Non-Alertable
            fffffa8008c74b00  QueueObject
        Not impersonating
        DeviceMap                 fffff8a001c1cf70
        Owning Process            fffffa800d3666c0       Image:         VISIO.EXE
        Attached Process          N/A            Image:         N/A
        Wait Start TickCount      3710528        Ticks: 13765657 (2:11:39:05.625)
        Context Switch Count      5              IdealProcessor: 2
        UserTime                  00:00:00.000
        KernelTime                00:00:00.000
        Win32 Start Address 0x00000000728b14aa
        Stack Init fffff88007f2cdd0 Current fffff88007f2c7a0
        Base fffff88007f2d000 Limit fffff88007f27000 Call 0
        Priority 10 BasePriority 8 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
        Kernel stack not resident.
        Child-SP          RetAddr           Call Site
        fffff880`07f2c7e0 fffff801`98f2b99c nt!KiSwapContext+0x76
        (Inline Function) --------`-------- nt!KiSwapThread+0xf4 (Inline Function @ fffff801`98f2b99c)
        fffff880`07f2c920 fffff801`98f36ddb nt!KiCommitThreadWait+0x23c
        fffff880`07f2c9e0 fffff801`992ceb6c nt!KeRemoveQueueEx+0x26b
        fffff880`07f2ca90 fffff801`992adcb5 nt!IoRemoveIoCompletion+0x4c
        fffff880`07f2cb20 fffff801`98f00d53 nt!NtRemoveIoCompletion+0x135
        fffff880`07f2cbd0 00000000`76fe2ad2 nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ fffff880`07f2cc40)
        00000000`03deec98 00000000`76fe2847 0x76fe2ad2
        00000000`03deeca0 00000023`770e0e48 0x76fe2847
        00000000`03deeca8 00000000`00000023 0x23`770e0e48
        00000000`03deecb0 00000000`00000000 0x23

        THREAD fffffa8007fb5080  Cid 1558.168c  Teb: 000000007ea0a000 Win32Thread: 0000000000000000 WAIT: (UserRequest) UserMode Non-Alertable
            fffffa800bff8800  ProcessObject
        Not impersonating
        DeviceMap                 fffff8a001c1cf70
        Owning Process            fffffa800d3666c0       Image:         VISIO.EXE
        Attached Process          N/A            Image:         N/A
        Wait Start TickCount      3710528        Ticks: 13765657 (2:11:39:05.625)
        Context Switch Count      3              IdealProcessor: 2
        UserTime                  00:00:00.000
        KernelTime                00:00:00.000
        Win32 Start Address 0x000000007450d97d
        Stack Init fffff88006844dd0 Current fffff88006844900
        Base fffff88006845000 Limit fffff8800683f000 Call 0
        Priority 10 BasePriority 8 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
        Kernel stack not resident.
        Child-SP          RetAddr           Call Site
        fffff880`06844940 fffff801`98f2b99c nt!KiSwapContext+0x76
        (Inline Function) --------`-------- nt!KiSwapThread+0xf4 (Inline Function @ fffff801`98f2b99c)
        fffff880`06844a80 fffff801`98f27c1f nt!KiCommitThreadWait+0x23c
        fffff880`06844b40 fffff801`992c7df6 nt!KeWaitForSingleObject+0x1cf
        fffff880`06844bd0 fffff801`98f00d53 nt!NtWaitForSingleObject+0xb6
        fffff880`06844c40 00000000`76fe2ad2 nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ fffff880`06844c40)
        00000000`004ceb28 00000000`76fe2941 0x76fe2ad2
        00000000`004ceb30 00000000`770e2fac 0x76fe2941
        00000000`004ceb38 00000000`00000023 0x770e2fac
        00000000`004ceb40 00000000`00000202 0x23
        00000000`004ceb48 00000000`01fdfd68 0x202
        00000000`004ceb50 00000000`0000002b 0x1fdfd68
        00000000`004ceb58 00000000`00000000 0x2b

In output we can find many helpful diagnostic information (what’s the state of process, threads, what are they waiting on etc.). We can also switch contexts to particular process/thread (by using OS wide ids of process/thread from the informational outputs of !process command) and debug as we are used to in user mode debugging:

0: kd> .process fffffa800d3666c0
Implicit process is now fffffa80`0d3666c0        

0: kd> .thread fffffa8007fb5080Implicit thread is now fffffa80`07fb5080

0: kd> k
  *** Stack trace for last set context - .thread/.cxr resets it
Child-SP          RetAddr           Call Site
fffff880`06844940 fffff801`98f2b99c nt!KiSwapContext+0x76
(Inline Function) --------`-------- nt!KiSwapThread+0xf4
fffff880`06844a80 fffff801`98f27c1f nt!KiCommitThreadWait+0x23c
fffff880`06844b40 fffff801`992c7df6 nt!KeWaitForSingleObject+0x1cf
fffff880`06844bd0 fffff801`98f00d53 nt!NtWaitForSingleObject+0xb6
fffff880`06844c40 00000000`76fe2ad2 nt!KiSystemServiceCopyEnd+0x13
00000000`004ceb28 00000000`76fe2941 0x76fe2ad2
00000000`004ceb30 00000000`770e2fac 0x76fe2941
00000000`004ceb38 00000000`00000023 0x770e2fac
00000000`004ceb40 00000000`00000202 0x23
00000000`004ceb48 00000000`01fdfd68 0x202
00000000`004ceb50 00000000`0000002b 0x1fdfd68
00000000`004ceb58 00000000`00000000 0x2b

But in addition to user mode debugging we have also all kernel mode debugging commands and access to the whole address space (including other processes).

0: kd> !object fffffa800bff8800
Object: fffffa800bff8800  Type: (fffffa800675f750) Process
    ObjectHeader: fffffa800bff87d0 (new version)
    HandleCount: 6  PointerCount: 1621333

0: kd> !process fffffa800bff8800 0
PROCESS fffffa800bff8800
    SessionId: 1  Cid: 15f8    Peb: 7f722b67000  ParentCid: 1558
    DirBase: 143f0d000  ObjectTable: fffff8a0126addc0  HandleCount: 174.
    Image: splwow64.exe

So in this particular case we found that Viso process was waiting on MS spooler service – so shortly hanging Visio was tracked down to network glitches with our network printer.

However debugging in kernel mode debugger can be challenging for many developers and even extensions – many commands are different in kernel mode and some commands even mean different things in user mode and kernel mode – like e.g. the “~” command that can be used to switch threads in user mode, but is used for switching of processors in kernel mode – this caused !analyze extension to case weird behavior for a long time for user mode breaks in kernel mode context.

What debugging type is used in current session

With this many ways of debugging it can be sometimes confusing to find out what type of session are we actually using (especially during the remoting scenarios). There is one helpful command to find out the type of debugging for us:

0:000> ||
  
 .  0 Live user mode: <Local>

0: kd> ||
  
 .  0 64-bit Full kernel dump: C:\WINDOWS\livekd.dmp

This way we can for example see that LiveKd actually ‘just’ creates full dump of a memory and then starts debugger to debug that dump.

Preparing session for debugging

Now I’m finally getting to the actual preparation of the debugging session – this usually means resolving of symbols/binaries/sources and finding owner of code that should perform further investigation. During the kernel mode debugging the first step should be making sure that the session is properly synchronized with the target – but I’m not going to discus this step further in this post.

Ensuring symbols resolution

Without the debugging symbols (private preferably), the debugging engine must sometimes ‘guess’ and as a result we cannot fully trust it. It can often happen that location of the crash (AV, unhandled exception …) is pointed to a wrong function (sometimes even module) until all the symbols for modules on the particular stack are resolved. Debugger will usually tell us that missing symbols might cause it to ‘guess’ wrong:

0:001> k
ChildEBP RetAddr
WARNING: Stack unwind information not available. Following frames may be wrong.
004efe54 764b8543 ntdll!DbgBreakPoint
004efe60 770fac69 KERNEL32!BaseThreadInitThunk+0xe
004efea4 770fac3c ntdll!RtlInitializeExceptionChain+0x85
004efebc 00000000 ntdll!RtlInitializeExceptionChain+0x58

Usually we’ll have on the stack OS binaries, our binaries and sometimes also third party binaries. Debuggers have a built-in command for pointing symbol path to the Microsoft symbol server (to the external one for the shipping builds, and to the internal for the internal only builds):

3: kd> .symfix
3: kd> .sympath
Symbol search path is: srv*
Expanded Symbol search path is: SRV*http://msdl.microsoft.com/download/symbols

Ideally you’ll not need a symbols for the third part components (hopefully they are either reliable, or they are not in the middle of the stack when the crash happens in your component) – as it’s usually hard to get symbols from third parties.

For your symbols (and same would apply if you would happen to be able to get symbols for third party components) you will ideally have them on a symbol server, in that case you need to prepend path to your symbols server by ‘srv*’ – this indicates that the path will be searched using the ‘index’ (which is a guid that is baked into the binary and symbol file at the time of build – this guid also ensures that only the symbol file coming from the same build as binary will match that binary – unless forcefully chosen to load mismatched symbol file). If the ‘srv*’ is not prepended, the supplied path will be searched as a flat store – each directory will be examined (and potentiall ‘dll’, ‘sym’, ‘exe’ and other similar subdirectories in single level nesting will be also examined) to find the matching symbol file. This can be time consuming task – especially if accidentally applied to a big symbols server. This – incorrectly set up symbol search path - is actually one of the most common reasons for debugger hanging during some task (including Visual Studio debugger).

In the debugger you can set your symbol search path by .sympath command and following is the syntax of the symbol path:

.sympath <local flat symbol path>;cache*<local cache for remote symbol locations>;<remote flat symbol path>;
		srv*<remote symbols server>

For example:

.sympath C:\MyBuildLocation\Symbols;cache*C:\MySymbolsCache;\\CorporateSymbolsServer\Symbols;
		srv*http://msdl.microsoft.com/download/symbols

However instead of setting this path every time you start the debugger it’s much more recommendable to set your _NT_SYMBOL_PATH environment variable to that path. As a bonus many other applications can use this – like Process Explorer, Process Monitor etc.

Once your symbols are correctly set, you can try to reload them with .reload command (or you may need to force reloading with /f if stripped symbols were already loaded) – you can also point it to specific binaries (and use wildcards):

0:001> .reload
Reloading current modules
..........................
0:001> k
ChildEBP RetAddr
004efe24 7713dcbc ntdll!DbgBreakPoint
004efe54 764b8543 ntdll!DbgUiRemoteBreakin+0x39
004efe60 770fac69 KERNEL32!BaseThreadInitThunk+0xe
004efea4 770fac3c ntdll!__RtlUserThreadStart+0x72
004efebc 00000000 ntdll!_RtlUserThreadStart+0x1b

Troubleshooting symbols resolution

Let me simulate symbols resolution problems by setting wrong symbol path:

0:001> .sympath aaa
Symbol search path is: aaa
Expanded Symbol search path is: aaa
WARNING: Inaccessible path: 'aaa'
0:001> .reload /f
Reloading current modules
.*** ERROR: Module load completed but symbols could not be loaded for notepad.exe
.*** ERROR: Symbol file could not be found.  Defaulted to export symbols for C:\WINDOWS\SysWOW64\SHCORE.DLL -
.*** ERROR: Symbol file could not be found.  Defaulted to export symbols for C:\WINDOWS\SysWOW64\dwmapi.dll -
.*** ERROR: Symbol file could not be found.  Defaulted to export symbols for C:\WINDOWS\SysWOW64\uxtheme.dll -
.*** ERROR: Symbol file could not be found.  Defaulted to export symbols for C:\WINDOWS\SysWOW64\WINSPOOL.DRV -
.*** ERROR: Symbol file could not be found.  Defaulted to export symbols for C:\WINDOWS\SysWOW64\bcryptPrimitives.dll -
.*** ERROR: Symbol file could not be found.  Defaulted to export symbols for C:\WINDOWS\SysWOW64\CRYPTBASE.dll -
.*** ERROR: Symbol file could not be found.  Defaulted to export symbols for C:\WINDOWS\SysWOW64\SspiCli.dll -
.*** ERROR: Symbol file could not be found.  Defaulted to export symbols for C:\WINDOWS\SysWOW64\SHLWAPI.dll -
.*** ERROR: Symbol file could not be found.  Defaulted to export symbols for C:\WINDOWS\SysWOW64\combase.dll -
.*** ERROR: Symbol file could not be found.  Defaulted to export symbols for C:\WINDOWS\SysWOW64\GDI32.dll -
.*** ERROR: Symbol file could not be found.  Defaulted to export symbols for C:\WINDOWS\SysWOW64\msvcrt.dll -
.*** ERROR: Symbol file could not be found.  Defaulted to export symbols for C:\WINDOWS\SysWOW64\KERNELBASE.dll -
.*** ERROR: Symbol file could not be found.  Defaulted to export symbols for C:\WINDOWS\SysWOW64\SHELL32.dll -
.*** ERROR: Symbol file could not be found.  Defaulted to export symbols for C:\WINDOWS\SysWOW64\ole32.dll -
.*** ERROR: Symbol file could not be found.  Defaulted to export symbols for C:\WINDOWS\SysWOW64\OLEAUT32.dll -
.*** ERROR: Symbol file could not be found.  Defaulted to export symbols for C:\WINDOWS\SysWOW64\KERNEL32.DLL -
.*** ERROR: Symbol file could not be found.  Defaulted to export symbols for C:\WINDOWS\SysWOW64\COMDLG32.dll -
.*** ERROR: Symbol file could not be found.  Defaulted to export symbols for C:\WINDOWS\SysWOW64\sechost.dll -
.*** ERROR: Symbol file could not be found.  Defaulted to export symbols for C:\WINDOWS\SysWOW64\IMM32.DLL -
.*** ERROR: Symbol file could not be found.  Defaulted to export symbols for C:\WINDOWS\SysWOW64\RPCRT4.dll -
.*** ERROR: Symbol file could not be found.  Defaulted to export symbols for C:\WINDOWS\SysWOW64\MSCTF.dll -
.*** ERROR: Symbol file could not be found.  Defaulted to export symbols for C:\WINDOWS\SysWOW64\ADVAPI32.dll -
.*** ERROR: Symbol file could not be found.  Defaulted to export symbols for C:\WINDOWS\SysWOW64\USER32.dll -
.*** ERROR: Symbol file could not be found.  Defaulted to export symbols for ntdll.dll -

Already in the basic output we some hints that there are problems with symbols loading and also where might be the issue. But let’s dig deeper – firs we want to verify that there is really problem with loading symbols for a particular module of interest:

0:001> lmvm ntdll
start    end        module name
770a0000 771f7000   ntdll      (export symbols)       C:\WINDOWS\SYSTEM32\ntdll.dll
    Loaded symbol image file: C:\WINDOWS\SYSTEM32\ntdll.dll
    Image path: ntdll.dll
    Image name: ntdll.dll
    Timestamp:        Wed Sep 19 22:32:50 2012 (505AAA82)
    CheckSum:         0015B576
    ImageSize:        00157000
    File version:     6.2.9200.16420
    Product version:  6.2.9200.16420
    File flags:       0 (Mask 3F)
    File OS:          40004 NT Win32
    File type:        2.0 Dll
    File date:        00000000.00000000
    Translations:     0409.04b0
    CompanyName:      Microsoft Corporation
    ProductName:      Microsoft« Windows« Operating System
    InternalName:     ntdll.dll
    OriginalFilename: ntdll.dll
    ProductVersion:   6.2.9200.16420
    FileVersion:      6.2.9200.16420 (win8_gdr.120919-1813)
    FileDescription:  NT Layer DLL
    LegalCopyright:   ⌐ Microsoft Corporation. All rights reserved.

And we can see that the (private or public) symbols are not being loaded. Let’s track down why:

0:001> !sym noisy
noisy mode - symbol prompts off
0:001> .reload /f ntdll.dll
DBGHELP: aaa\wntdll.pdb - file not found
DBGHELP: aaa\dll\wntdll.pdb - file not found
DBGHELP: aaa\symbols\dll\wntdll.pdb - file not found
DBGHELP: C:\WINDOWS\SYSTEM32\wntdll.pdb - file not found
DBGHELP: wntdll.pdb - file not found
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for ntdll.dll -
DBGHELP: ntdll - export symbols

Now we see all the paths that are being searched and we can go find why the correct symbol file is not present on any of those paths. Once we fix the issue or/and fix our symbol search path, we can successfully reload symbols.

Sometimes engine doesn’t know which symbol file to load as it doesn’t even have the image loaded (this can for example happen when we debug some minidumps). For this reason we might want to have our images loading set up.

Executable images loading

When windows runs your application it loads the executable binaries (.exe, .dll) into memory – at which point we call them executable images. As they are in memory, they can be part of memory dump of a application. However some smaller dumps may not have them, also it might happen that the image was unloaded – so there can be situations where the debugger might need to load those from separate location (this will become extremely helpful during managed code debugging from a dump – I’ll write about this in separate post).

Executable image path has same rules as symbols search path – and in fact in many situations we might want to set them to same locations as (despite many people don’t realize this) binaries can be also indexed on symbols server.

.exepath C:\MyBuildLocation\Binaries;cache*C:\MySymbolsCache;\\CorporateSymbolsServer\Symbols;
		srv*http://msdl.microsoft.com/download/symbols

And same as with symbols, it’s more advantageous to have this path set already in environment – we can use _NT_EXECUTABLE_IMAGE_PATH variable for this purpose.

Source code loading

Why wouldn’t we make our debugging experience less frustrating by debugging with source codes? Debugger engine can extract the source paths from the symbol files. If we would open a symbol file (e.g. ‘foo.pdb’) in simple text editor we would see that it contains locations of source code – those locations are taken at the build time. So if you debug on the same machine as where you build your code, or if your debug machine has same structure of code files in same location as your build machine, the debugger will be able to ‘magically’ bring up the source code file – but it’s not guaranteed that it’s a correct version of that source file (let’s say someone changed it from the time the debugged binary was build). Therefore the most recommendable way of having the best source code debugging experience is to use a source control system and source index your symbols.

Source indexing symbols means that the symbol file will have inside itself information about how to retrieve the file from source control server and which version to retrieve (basically it has section with translation of local paths to the full commands to the source control). You than need to make sure that the debugger has access to the toll for retrieving sources from source control (e.g. p4c.exe for perforce. You can just put it’s location to the path) and that the source path indicates that you want to use ‘source server’.

To indicate that you want to use source server, set the source path to ‘srv*’ string (beware that syntax and meaning is completely different from symbols or executable image path! for source path you just use ‘srv*’ with no other specification and it means that the debugger will attempt to retrieve the source control commands from the symbol files). To use local path point the source path to the flat path with source codes (no recursive search will be performed!). Or you can combine both approaches. Use .srcpath to set your source path:

.srcpath srv*;<any additional flat source locations>

Or if you can use .srcfix command for this (it sets the source path to the ‘srv*’ and any additional path that you pass to the .srcfix+ <path>).

However the most recommendable way is again by using environment variable - _NT_SOUCE_PATH

Source code loading during remote debugging

As was mentioned earlier – during the basic remote debugging, all commands are being executed on the remote system. This may be problematic in case where you debugging remote issue, however have source files on your local machine, or have the source control utility on your machine. In this situation .srcpath or .srcfix commands will have no effect (which can be confusing). You will need to set local source path and/or use local source server:

.lsrcfix
.lsrcpath+ <local source codes path>

This will resolve source codes on your local machine (as opposed to the remote machine actually running the debugging engine), and source codes will finally ‘magically’ load (unless you have other issues that I’ll try to cover in a second). Also keep in mind that those two commands cannot be scripted.

Troubleshooting source code loading

Sometimes you’re changing the frames in stack and expect the appropriate source code to appear, but nothing happens. For those reason there is a way to troubleshoot such an issue (I’ve censored some irrelevant information):

0:000> .lines -e
Line number information will be loaded
0:000> kn
 # Child-SP          RetAddr           Call Site
00 00000070`af71f6f0 000007ff`ecc72cc9 ntdll!LdrpDoDebuggerBreak+0x30 [d:\XXXXXX\ntdll\ldrinit.c @ 2737]
01 00000070`af71f730 000007ff`ecc0216a ntdll!LdrpInitializeProcess+0x1927 [d:\XXXXXX\ntdll\ldrinit.c @ 5131]
02 00000070`af71fa30 000007ff`ecbf32ae ntdll!_LdrpInitialize+0xee9a [d:\XXXXXX\ldrinit.c @ 1334]
03 00000070`af71faa0 00000000`00000000 ntdll!LdrInitializeThunk+0xe [d:\XXXXXX\ntdll\ldrstart.c @ 90]

0:000> !srcnoisy 3Noisy source output: on
Noisy source server output: on
Filter out everything but source server output: on
0:000> .frame 3
03 00000070`af71faa0 00000000`00000000 ntdll!LdrInitializeThunk+0xe [d:\XXXXXX\ntdll\ldrstart.c @ 90]
DBGENG:  Check plain file:
DBGENG:    'd:\XXXXXX\ntdll\ldrstart.c' - not found
0:000> .srcfix
DBGHELP: Symbol Search Path: srv*c:\mysymcache*http://XXXXXX
Source search path is: SRV*
DBGENG:  Scan srcsrv SRV* for:
DBGENG:    '!d:\XXXXXX\ntdll\ldrstart.c'
SRCSRV:  XX.exe -p XXXXXXXXXXXXX.ntdev.microsoft.com:2020 print -o "D:\Debuggers_64bit\src\
			XXXXXXXXXX\ntdll\ldrstart.c\1\ldrstart.c" -q //depot/XXXXXXXXXX/ntdll/ldrstart.c#1
DBGENG:      found file 'd:\XXXXXXXXXX\ntdll\ldrstart.c'
DBGENG:      server path 'SRV*'
DBGENG:      local 'D:\Debuggers_64bit\src\XXXXXXXXXXXX\ntdll\ldrstart.c\1\ldrstart.c'

So we see that we can use .lines command to enable (or disable with -d) displaying of that build time paths baked into the symbol file. Alse we can see that !srcnoisy can switch on more verbose logging on what’s happening during sources loading. Finally we see how is the source file loaded from source control server – provided that symbol file was properly source indexed. Here is what would happen if symbol file wouldn’t be source indexed:

0:000> .frame 10
SRCSRV:  d:\YYYYYYYYYYY\identityauthority.cpp not indexed

Finishing the debugging

Once we are done with debugging session we have several option of ending the session. If we are not debugging live user mode session, we might not care about the way of exiting – simple close of the debugger app or q command might be enough. However when debugging the live user mode process we might want to preserve the process. To end debugging without killing the target application use .detach command or qd (quit and detach) command, or best way might be to start the debugger with –pd option, which will cause that target application will remain running, no matter how you quit your debugging session.

I am a software developer in Microsoft Corporation working on an internal testing platform/framework for testing Windows (externally shipping as the Windows Logo Toolkit). Being daily in the first line during the thorough ‘attacks’ of large amounts of various tests; designing high reliability software, hardening legacy code and debugging user mode and kernel mode crashes (from the basic AVs to the most obscure heap corruptions) is bread and butter of our daily work.
I decided to share some of the gained experience and knowledge trough the posts on this blog (as the time will allow). I hope this will not only be a future reference for myself, but also a helpful source of information for others. Let’s see…

But now – Welcome on my blog!!