The team i belong to (Portugal Adcs) has a blog called Deviations (https://blogs.msdn.com/deviations) and from time to time i will post some stuff over there (when that happens i will put a link here).

My latest post there was about live debugging using Windbg and you can check it out at

https://blogs.msdn.com/deviations/archive/2009/03/18/under-the-hood-with-live-debugging.aspx

Have fun
Bruno

Yesterday, I and a co-worker were troubleshooting an strange issue that didn´t occur in dev but in staging (if i get a cent every time i ear this) a MessageSecurityException was being thrown stating unauthorized access. If we allowed Anonymous Access then no exception was being thrown. Below is the scenario.

 Scenario

• Single Machine environment
• Windows 2003 and IIS6
• Both WebApp and WebService on same pool and running under an account with privileges
• WebApp calling locally WebService
• Integrated Authentication (Anonymous not allowed)

Details

So, we decided to take a memory dump on System.ServiceModel.Security.MessageSecurityException using DebugDiag.(https://www.microsoft.com/downloadS/details.aspx?FamilyID=28bd5941-c458-46f1-b24d-f60151d875a3&displaylang=en)

Lets look at the faulting thread

Now, lets look at exception details

Also in IIS Logs we could see 401.1 unauthorized.

In our case the solution was to install security update 957097. More details, info on workaround or download update at

https://support.microsoft.com/default.aspx?scid=kb;EN-US;896861 (You receive error 401.1 when you browse a Web site that uses Integrated Authentication and is hosted on IIS 5.1 or IIS 6)

After doing this we decided to look at dev environment, and surprise, this update was installed.

I know you all have eared this before, so here it goes again: it is very important to keep environments (dev, stage, production) the most similar to the extent possible.

Have fun.

Bruno

My first mistake was not to use the same approach I always use in customers (do what I say not what I do J ) and started trying the try…error approach. The only thing this did was to waste my time with successive reboots, remove profile, add profile and other stuff. In my machine I have two profiles: the local admin one and another that I use to connect to Microsoft. The problems were in this second profile. The first one worked without any problems.

So I decided to stop for a minute and follow the typical methodology I always follow. So since my suspicions were on a driver or some program I had installed over the weekend (beta versions, don´t get me wrong I love beta versions because I believe they are a great way to help products getting better and better).

Since Switch User was working, the first step was to reproduce the problem. So I logged in with the profile that was having problems and then I switched to my local admin account. Then I started Process Explorer (aka ProcExp, you can download it from https://technet.microsoft.com/en-us/sysinternals/bb896653.aspx ). If you don´t know this tool, you can think of it as task manager on steroids and is a very valuable troubleshooting tool.

After opening ProcExp you will see a list of all processes running on your system and also the account that started the process (this allowed me to see processes that were running in my problematic profile).

To see the username, you have to right click on one of the columns and “Select Columns” > “Username”. Then if you click order by Username, you will notice that each group will have a different color. You can see this in the image below (i´ve removed the usernames from my machine, that´s why you see a blank column under Username)

Process Explorer allows you to see threads and call stacks for each process, so my approach was to look at processes that were executing under DOMAIN\USER and dig into each one of them trying to find strange behaviors. You might ask, what is a strange behavior? I would say, it depends (wow, I’ve never heard that answer before). In my case I decided to start looking at processes that were consuming more CPU and those with names that I was not familiar with.

So how do you look at threads and call stacks? Right click on a process and select Properties. Inside Properties you have a tab called Threads. Click on it.

Here you can try to identify some thread that does not look good, or it might be that they are all hang (these are just some examples, and the more experience you have the easier it would be for you) . Let´s imagine I had one thread that was consuming almost all CPU and then i could look at the stack (select your thread and click on STACK)

So I did this for a couple of processes until I identified the bad boy. Then I disabled it, switched user and “voilá” all my problems were gone (and part of my working day).

Paulo, after all it was not the beta version you gave me that got me into troubles J

Happy debugging!!!

Bruno

So, I suggest you give a look at it. You can get it at https://www.editpadpro.com/editpadlite.html

Bruno

Every time I post there I will put here a link to the post (I will talk mostly about troubleshooting).

My first post there is about Developer Tools that come with Internet Explorer 8 and how they can help us troubleshooting Jscript errors (I know, they are a big pain for us all developers J ).

The post can be seen at https://blogs.msdn.com/deviations/archive/2008/10/09/debug-jscript-with-internet-explorer-8.aspx.

About the team blog, you can check it at https://blogs.msdn.com/deviations.

Bruno

The first time someone showed me Windbg and all the stuff around production debugging like memory dump, … I was a little (maybe not just a little) scared on this new world that had just opened in front of me. So my next question was: where do I start from? And probably like almost everyone that starts exploring this “new world” there are excellent blogs like https://blogs.msdn.com/tess (If Broken it is, fix it you should), https://blogs.msdn.com/dougste (Notes from a dark corner), https://blogs.msdn.com/tom (ASP.NET Debugging) and many others.

One of my biggest problems at first was to understand (and remember) what and when I should collect information that could help me to troubleshoot the issue. The blogs above have a lot of information about taking the first steps. To simplify this task I´ve created a little graphical memo that would help me. Now don´t take this flowchart as the “holy grail” but I believe it can help (at least until you get the basics in your mind).

Right click on the image to save it and it will be more “readable”.

Have fun

Bruno

“The .shell command launches a shell process and redirects its output to the debugger, or to a specified file.”

So, why would I find that interesting and useful in my day to day work to launch a shell process ? Maybe to impress friends (that know nothing about debugging) with strange commands or make it sound very complicated. Well, actually I use .shell in one of the most simple tasks we all do everyday and that´s finding text. Now you might be thinking, so why don´t you use Ctrl+F and find what you want ? (We will talk about this in a couple of minutes)

So, has I was saying, I use it a lot to find text inside memory dumps and this saves me time. Since .shell launches a shell process, the key here is to use the old FIND command from DOS to help us. FIND allows to search for text inside a file. If you open a command line and do FIND /? You will see something like below.

The sample below is an example (a very simple one) of finding a specific string in the call stack. I use it a lot to find specific values inside objects properties but i´m sure you will find other useful uses for this command.

The argument –ci specifies that the output of the command “~*kb” is to be used as input for FIND command. There are some more options that you can look at Debugging Tools For Windows help.

Till next time. Have Fun!!!

Bruno

Typically when i´m looking at a classic ASP memory dump I like to use the extension that comes with DebugDiag called IISInfo.dll that helps me getting a nice view on requests executing. Below are the commands available with this extension (I won´t get into details but they are pretty much self explanatory).

In this specific case, after looking at requests executing all were getting struck on Xmlhttp send

The source code was showing that a XMLHTTP object was being created and then a request was made. So what was the problem about creating a XMLHTTP object ? Well, sending XMLHTTP requests from server is not supported and can lead to poor performance or even worse.  Our solution was to use the SERVERXMLHTTP.

Below are some links of interest around XMLHTTP vs SERVERXMLHTTP.

“Active Server Pages (ASP) script or Internet Server API (ISAPI) code that attempts to use XMLHttpRequest (Microsoft.XMLHTTP) functionality of the Microsoft XML engine (MSXML) to send XML requests to another Web server may function incorrectly or perform poorly.”

https://support.microsoft.com/kb/237906/en-us  (PRB: Loading Remote XML or Sending XML HTTP Requests from Server Is Not Supported)

 “Using ServerXMLHTTP or WinHTTP objects to make recursive Hypertext Transfer Protocol (HTTP) requests to the same Internet Information Server (IIS) server is not recommended. More specifically, the calling Active Server Page (ASP) should not send requests to an ASP in the same virtual directory or to another virtual directory in the same pool or process. This can result in poor performance due to thread starvation.”

https://support.microsoft.com/default.aspx?scid=kb;EN-US;316451 (INFO: Do Not Send ServerXMLHTTP or WinHTTP Requests to the Same Server)

XMLHTTP and ServerXMLHTTP?

As the name suggests, ServerXMLHTTP is recommended for server applications and XMLHTTP is recommended for client applications. XMLHTTP has some advantages such as caching and auto-discovery of proxy settings support. It can be used on Windows 95 and Windows 98 platforms, and it is well-suited for single-user desktop applications.”

https://support.microsoft.com/kb/290761 (Frequently asked questions about ServerXMLHTTP)

Have fun!!!

Bruno

When i´m looking at a managed memory dump almost always there is the need to look inside the code so that I can get a better view on how the application i´m troubleshooting works. Since usually I don´t have access to source code the path to follow is to get the image loaded in memory and save it to disk. Then I use (and I bet that you also know this tool) Lutz fantastic tool .NET Reflector   (now acquired by RedGate https://www.red-gate.com/products/reflector/ ).

If you are using sos/psscor extension then there is available the command !SaveModule that allows you to get this file. The syntax is pretty simple help provided by executing !Help !SaveModule is pretty self explanatory.

Now if by any chance you can´t use sos/psscor or you want to get a image from a unmanaged memory dump you can use .WriteMem command to get the job done. All you have to do is get the base address and size of the asembly

The .writemem command writes a section of memory to a file. If you execute .hh .writemem on windbg you will get

Bruno

I can´t post the details about the next step (finding the trace and filters) because I used private symbols but after you see this call stack I suggest that you try to find what traces are running in your environment at the time the issue occurred.

Have fun!!!

Bruno