Web Technology Blog

Azure App Service - What is taking my disk space for my Azure Web App ?

PuneetG — Tue, 06 Jun 2017 08:58:23 GMT

While working on Azure App service, you may notice disk full errors. These may surface in different ways for e.g.

The application may get the error "System.IO.IOException: There is not enough space on the disk".
You may see disk errors while browsing to the KUDU Console.
You try to publish using VSTS or Visual Studio and you get an error like "ERROR_NOT_ENOUGH_DISK_SPACE: Web deployment task failed. (Web Deploy detected insufficient space on disk)".

In this post, I will cover how you can identify why you are getting these errors, and identify what inside your Web App might be consuming disk space. First and foremost, it is important to understand that each App Service Plan gets an allocation of disk space corresponding to the pricing tier. At the time of writing of this blog, this is the amount of disk space that is available per pricing tier ( This might change in future so you should check out https://azure.microsoft.com/en-in/pricing/details/app-service/plans/ which should have the updated information.)

	FREE	SHARED	BASIC	STANDARD	PREMIUM
Disk space	1 GB	1 GB	10 GB	50 GB	250 GB

All of your Web Apps in the same App Service Plan will share the disk space. If your files on the disk exceed this limit, you will start getting disk full kind of errors. Your files will include one or more of the items in the following list.

The Web Site Content
The App_Data for an Asp.Net Application
The Logfiles folder.

The reason why it becomes a little tricky to figure out this issue is because you have may have multiple Web Apps in an App Service Plan. If one of the Web Apps is taking more disk space than the others, you may not be able to easily identify which Web App to investigate. However, to help you, the Quotas blade at the Web App level in Azure portal (https://portal.azure.com/) allows you identify this issue easily. Here is a screenshot of what I see when search for Quotas in the settings for my Web App. In the above screenshot, you can see two quotas.

The File System Usage (App) which is the disk space consumed by my Web App.
The File System Usage (App Service Plan) which is the disk space consumed by the App Service plan.

The above image illustrates that the App Service plan is consuming 6.13GB of disk space (out of 50 GB total as my Web App is on a Standard Plan which gives me 50 GB) and my Web App is consuming 4.68 GB out of this 6.13 GB. The remaining disk space is consumed by the other Web Apps in the App Service Plan. So from here, I can quickly identify whether my current Web App (i.e. the Web App currently selected in the portal) ,is consuming high disk space or if the disk space is consumed by another Web App in the same App Service Plan. If the current Web App is consuming the disk space then I have already identified the culprit, otherwise I will look at the other Web Apps in my App Service Plan and check the Quotas blade for those Web Apps individually. To list all of the Web Apps in the App Service plan , I can click on App Service Plan for my Web App and click on the Apps setting. This will show me all the sites under my App Service Plan and from there on , I can click on each Web App and look at the Quotas blade for that Web App to see how much disk space that Web App is consuming. This maybe complex if there are many Web Apps under the same App Service Plan. Please note that this part of the portal is subject to change and please feedback on http://feedback.azure.com if you have any input. In the meantime, I wrote a small PowerShell script that can help you find this information. Just pass the Web App name and the subscription ID and the script will identify the App Service Plan for the Web App and show App Service Plan disk usage and a breakup of disk space consumed by all Web App and slots in your App Service Plan. Once you identify the right Web App, you may want to understand what content inside that Web App is consuming high disk space. For that, you can install the Disk Usage Site extension on that Web App to identify what exactly within that Web App is consuming the maximum space. There is a good external blog that talks about how to use this extension http://fenglu.me/2017/02/18/Review-and-manage-your-web-site-space-in-Azure/ so I am not going to repeat the steps but using this extension you can exactly identify which sites or folders within your WebApp content are consuming high disk space. Hope this information helps !!!

Automating WebApps hosted in Azure App Service through PowerShell - ARM Way

PuneetG — Mon, 21 Mar 2016 13:27:01 GMT

Azure Resource Manager is a powerful way of automating almost anything in Azure. In this blog I will talk about how we use the ARM based PowerShell CmdLets to automate common tasks for managing / maintaining WebApps in Azure App Service. First of all, I want to mention that there are two modes for writing PowerShell scripts to automate Azure. When you install Azure PowerShell two sets of modules get installed namely Azure Service Management (ASM) and Azure Resource Management(ARM). Both of these had different powershell commands to achieve the same result. In this blog post, I will talk about the ARM way of automating Azure Resources because ARM is the preferred way going forward and ARM is what the latest Azure portal (https://portal.azure.com) uses and is more promising from a feature set. Installing Azure PowerShell The easiest way to install Azure PowerShell is through Web Platform Installer (WEB PI). If you do not have Web Platform Installer, install it first from https://www.microsoft.com/web/downloads/platform.aspx. Once Web Platform installer is installed, launch it by going to Start and typing Web Platform Installer. Once WEB PI launches, search for Azure PowerShell and choose Microsoft PowerShell from the list below and click Add and then Install to install Azure PowerShell The download for Azure PowerShell is roughly 30 MB so shouldn’t take long to install. Once installed, you can just type Windows PowerShell in the Start Menu and the PowerShell Command window will open which will allow you to run the Azure PowerShell Commands. I prefer using the Windows PowerShell ISE to write my PowerShell scripts just because it provides you intellisense, color coding and lets you easily view details of the CmdLets that are available in PowerShell modules so if you choose AzureRM.Websites in the Modules dropdown, you can see the list of available cmdlets for WebApps like this. All these commands are documented in detailed in MSDN @ https://msdn.microsoft.com/en-us/library/mt619237.aspx and I won’t spend too much time talking about each command as the names are self-explanatory. One last thing that I want to talk about before diving in the code is Azure Resource Manager Explorer which is basically an online tool that gives you a tree-view of all your resources in Azure. If you are often involved in writing PowerShell scripts for Azure then knowing about this tool is a must. You can watch a small video @ https://azure.microsoft.com/en-us/documentation/videos/azure-resource-manager-explorer-with-david-ebbo/ to get a quick understanding of how this works and this tool is what I have used in most of the scripts that I created. What is even more cool is that the ARM Explorer is open source and available at https://github.com/projectkudu/ARMExplorer. To access Azure Resource Manager Explorer, you browse to https://resources.azure.com and sign in with the same credentials that you use to sign-in to the Azure Portal and the ARM Explorer shows you all your resources in a tree view. You can search for a resource name and see more details of a particular resource. So in the below screenshot, I searched for one my existing sites called demowebcamp and I see the following So here you can see all he configurations for your WebApp. What is even cool is if you click on the last tab (PowerShell), it shows you same powershell commands that you can use to create \ modify \ delete your resource like this Notice that in the above window, we are not even using any AzureRM.Websites Cmdlet (like Get-AzureRMWebApp or Set-AzureRMWebApp) and the tool is showing the generic Azure Resource Manager Cmdlets (like Get-AzureRMResource, Set-AzureRMResource, Remove-AzureRMResource) with the right set of parameters applied for WebApp so configuring something in a WebApp now is as easy as finding out where the setting lies in the resource tree shown by https://resources.azure.com and then clicking the PowerShell tab to understand how to write the script for changing that particular setting. So whenever we want to write some powershell script which has to modify or configure something, just follow this approach

Open https://resources.azure.com and find the resource that you want to configure (For e.g. WebApp or App Service Plan)
Expand the tree nodes beneath the resource to identify the setting that you want to configure
Switch to the PowerShell tab in the tool to identify the right set of PowerShell command corresponding to that operation.

With that, let’s see the code for some of the common tasks that you might want to perform on WebApp

Stopping / Starting / Restarting WebApp
Creating A New WebApp in a New Resource Group
Downloading Publish Profile For a WebApp
Configuring Always-On for WebApp
Adding a Virtual Directory Marked as an Application to the site
Scaling the WebApp from one pricing Tier to another
Configure Connection String for a WebApp
Append New AppSetting to the existing list of App Settings for a WebApp
Adding a HOSTNAME for a WebApp
Adding a SSL Binding for a WebApp
Adding a NEW Slot to a WebApp
Adding a Sticky To Slot Setting for a WebApp
Performing Swap operation on a WebApp
Delete a WebApp
Modify CORS setting on the WebApp
List Sites in App Service Plan
List Disk Usage of all Sites in App Service Plan

Most of the snippets below use the following function which is required to get the resource group name for a WebApp so this should be included in the script snippet Stopping / Starting / Restarting WebApp Creating A New WebApp in a New Resource Group Downloading Publish Profile For a WebApp Stopping / Starting / Restarting WebAppConfiguring Always-On for WebAppAdding a Virtual Directory Marked as an Application to the siteScaling the WebApp from one pricing Tier to anotherConfigure Connection String for a WebAppAppend New AppSetting to the existing list of App Settings for a WebAppAdding a HOSTNAME for a WebAppAdding a SSL Binding for a WebAppAdding a NEW Slot to a WebAppAdding a Sticky To Slot Setting for a WebApp Performing Swap Operation on a WebAppDelete a WebApp Modify CORS Settings For WebAppList sites in App Service Plan Find Disk Usage for All sites in App Service Plan If there is some common operation that you need to automate, then kindly mention it in the comments and I will try to update the blog with the code for that. Hope this helps !!!

Azure Website SiteExtension to view EventLogs

PuneetG — Wed, 24 Sep 2014 08:58:07 GMT

Did you know that Microsoft Azure websites have an amazing feature called Site Extensions ? Site extensions enables you to write a set to tools that can be installed on any Azure website once your extension is deployed in the SiteExtensions Gallery. To know more about Azure site extensions and Azure SiteExtensions gallery, please read http://azure.microsoft.com/blog/2014/06/20/azure-web-sites-extensions/ Site Extensions can be managed from the Kudu Console (more info here ) or the new Azure Portal at http://portal.azure.com . In the new portal you’ll find the settings under Web Sites when you click on your Web Site. When this blade has loaded you’ll find the Extensions section under Configuration Settings. If you’re not familiar with the new portal yet then this Azure Friday session is a great view. Every Azure web site has an EventLog.xml which is stored in D:\home\Logfiles folder. This eventlog.xml is a dump of Application Event logs and can be really useful in identifying runtime errors that occur on your web site as mentioned here. If you notice the event log is an XML format and can become hard to read if you have too many events logged. Also it contains events dumped in “raw” format, something like below and is not easily readable. This site extension will provide you with a formatted view of the Event Log which should be more easily consumed (and slightly prettier as well). Here is a screenshot of the same event when viewed in the site extension. Hope you find this useful. A big thanks to Rakki Muthukumar for helping with the styling of this page. To install this Site extension on your Azure Website

Access the KUDU console for you web site by accessing https://*****.scm.azurewebsites.net where ***** is the name of your Azure Web site (****.azurewebsite.net)
Click on the Site Extensions tab
Then click on Gallery and Choose Azure Web Sites Event Viewer

Once it is installed, the web page will give you a message saying that the site needs to be restarted and will give you a Restart Site button right there.
Click on the Restart site button and navigate to the Installed section
Hit the PLAY button to launch the Event Viewer.
Once this site extension is installed, you can directly launch it by going to https://*****.scm.azurewebsites.net/EventViewer

I would appreciate any feedback in this regard !!!

How to get the WS-AT Transactions Tab in DCOMCNFG ?

PuneetG — Fri, 05 Oct 2012 09:01:15 GMT

If you worked with Distributed Transactions, you may already be aware that they are configured in “Component services” MMC. We can also configure WS-AT transactions in “Component Services”, but there are a few steps required to configure it properly. In this blog I will talk about how to enable this functionality in the MMC and the various issues that you can run into while configuring it. By default on a new installation of any Operating System, the WS-AT TAB is not present in the Component Services MMC under the MSDTC settings and this is how the default UI looks like. WsatUI.dll is the managed UI Extension DLL that contains this TAB functionality and it needs to be registered using the regasm tool. So where to get WSATUI.DLL from? This DLL ships as a part of the Windows SDK for .NET Framework and if the machine has.NET Framework 4.0, you need to install the Windows 7 SDK for .NET Framework 4.0 from http://www.microsoft.com/en-us/download/dlx/listdetailsview.aspx?id=8279 and if the machine has the .NET Framework 3.0 or .NET Framework 3.5 installed, then you can install the Microsoft Windows SDK for Windows 7 and .NET Framework 3.5 SP1 from http://www.microsoft.com/en-us/download/details.aspx?id=3138. It is a huge download and it can take some time to download and install. Also don’t get confused by the name of the SDK – the name says Windows 7 SDK for .net framework 4.0 but this can be installed on Windows Server 2003 or Windows Server 2008 also. To make it a bit more clear, the Windows 7 SDK for .net framework 4.0 is a Software development Kit for creating apps that run on Windows 7 and .Net Framework and can be installed on prior operating systems too. Another important thing to note is that you shouldn’t worry about the CPU architecture or the version of .NET Framework that you are using in your application because this DLL is used by Component Services MMC only and your application will never be using this DLL directly. So even if your application uses .NET Framework 3.0 or 3.5 on a machine that has .NET 4.0 installed, you should still install the Windows 7 SDK for .NET Framework 4.0 Registering the WSATUI.DLL After the installation open the SDK Command shell from the Start -> Programs -> Microsoft Windows SDK menu (on VISTA and above operating systems, right click on the Command shell and choose “Run As Administrator” and run the right REGASM command (assuming you installed the SDK in the default locations) Table 1

Operating System Processor Architecture	Highest Version of Framework installed on the computer	Installed SDK Version	Command to Run
32 Bit	3.0 , 3.5	V6.0	Regasm /codebase “c:\Program Files\Microsoft SDKs\Windows\v6.0\Bin\WsatUI.dll”
		V6.1	Regasm /codebase “c:\Program Files\Microsoft SDKs\Windows\v6.1\Bin\WsatUI.dll”
		v7.0	Regasm /codebase “c:\Program Files\Microsoft SDKs\Windows\v7.0\Bin\WsatUI.dll”
	4.0 (see note below)	v7.1	Regasm /codebase “c:\Program Files\Microsoft SDKs\Windows\v7.1\Bin\WsatUI.dll”
64 Bit	3.0 or 3.5	V6.0	Regasm /codebase “c:\ Program Files\Microsoft SDKs\Windows\v6.0\Bin\x64\WsatUI.dll”
		V6.1	Regasm /codebase “c:\ Program Files\Microsoft SDKs\Windows\v6.1\Bin\ x64\WsatUI.dll”
		v7.0	Regasm /codebase “c:\ Program Files\Microsoft SDKs\Windows\v7.0\Bin\ x64\WsatUI.dll”
	4.0 (see Note below)	v7.1	Regasm /codebase “c:\ Program Files\Microsoft SDKs\Windows\v7.1\Bin\ x64\WsatUI.dll”

NOTE: The version of Windows 7 SDK for .NET Framework 4.0 have an issue that while running the regasm command you may get an error complaining about the assembly Authenticode signatures and you may get an error like this. C:\Program Files\Microsoft SDKs\Windows\v7.1>regasm /codebase "C:\Program Files\Microsoft SDKs\Windows\v7.1\Bin\wsatui.dll" Microsoft (R) .NET Framework Assembly Registration Utility 4.0.30319.1 Copyright (C) Microsoft Corporation 1998-2004. All rights reserved. RegAsm : error RA0000 : Could not load file or assembly 'WsatUI, Version=3.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. Strong name validation failed. (Exception from HRESULT: 0x8013141A) This happens due to a small issue in the file that is packaged in the SDK. If you run in to this issue, you can do one of the two things to fix this problem

Turn off strong name validation just on the WSATUI.DLL by running the following command from the same SDK CMD shell

C:\Program Files\Microsoft SDKs\Windows\v7.1>sn -Vr "c:\Program Files\Microsoft SDKs\Windows\v7.1\bin\wsatui.dll" Microsoft (R) .NET Framework Strong Name Utility Version 4.0.30319.1 Copyright (c) Microsoft Corporation. All rights reserved. Verification entry added for assembly 'WsatUI,B03F5F7F11D50A3A' After this you should be able to run the REGASM command

If you don’t want to turn off the strong name validation on this DLL, then you need to replace the WSATUI.DLL in the folder c:\Program Files\Microsoft SDKs\Windows\v7.1\Bin\ with the DLL from a machine which has got Visual Studio 2010 installed. On a machine that has Visual Studio 2010 installed, you can get this DLL from the folder C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\Bin\WSATUI.dll and copy it to the destination server. Once that is done you can run the regasm command and it should work.

If the above steps are followed, the DLL’s should be registered correctly and you should be able to see the WS-AT tab, however if this is not done correctly or if there is some kind of a version mismatch, you may see an error like the below one when you try to go to the WS-AT TAB in the Component Services The error “The installed version of Windows Communication Foundation does not match the version of the WSAT configuration tool” will come if the version of the WSATUI.DLL registered on the server is lower than the highest version of the framework installed on the server. To check the highest version of .net framework installed go to HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\NET Framework Setup\NDP and look for the highest version. So from the below screenshot, you can see that the machine has .net framework 4.0 installed The next thing to check is the version of the WSATUI.DLL that is registered on the server and that can be looked inside HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\WSAT\3.0. This key will have a string value called PATH and that will tell us the location of the WSATUI.DLL. If the location of WSATUI.DLL shows as 6.0,6.1 or 7.0 , WSATUI.dll will only support 3.5 Framework version. If it shows as registered from v7.1 path, it can work with 4.0 framework version. So if you see that the version of the .NET Framework installed on the machine is higher than the version that WSATUI.DLL supports, then follow the combinations given in the Table-1 (above in this article) and install the right version of SDK and register the WSATUI.DLL from the correct path and that should fix this error.

Finding out the Thread which is actually handling the WCF request for the HttpContext

PuneetG — Tue, 11 Jan 2011 09:06:54 GMT

In the recent past, the PSSCOR2 debugger extension was made public. This debugger extension contains a very useful command called !aspxpages which shows you information about various asp.net requests running in the dump, the time they have been executing and the thread associated with the System.Web.HttpContext. Here is an example of what this command shows you

0:000> !aspxpages
Going to dump the HttpContexts found in the heap.
Loading the heap objects into our cache.
HttpContext    Timeout  Completed     Running  ThreadId ReturnCode   Verb RequestPath+QueryString
0x01bbf914    11647 Sec       yes                   XXX        200   POST /MySleepingWcfService/Service.svc 
0x01bc0ba4    11647 Sec       yes                   XXX        200   POST /MySleepingWcfService/Service.svc 
0x01bc17c4    11647 Sec       yes                   XXX        200   POST /MySleepingWcfService/Service.svc 
0x01bc4660    11647 Sec       yes                   XXX        200   POST /MySleepingWcfService/Service.svc 
0x01bc56a4    11647 Sec       yes                   XXX        200   POST /MySleepingWcfService/Service.svc 
0x01bc6660    11647 Sec       yes                   XXX        200   POST /MySleepingWcfService/Service.svc 
0x01bc76a4    11647 Sec       yes                   XXX        200   POST /MySleepingWcfService/Service.svc 
0x01bc8660    11647 Sec       yes                   XXX        200   POST /MySleepingWcfService/Service.svc 
0x01bc96a4    11647 Sec       yes                   XXX        200   POST /MySleepingWcfService/Service.svc 
0x01bca6fc    11647 Sec       yes                   XXX        200   POST /MySleepingWcfService/Service.svc 
0x01bcb740    11647 Sec       yes                   XXX        200   POST /MySleepingWcfService/Service.svc 
0x01bcc6fc    11647 Sec       yes                   XXX        200   POST /MySleepingWcfService/Service.svc 
0x01bcd740    11647 Sec       yes                   XXX        200   POST /MySleepingWcfService/Service.svc 
0x01bce6fc    11647 Sec       yes                   XXX        200   POST /MySleepingWcfService/Service.svc 
0x01bcf740    11647 Sec       yes                   XXX        200   POST /MySleepingWcfService/Service.svc 
0x01bd0784    11647 Sec       yes                   XXX        200   POST /MySleepingWcfService/Service.svc 
0x01bd17c8    11647 Sec        no        29 Sec      27        200   POST /MySleepingWcfService/Service.svc 
0x01bd2814    11647 Sec       yes                   XXX        200   POST /MySleepingWcfService/Service.svc 
0x01bd3858    11647 Sec       yes                   XXX        200   POST /MySleepingWcfService/Service.svc 
0x01bd4814    11647 Sec        no        29 Sec      29        200   POST /MySleepingWcfService/Service.svc 
0x01bd68dc    11647 Sec        no        29 Sec      24        200   POST /MySleepingWcfService/Service.svc 
0x01bdb404    11647 Sec        no        29 Sec      31        200   POST /MySleepingWcfService/Service.svc 
0x01be3404    11647 Sec        no        28 Sec      33        200   POST /MySleepingWcfService/Service.svc 
0x01bfd404    11647 Sec        no        28 Sec      35        200   POST /MySleepingWcfService/Service.svc 
0x01c0f41c    11647 Sec        no        27 Sec      37        200   POST /MySleepingWcfService/Service.svc 
0x01c1541c    11647 Sec        no        27 Sec      39        200   POST /MySleepingWcfService/Service.svc 
0x01c3c634    11647 Sec       yes                   XXX        200   POST /MySleepingWcfService/Service.svc 
0x01c404c8    11647 Sec        no        25 Sec      41        200   POST /MySleepingWcfService/Service.svc 
0x01c6264c    11647 Sec        no        25 Sec      44        200   POST /MySleepingWcfService/Service.svc 
Total 29 HttpContext objects

Using this information you can find out the thread which is handling your request and if the request is taking a longer time to execute, you can check the callstack of the thread to see what it is doing and why it is taking a longer time to execute and this is pretty helpful. If you are hosting WCF services inside IIS, the same command will show you the information about the HttpContexts associated with the WCF requests and the thread handing the request but because of the way the WCF requests work inside IIS, the thread reported by the !aspxpages command will always show you the call-stack like below.

0:027> !clrstack
OS Thread Id: 0x2740 (27)
ESP       EIP     
0f4af060 775564f4 [HelperMethodFrame_1OBJ: 0f4af060] System.Threading.WaitHandle.WaitOneNative(Microsoft.Win32.SafeHandles.SafeWaitHandle, UInt32, Boolean, Boolean) 
0f4af10c 6f6e685f System.Threading.WaitHandle.WaitOne(Int64, Boolean) 
0f4af128 6f6e6815 System.Threading.WaitHandle.WaitOne(Int32, Boolean) 
0f4af13c 6f6e67dd System.Threading.WaitHandle.WaitOne() 
0f4af144 79962000 System.ServiceModel.Activation.HostedHttpRequestAsyncResult.ExecuteSynchronous(System.Web.HttpApplication, Boolean)
0f4af174 799618f4 System.ServiceModel.Activation.HttpModule.ProcessRequest(System.Object, System.EventArgs)
0f4af19c 6574ec35 System.Web.HttpApplication+SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() 
0f4af1b4 65742a5c System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef) 
0f4af1f4 6574e1f3 System.Web.HttpApplication+ApplicationStepManager.ResumeSteps(System.Exception) 
0f4af244 65741fdc System.Web.HttpApplication.System.Web.IHttpAsyncHandler.BeginProcessRequest(System.Web.HttpContext, System.AsyncCallback, System.Object) 
0f4af260 6574554c System.Web.HttpRuntime.ProcessRequestInternal(System.Web.HttpWorkerRequest) 
0f4af294 657451f3 System.Web.HttpRuntime.ProcessRequestNoDemand(System.Web.HttpWorkerRequest) 
0f4af2a4 6574438c System.Web.Hosting.ISAPIRuntime.ProcessRequest(IntPtr, Int32) 
0f4af4b8 70290256 [ContextTransitionFrame: 0f4af4b8]  
0f4af4ec 70290256 [GCFrame: 0f4af4ec]  
0f4af648 70290256 [ComMethodFrame: 0f4af648]

The reason for this is the way WCF runtime handles the request. You will always see two threads to process the WCF request. One thread is the CLR ThreadPool thread (which will look something like the above thread) which is the worker thread that comes from ASP.NET and the other thread is an I/O thread that is managed by the WCF IOThreadScheduler (actually created by ThreadPool.UnsafeQueueNativeOverlapped). More details on this behavior are mentioned in this blog. Most of the times you are interested in finding out the actual WCF thread which is servicing this request because that would be the thread which is actually executing the method code inside your WCF service and if your service it service method is taking a long time to execute, that would be thread that would be the thread that requires attention. At this point, there is no direct command to show you the WCF I/O thread. To find that out, you have to look at the managed call-stack of all the threads in a dump file and find out the threads which have a call to System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke function and track that back to the ASP.NET Worker thread to assosiate it with the actual ASP.NET worker thread shown in !aspxpages output. So from this dump file, I just picked one such thread which had a call to System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke function (Thread 45).

OS Thread Id: 0x23b4 (45)
ESP       EIP     
0fc8e428 775564f4 [HelperMethodFrame: 0fc8e428] System.Threading.Thread.SleepInternal(Int32) 
0fc8e47c 0f3c00a2 Service.SleepyMethod(Int32)
0fc8e48c 0e6301cd DynamicClass.SyncInvokeSleepyMethod(System.Object, System.Object[], System.Object[])
0fc8e4a0 79957338 System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(System.Object, System.Object[], System.Object[] ByRef)
0fc8e558 79955572 System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(System.ServiceModel.Dispatcher.MessageRpc ByRef)
0fc8e5ac 799550e3 System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(System.ServiceModel.Dispatcher.MessageRpc ByRef)
0fc8e5e4 79955000 System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage4(System.ServiceModel.Dispatcher.MessageRpc ByRef)
0fc8e610 79954f66 System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage3(System.ServiceModel.Dispatcher.MessageRpc ByRef)
0fc8e624 79954d43 System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage2(System.ServiceModel.Dispatcher.MessageRpc ByRef)
0fc8e638 799544c7 System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage1(System.ServiceModel.Dispatcher.MessageRpc ByRef)
0fc8e658 7995420b System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean)
0fc8e69c 7995415d System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.Dispatch(System.ServiceModel.Dispatcher.MessageRpc ByRef, Boolean)
0fc8e6ac 79953b3e System.ServiceModel.Dispatcher.ChannelHandler.DispatchAndReleasePump(System.ServiceModel.Channels.RequestContext, Boolean, System.ServiceModel.OperationContext)
0fc8e854 79952b4f System.ServiceModel.Dispatcher.ChannelHandler.HandleRequest(System.ServiceModel.Channels.RequestContext, System.ServiceModel.OperationContext)
0fc8e888 79952893 System.ServiceModel.Dispatcher.ChannelHandler.AsyncMessagePump(System.IAsyncResult)
0fc8e8a0 7991cf05 System.ServiceModel.Dispatcher.ChannelHandler.OnContinueAsyncReceive(System.Object)
0fc8e8b0 79959f86 System.ServiceModel.Channels.IOThreadScheduler+CriticalHelper+WorkItem.Invoke2()
0fc8e8f0 79959f28 System.ServiceModel.Channels.IOThreadScheduler+CriticalHelper+WorkItem.OnSecurityContextCallback(System.Object)
0fc8e8f8 6f6bfe35 System.Security.SecurityContext.Run(System.Security.SecurityContext, System.Threading.ContextCallback, System.Object) 
0fc8e910 79959eed System.ServiceModel.Channels.IOThreadScheduler+CriticalHelper+WorkItem.Invoke()
0fc8e924 79959e70 System.ServiceModel.Channels.IOThreadScheduler+CriticalHelper.ProcessCallbacks()
0fc8e958 79959cca System.ServiceModel.Channels.IOThreadScheduler+CriticalHelper.CompletionCallback(System.Object)
0fc8e984 79959c3f System.ServiceModel.Channels.IOThreadScheduler+CriticalHelper+ScheduledOverlapped.IOCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)
0fc8e990 6193e6dd System.ServiceModel.Diagnostics.Utility+IOCompletionThunk.UnhandledExceptionFrame(UInt32, UInt32, System.Threading.NativeOverlapped*)
0fc8e9c4 6f6bcda4 System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32, UInt32, System.Threading.NativeOverlapped*) 
0fc8eb64 70171b6c [GCFrame: 0fc8eb64]  
0fc8ecb8 70171b6c [ContextTransitionFrame: 0fc8ecb8]

The rest of the post will walk you through the steps on finding out the HttpContext associated with the above thread. We start by running the command !dso (!dumpstackobjects) on this thread and picking up the object of type System.ServiceModel.Channels.IOThreadScheduler+CriticalHelper+WorkItem.

0:045> !dso
OS Thread Id: 0x23b4 (45)
ESP/REG  Object   Name
0fc8e480 05f24538 Service
0fc8e48c 05ae2eb8 System.ServiceModel.Dispatcher.SyncMethodInvoker

0fc8e8f0 05f241d8 System.Security.SecurityContext
0fc8e8fc 05f241c4 System.ServiceModel.Channels.IOThreadScheduler+CriticalHelper+WorkItem
0fc8e900 05f241d8 System.Security.SecurityContext
0fc8e90c 05f241c4 System.ServiceModel.Channels.IOThreadScheduler+CriticalHelper+WorkItem
0fc8e924 01b3a5fc System.Object
0fc8e928 05f241c4 System.ServiceModel.Channels.IOThreadScheduler+CriticalHelper+WorkItem
0fc8e958 01b3a5fc System.Object
0fc8e974 01b3a728 System.ServiceModel.Diagnostics.Utility+IOCompletionThunk
0fc8e990 01b3a728 System.ServiceModel.Diagnostics.Utility+IOCompletionThunk
0fc8e9ac 01b3a688 System.Threading.Overlapped
0fc8eb98 01c1ab84 System.Threading.OverlappedData

Once you find this object, you can dump this out using !do

0:045> !do 05f241c4 
Name: System.ServiceModel.Channels.IOThreadScheduler+CriticalHelper+WorkItem
MethodTable: 79a781a8
EEClass: 796d6aa8
Size: 20(0x14) bytes
GC Generation: 0
(C:\Windows\assembly\GAC_MSIL\System.ServiceModel\3.0.0.0__b77a5c561934e089\System.ServiceModel.dll)
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
6f73ef38  40019b2        4 ...ding.WaitCallback  0 instance 05af63f0 callback
6f7584dc  40019b3        8        System.Object  0 instance 05f15538 state
6f735cf8  40019b4        c ...y.SecurityContext  0 instance 05f241d8 context
6f735204  40019b1      374 ...g.ContextCallback  0   shared   static securityContextCallback
    >> Domain:Value  01235fc0:NotInit  01292638:05aa2278 <<

From here onwards you have a do a series of !do commands till you reach the HttpContext object. This is how the object hierarchy looks like System.ServiceModel.Channels.IOThreadScheduler+CriticalHelper+WorkItem (state) -> SecuritySessionServerSettings+ServerSecuritySessionChannel+ReceiveRequestAsyncResult (innerRequestContext)-> System.ServiceModel.Activation.HostedHttpContext(result) -> System.ServiceModel.Activation.HostedHttpRequestAsyncResult (context) -> System.Web.HttpApplication (_context) -> System.Web.HttpContext So picking up the state field from IOThreadScheduler+CriticalHelper+WorkItem we get SecuritySessionServerSettings+ServerSecuritySessionChannel+ReceiveRequestAsyncResult

0:045> !do 05f15538 
Name: System.ServiceModel.Security.SecuritySessionServerSettings+ServerSecuritySessionChannel+ReceiveRequestAsyncResult
MethodTable: 79a2c2d0
EEClass: 7973d4d8
Size: 72(0x48) bytes
GC Generation: 0
(C:\Windows\assembly\GAC_MSIL\System.ServiceModel\3.0.0.0__b77a5c561934e089\System.ServiceModel.dll)
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
6f73fd5c  40009ae        4 System.AsyncCallback  0 instance 05af63d0 callback
6f7584dc  40009af        8        System.Object  0 instance 05f15428 state
6f75eadc  40009b0       1c       System.Boolean  1 instance        1 completedSynchronously
6f75eadc  40009b1       1d       System.Boolean  1 instance        1 endCalled
6f758a54  40009b2        c     System.Exception  0 instance 00000000 exception
6f75eadc  40009b3       1e       System.Boolean  1 instance        1 isCompleted
6f7424a4  40009b4       10 ....ManualResetEvent  0 instance 00000000 manualResetEvent
6f7584dc  40009b5       14        System.Object  0 instance 05f15580 thisLock
79a77030  40009b6       18 ...viceModelActivity  0 instance 00000000 callbackActivity
79a43d6c  4002dd7       20 ...itySessionChannel  0 instance 05f0e620 channel
79a04894  4002dd8       24 ...ls.RequestContext  0 instance 05f08504 innerRequestContext
79a7b610  4002dd9       28 ....Channels.Message  0 instance 05f15868 requestMessage
79a3c6b0  4002dda       2c ...lCorrelationState  0 instance 05f241b0 correlationState
6f75eadc  4002ddb       1f       System.Boolean  1 instance        0 expired
79a7bbcc  4002ddc       30 ...del.TimeoutHelper  1 instance 05f15568 timeoutHelper
6f73ef38  4002dd5      780 ...ding.WaitCallback  0   shared   static onWait
    >> Domain:Value  01235fc0:NotInit  01292638:05e3066c <<
6f73fd5c  4002dd6      784 System.AsyncCallback  0   shared   static onReceive
    >> Domain:Value  01235fc0:NotInit  01292638:05e306bc <<

0:045> !do 05f08504 
Name: System.ServiceModel.Activation.HostedHttpContext
MethodTable: 79a09b98
EEClass: 79696fcc
Size: 68(0x44) bytes
GC Generation: 0
(C:\Windows\assembly\GAC_MSIL\System.ServiceModel\3.0.0.0__b77a5c561934e089\System.ServiceModel.dll)
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
6f739e70  4000ab9       18      System.TimeSpan  1 instance 05f0851c defaultSendTimeout
6f739e70  4000aba       20      System.TimeSpan  1 instance 05f08524 defaultCloseTimeout
79a432d4  4000abb       10         System.Int32  1 instance        2 state
79a7b610  4000abc        4 ....Channels.Message  0 instance 05f0bbbc requestMessage
6f758a54  4000abd        8     System.Exception  0 instance 00000000 requestMessageException
6f75eadc  4000abe       14       System.Boolean  1 instance        0 replySent
6f75eadc  4000abf       15       System.Boolean  1 instance        0 replyInitiated
6f75eadc  4000ac0       16       System.Boolean  1 instance        0 aborted
6f7584dc  4000ac1        c        System.Object  0 instance 05f08548 thisLock
79a02b04  4000e15       28 ...annels.HttpOutput  0 instance 00000000 httpOutput
79a029ac  4000e16       2c ...hannels.HttpInput  0 instance 05f08554 httpInput
79a03eb0  4000e17       30 ...tpChannelListener  0 instance 05adeaf4 listener
79a8a134  4000e18       34 ...tyMessageProperty  0 instance 00000000 securityProperty
79a09d4c  4000e3a       38 ...dRequestContainer  0 instance 05f0d630 requestContainer
79a095b0  4000e3b       3c ...equestAsyncResult  0 instance 01c40c5c result

0:045> !do 01c40c5c 
Name: System.ServiceModel.Activation.HostedHttpRequestAsyncResult
MethodTable: 79a095b0
EEClass: 79696ae8
Size: 60(0x3c) bytes
GC Generation: 0
(C:\Windows\assembly\GAC_MSIL\System.ServiceModel\3.0.0.0__b77a5c561934e089\System.ServiceModel.dll)
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
6f73fd5c  40009ae        4 System.AsyncCallback  0 instance 01b3a534 callback
6f7584dc  40009af        8        System.Object  0 instance 01c3fba4 state
6f75eadc  40009b0       1c       System.Boolean  1 instance        0 completedSynchronously
6f75eadc  40009b1       1d       System.Boolean  1 instance        0 endCalled
6f758a54  40009b2        c     System.Exception  0 instance 00000000 exception
6f75eadc  40009b3       1e       System.Boolean  1 instance        0 isCompleted
6f7424a4  40009b4       10 ....ManualResetEvent  0 instance 00000000 manualResetEvent
6f7584dc  40009b5       14        System.Object  0 instance 01c40c98 thisLock
79a77030  40009b6       18 ...viceModelActivity  0 instance 00000000 callbackActivity
79a096b0  4000e49       20 ...ersonationContext  0 instance 01c40ca4 impersonationContext
79a300b4  4000e4a       24 ....HostedThreadData  0 instance 00000000 hostedThreadData
6580e674  4000e4b       28 ...b.HttpApplication  0 instance 01c3d1f4 context
6ee9fb70  4000e4c       2c           System.Uri  0 instance 05f08034 originalRequestUri
6ee9fb70  4000e4d       30           System.Uri  0 instance 05f081f0 requestUri
6f75ab0c  4000e4e       34         System.Int32  1 instance        0 state
6f75eadc  4000e4f       1f       System.Boolean  1 instance        0 flowContext
6f765e44  4000e43      178 ...l.WindowsIdentity  0   shared   static anonymousIdentity
    >> Domain:Value  01235fc0:NotInit  01292638:NotInit  <<
6f73ef38  4000e44      17c ...ding.WaitCallback  0   shared   static waitOnBeginRequest
    >> Domain:Value  01235fc0:NotInit  01292638:NotInit  <<
6f73ef38  4000e45      180 ...ding.WaitCallback  0   shared   static waitOnBeginRequestWithFlow
    >> Domain:Value  01235fc0:NotInit  01292638:NotInit  <<
6f735204  4000e46      184 ...g.ContextCallback  0   shared   static contextOnBeginRequest
    >> Domain:Value  01235fc0:NotInit  01292638:NotInit  <<
6f73fd5c  4000e47      188 System.AsyncCallback  0   shared   static processRequestCompleteCallback
    >> Domain:Value  01235fc0:NotInit  01292638:NotInit  <<
6f73c368  4000e48        0 ...ng.AutoResetEvent  0   shared TLstatic waitObject
    >> Thread:Value 26b8:01b3a4b8 2740:01b5e4fc 1e10:01b6dca4 6c8:01bdffe0 724:01be7948 188c:01c00938 240c:01c1298c 13d0:01c18964 2260:01c3fba4 20fc:01c65bd0 <<

0:045> !do 01c3d1f4 
Name: System.Web.HttpApplication
MethodTable: 6580e674
EEClass: 65610c98
Size: 116(0x74) bytes
GC Generation: 0
(C:\Windows\assembly\GAC_32\System.Web\2.0.0.0__b03f5f7f11d50a3a\System.Web.dll)
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
6580e85c  4000dde        4 ...pApplicationState  0 instance 01afaac4 _state
6580cf64  4000ddf        8 ...m.Web.HttpContext  0 instance 00000000 _initContext
6580eba0  4000de0        c ...b.HttpAsyncResult  0 instance 01c407d0 _ar
6580eb14  4000de1       10 ...pModuleCollection  0 instance 01c3d494 _moduleCollection
6eea00a0  4000dfc       14 ....EventHandlerList  0 instance 01c3e300 _events
658142c4  4000dfd       18 ...ventHandlersTable  0 instance 01c3e40c _asyncEvents
65815b14  4000dfe       1c ...ation+StepManager  0 instance 01c3f1e4 _stepManager
6f73ef38  4000dff       20 ...ding.WaitCallback  0 instance 01c3f1c4 _resumeStepsWaitCallback
6f735504  4000e00       24     System.EventArgs  0 instance 01b34924 _appEvent
6f75af14  4000e01       28 ...ections.Hashtable  0 instance 01c3d268 _handlerFactories
6f75a8f0  4000e02       2c ...ections.ArrayList  0 instance 00000000 _handlerRecycleList
6f75eadc  4000e03       68       System.Boolean  1 instance        0 _hideRequestResponse
6580cf64  4000e04       30 ...m.Web.HttpContext  0 instance 01c404c8 _context
6f758a54  4000e05       34     System.Exception  0 instance 00000000 _lastError
6f75eadc  4000e06       69       System.Boolean  1 instance        1 _timeoutManagerInitialized



0:045> !do 01c404c8 
Name: System.Web.HttpContext
MethodTable: 6580cf64
EEClass: 6560fff8
Size: 192(0xc0) bytes
GC Generation: 0
(C:\Windows\assembly\GAC_32\System.Web\2.0.0.0__b03f5f7f11d50a3a\System.Web.dll)
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
6580e714  4000fe6        4 ...IHttpAsyncHandler  0 instance 01c3d1f4 _asyncAppHandler
6580e674  4000fe7        8 ...b.HttpApplication  0 instance 01c3d1f4 _appInstance
6580e754  4000fe8        c ....Web.IHttpHandler  0 instance 00000000 _handler
6580e9a4  4000fe9       10 ...m.Web.HttpRequest  0 instance 01c40588 _request
6580ea04  4000fea       14 ....Web.HttpResponse  0 instance 01c40634 _response
6580eac4  4000feb       18 ...HttpServerUtility  0 instance 01c40cb8 _server
6f737c24  4000fec       1c ...Collections.Stack  0 instance 00000000 _traceContextStack
658115e0  4000fed       20 ....Web.TraceContext  0 instance 00000000 _topTraceContext
6f75af14  4000fee       24 ...ections.Hashtable  0 instance 00000000 _items
6f75a8f0  4000fef       28 ...ections.ArrayList  0 instance 00000000 _errors
6f758a54  4000ff0       2c     System.Exception  0 instance 00000000 _tempError
6f75eadc  4000ff1       90       System.Boolean  1 instance        0 _errorCleared
6f740378  4000ff2       30 ...ncipal.IPrincipal  0 instance 01b34984 _user
6f75b188  4000ff3       80        System.IntPtr  1 instance        0 _pManagedPrincipal
6580e3dc  4000ff4       34 ...ofile.ProfileBase  0 instance 00000000 _Profile
6f739f0c  4000ff5       a0      System.DateTime  1 instance 01c40568 _utcTimestamp
65810df4  4000ff6       38 ...HttpWorkerRequest  0 instance 01c3fea0 _wr
6f75bd2c  4000ff7       a8 ...Services.GCHandle  1 instance 01c40570 _root
6f75b188  4000ff8       84        System.IntPtr  1 instance 10490900 _ctxPtr
65809b78  4000ff9       3c ...m.Web.VirtualPath  0 instance 00000000 _configurationPath
6f75eadc  4000ffa       91       System.Boolean  1 instance        0 _skipAuthorization
6f75b4bc  4000ffb       40 ...ation.CultureInfo  0 instance 00000000 _dynamicCulture
6f75b4bc  4000ffc       44 ...ation.CultureInfo  0 instance 00000000 _dynamicUICulture
6f75ab0c  4000ffd       88         System.Int32  1 instance        0 _serverExecuteDepth
6f737c24  4000ffe       48 ...Collections.Stack  0 instance 00000000 _handlerStack

This context is the actual HttpContext reported inside the !aspxpages pages and using this technique you can associate the WCF request processing threads with the ASP.NET worker thread that handled the request in the first place, so going back to the output given by !aspxpages pages, we can see that the HttpContext (01c404c) to the below entry

0x01c404c8 11647 Sec no 25 Sec 41 200 POST /MySleepingWcfService/Service.svc

 

So this concludes that the HttpContext 01c404c8 for which the ASP.NET worker thread is 41 is actually getting processed on thread 45. The steps are a little lengthy but at times you really need to find this information. You can even write a debugger script to automate this whole process. HAPPY DEBUGGING !!!

EventId 4196 - Troubleshooting Crashes in COM+ Applications

PuneetG — Sat, 03 Jul 2010 09:11:22 GMT

Typically we expect that a piece of code that works once should work ALWAYS but this is not as simple as it sounds. Even though you have tested your component or application enough, it is not always possible to test all the codepaths and all sorts of situations that can arise in an application. As a result of this you may end up seeing intermittent or unexpected crashes in your application. Today we will learn about collecting some valuable information for unexpected crashes that can happen in COM+ components and how to deal with them. What is a Crash? – The term CRASH is used by different people in various different ways. In our(or my) definition a crash is an unexpected termination of a process as result of an unhandled exception in the code. If COM+ encounters an unhandled exception in the component code the COM+ Runtime terminates the process immediately and logs the following event in the event viewer letting the user know that a crash has happened. This is called COM+ Failfast This particular behavior of COM+ terminating the process on an unexpected error condition is termed as COM+ Failfast. This blog will list out the steps required to troubleshoot a COM+ Failfast and it can be summarized as:-

Generating the Symbols for the application
Configuring the debugger to collect dumps on the right exception
Collecting the dump file
Analyzing the dump using a windows debugger.
Making relevant code changes to correct the problem.

Now we will look at each of the above steps in detail To walk you through the process, I wrote a COM component in VB 6.0 and configured it in COM+. I intentionally have some bad code in my component which is causing the component to Crash. I am calling the component from a very simple VBSCRIPT file like this.

Dim MathComponent
Set MathComponent = Createobject("Arithmetic.Maths")
wscript.echo = MathComponent.DoSomeCalculation(1,2,3)

On running the above script file, I can see a crash in the DLLHOST process and the following event is logged in the event viewer.

Event Type: Error Event Source: COM+ Event Category: Unknown Event ID: 4786 Date: 7/3/2010 Time: 10:56:45 AM User: N/A Computer: TURTLE86 Description: The system has called a custom component and that component has failed and generated an exception. This indicates a problem with the custom component. Notify the developer of this component that a failure has occurred and provide them with the information below. Component Prog ID: Server Application ID: {9E761F3D-4C33-4158-9EC4-B3C5831D346F} Server Application Instance ID: {002B45BE-CAC5-4E9B-8811-B21EDD73B278} Server Application Name: MathsApplication The serious nature of this error has caused the process to terminate. Exception: C0000008 Address: 0x7C8285F3 Call Stack: ntdll!KiRaiseUserExceptionDispatcher + 0x37 ntdll!KiFastSystemCallRet + 0x0 Arithmetic!DllCanUnloadNow + 0x272 Arithmetic!DllCanUnloadNow + 0x1c4 Arithmetic!DllCanUnloadNow + 0x117 Arithmetic!DllCanUnloadNow + 0x6a OLEAUT32!DispCallFunc + 0xab MSVBVM60!__vbaAptOffset + 0x68b MSVBVM60!BASIC_CLASS_Invoke + 0x65 OLEAUT32!DispGetParam + 0x718 OLEAUT32!DllRegisterServer + 0x5bf RPCRT4!NdrAsyncServerCall + 0x1e7 RPCRT4!CStdStubBuffer_Invoke + 0x82 OLEAUT32!DispGetParam + 0x5b6 ole32!StgGetIFillLockBytesOnFile + 0x13d32 ole32!StgGetIFillLockBytesOnFile + 0x13cdf ole32!DcomChannelSetHResult + 0xaab ole32!DcomChannelSetHResult + 0x495 ole32!CoFreeUnusedLibrariesEx + 0xb06 ole32!StgGetIFillLockBytesOnFile + 0x13bec ole32!StgGetIFillLockBytesOnFile + 0x13a7d ole32!StgGetIFillLockBytesOnFile + 0x12f64 ole32!CoFreeUnusedLibrariesEx + 0x9f5 ole32!CoFreeUnusedLibrariesEx + 0x9c0 USER32!LoadCursorW + 0x4cf5 USER32!LoadCursorW + 0x4e86 USER32!TranslateMessageEx + 0x10d USER32!DispatchMessageW + 0xf COMSVCS!DllUnregisterServer + 0x270 COMSVCS!DllUnregisterServer + 0x180 COMSVCS!DllUnregisterServer + 0xc6c COMSVCS!DllUnregisterServer + 0xf4d msvcrt!_endthreadex + 0xa3 kernel32!GetModuleHandleA + 0xdf

The above event entry also contains the callstack information for the thread on which the unhandled exception is thrown but there is just a small problem with the callstack that is shown in the Event. The callstack may or may not show you the right function names because of missing symbols. For e.g. the stack is showing me that functions from my DLL Arithmetic.dll (highlighted in blue) were called but it just showing one function name DllCanUnloadNow where as in my VB code I never wrote a function called DllCanUnloadNow(). So what is the DllCanUnloadNow function. Well that is what the debugger is able to match the closest based on the default export function table of a DLL and it needs symbols to translate the memory offsets to function names. Now what exactly is a symbol ??? A word about symbols !!! It is very critical for us to know the exact function where the problem happened (and if possible the exact line of code) and we can find that information only if we have the right symbols for our DLLs. Symbols (or pdb files) are binary files that are generated by compilers during code generation which help you translate the function offsets in a module to the RIGHT function names. More information on symbols and their importance can be found on this link and thousands of other articles and blogs available on the internet. In older versions of Visual Studio (as old as Visual Studio 6.0), symbols were NOT generated for DLL’s \EXE’s by default. I just want to provide the steps needed to generate the PDB files for VB 6.0 and VC 6.0 applications.

To generate the symbols for DLL’s written in Visual Basic 6.0

Open the Visual Basic Project in Visual Studio 6.0.

From the Project menu option select the Properties… option.

Go to the third tab (Compile) and check the Checkbox for Create Symbolic Debug Info

After selecting this, Recompile the project. This time you would see a file with a .PDB extension generated with the same name as the name of the DLL and in the same folder where the DLL is generated.

To generate the symbols for a VC++ DLL

Open the Project in Visual Studio.

Go to project properties.

Under the Linker Section, select the Debugging Sub Section and select YES for Generate Debug Info.

Rebuild the project.

After generating the symbols for your component, it is very important that you redeploy the newer DLL that got generated to the server because for a debugger to align the symbols properly the timestamp of the PDB and the timestamp of the DLL has to match. The next thing to do after deploying symbols is to use a Windows native debugger to capture a dump on the exception of interest and debug it using the symbols that we generated. There are some native debuggers available for this purpose (like WINDBG, CDB, NTSD etc.) but I will talk about the Debug Diagnostic Tool which is easy to use and understand. You can get the Debug Diagnostic tool from Microsoft download center and install it on the server where you are seeing the crash. Configuring the Debugger Debug Diagnostic tool lets you generate as well analyze the dumps. This tool provides you a friendly GUI that can be configured to take dumps of any process. You can follow these steps to configure dumps for the COM+ Application which is crashing.

Launch the tool by going to Start- > Programs -> Debug Diagnostic Tool 1.1 ->DebugDiag 1.1
In the Select Rule Type action, select Crash and click Next
In the Select Target Type section, select A specific MTS/COM+ Application.
From the list select the COM+ Application which is experiencing the Crash.
In the Advanced Confirmation section, click on Breakpoints and then click Add Breakpoint
Select ComSvcs!ComSvcsExceptionFilter (I will talk about this shortly) and change the Action Type to Full User Dump. You can change the Action Limit to 5 which configures the tool to take 5 dumps and then complete the rule.
Click Ok and click Save and Close
Click Next and in the Select Dump Location and Rule Name dialog, you can configure a friendly rule name and specify a location on the hard drive where the dump will be generated.
Click Next and Finish to Activate the rule. Once the wizard completes the rule should come in the Rules section with a status of Active and should look like this.

For COM+ Applications, it is better to configure the ComSvcs!ComSvcsExceptionFilter breakpoint because that function implements the COM+ Failfast logic. Configuring the breakpoint on this function ensures that we are getting the dumps at the right time. Collecting the Dumps At this point you just have to wait for the crash to happen again. If this is an easily reproducible issue, you may just go ahead and reproduce it once so that the debugger is able to take the dumps. Whenever the dumps are collected by this tool, you will notice the UserDump Count variable incremented by 1. There may be multiple dumps generated by the tool based on what you configured it for but the one you are most interested to look at is the one which contains ComSvcsExceptionFilter in the name. The dump file that got generated on my machine as a result of this tool had the name dllhost__PID__6492__Date__07_03_2010__Time_11_56_44AM__725__ComSvcs!ComSvcsExceptionFilter.dmp Analyzing the Dump File Now starts the fun part :-). As I mentioned before, you can use any native Windows Debugger to analyze the dump file but to keep things simple again, we can use the built in analyzer of Debug Diagnostic tool to view the relevant information about the dump file. To analyze the dump using DebugDiag, you just double click on the .dmp file that you generated in the previous step which will launch the DebugDiag analyzer automatically or you can follow these steps.

Open Debug Diagnostic tool.
Go to Advanced Analysis tab
Click on Add Data Files
Browse to the dump file generated in the previous step and click on Start Analysis.

The moment you click on Start Analysis, the Debug Diagnostic tool will start downloading symbol files from the Microsoft Symbol server for the relevant DLLs and after running some expert analysis on the dumps, it will launch an Internet Explorer window with the HTML report of the analysis. So running the analysis on my dump file showed me this in the DebugDiag report.

Analysis Summary

Type	Description	Recommendation
Error	The following threads in dllhost__PID__6492__Date__07_03_2010__Time_11_56_44AM__725__ComSvcs!ComSvcsExceptionFilter.dmp are blocked by an unhandled exception which caused a COM+ FailFast to occur.( 13 ) 4.55% of threads blocked	Please see the Recovered Call Stack for thread 13 based on the restored exception and context record passed to the exception filter.

If you click on proper thread number in the above report, you can actually see the callstack of the thread which caused the exception and notice how nicely the report is showing you the recovered callstack for the exactly exception. In my case, I see the following callstack

Recovered stack for thread 13

Function	Arg 1	Arg 2	Arg 3
ntdll!KiRaiseUserExceptionDispatcher+37	7c826d49	77e63eb3	00000006
ntdll!KiFastSystemCall+3	77e63eb3	00000006	0114eacc
ntdll!ZwClose+c	00000006	0114eacc	11001d14
kernel32!CloseHandle+59

<><>

00000006

000d6df8

000b9f98

Arithmetic!DllCanUnloadNow+272

000d4cb8

00000001

00000002

Arithmetic!DllCanUnloadNow+1c4

000d4cb8

00000001

00000002

Arithmetic!DllCanUnloadNow+117

000d4cb8

00000001

00000002

Arithmetic!DllCanUnloadNow+6a

000d4cb8

00000001

00000002

oleaut32!DispCallFunc+16a

000d4cb8

0000001c

00000004

msvbvm60!VBStrToLong+cf

000d4cb8

11001320

60030000

msvbvm60!rtFindFirstFile+185

000d4cb8

60030000

000d63dc

oleaut32!IDispatch_Invoke_Stub+52

000d4cb8

60030000

000d63dc

oleaut32!IDispatch_RemoteInvoke_Thunk+5b

0114f650

000d52e0

000ba4e8

rpcrt4!NdrStubCall2+214

000d52e0

000bf458

000ba4e8

rpcrt4!CStdStubBuffer_Invoke+c6

000d52e0

000ba4e8

000bf458

oleaut32!CStubWrapper::Invoke+9f

000c14f8

000ba4e8

000bf458

ole32!SyncStubInvoke+37

000ba4e8

000ab790

000bf9d0

ole32!StubInvoke+a7

000ba4e8

000b9e48

000c14f8

ole32!CCtxComChnl::ContextInvoke+ec

000bf458

00000000

000c14f8

ole32!MTAInvoke+1a

000ba4e8

00000001

000c14f8

ole32!STAInvoke+48

000ba4e8

00000001

000c14f8

ole32!AppInvoke+a3

d0908070

000bf458

000c14f8

ole32!ComInvokeWithLockAndIPID+2c5

000ba490

000a1350

00000400

ole32!ComInvoke+ca

000ba490

00000400

000c71f8

ole32!ThreadDispatch+23

000ba490

0114fe0c

776bc27c

ole32!ThreadWndProc+fe

000b9b70

00000400

0000babe

user32!InternalCallWinProc+28

776bc27c

00550f40

00000400

user32!UserCallWinProcCheckWow+151

00000000

776bc27c

00550f40

user32!DispatchMessageWorker+327

000cbdd4

00000000

0114fecc

user32!DispatchMessageW+f

000cbdd4

000cbdd0

000cbd40

comsvcs!CSTAQueueLessMessageWork::DoWork+4e

000cbd60

000cbd40

000cbdc4

comsvcs!CSTAThread::DoWork+18

000cbdd0

00000001

000cbd40

comsvcs!CSTAThread::ProcessQueueWork+37

00000000

00037918

00037a48

comsvcs!CSTAThread::WorkerLoop+190

000cbd40

00000000

msvcrt!_endthreadex+a3

00037a48

00000000

kernel32!BaseThreadStart+34

77bcb4bc

00037a48

00000000

So all that is fine but why am I seeing DllCanUnloadNow and not the right function name? The reason for that is that we have not configured Debug Diagnostic to load the right symbols for my DLL. This configuration in Debug Diagnostic tool is done under the Tools menu by selecting Options and Settings. These are the default settings. All you need to do is append the path to the folder where the PDB files for your component are to “Symbol Search Path for analysis”. So in my case, the PDB files are present inside E:\COMProjects\ArithmeticComponent folder, so my Symbol Search Path for analysis will become srv*c:\symcache*http://msdl.microsoft.com/downloads/symbols;E:\COMProjects\ArithmeticComponent Now if I run Debug Diagnostic tool again on the same report, I can see the functions reported in a better way but before doing that I would like to turn on one more setting under the Preferences tab under Options and Setting and that is selecting “Include source and line information in analysis reports” After making these changes I re-run my report and now if I look at the Recovered Callstack for Thread 13, I can see the right function names

Recovered stack for thread 13

Function	Arg 1	Arg 2	Arg 3	Source
ntdll!KiRaiseUserExceptionDispatcher+37	7c826d49	77e63eb3	00000006
ntdll!KiFastSystemCall+3	77e63eb3	00000006	0114eacc
ntdll!ZwClose+c	00000006	0114eacc	11001d14
kernel32!CloseHandle+59	00000006	000d6df8	000b9f98
Arithmetic!Maths::CallThirdFunction+59	000d4cb8	00000001	00000002	E:\COMProjects\ArithmeticComponent\Maths.cls @ 37 + 19
Arithmetic!Maths::CallSecondFunction+58	000d4cb8	00000001	00000002	E:\COMProjects\ArithmeticComponent\Maths.cls @ 29 + 18
Arithmetic!Maths::CallFirstFunction+58	000d4cb8	00000001	00000002	E:\COMProjects\ArithmeticComponent\Maths.cls @ 24 + 18
Arithmetic!Maths::DoSomeCalculation+58	000d4cb8	00000001	00000002	E:\COMProjects\ArithmeticComponent\Maths.cls @ 19 + 18
oleaut32!DispCallFunc+16a	000d4cb8	0000001c	00000004
msvbvm60!VBStrToLong+cf	000d4cb8	11001320	60030000
msvbvm60!rtFindFirstFile+185	000d4cb8	60030000	000d63dc
oleaut32!IDispatch_Invoke_Stub+52	000d4cb8	60030000	000d63dc
oleaut32!IDispatch_RemoteInvoke_Thunk+5b	0114f650	000d52e0	000ba4e8
rpcrt4!NdrStubCall2+214	000d52e0	000bf458	000ba4e8
rpcrt4!CStdStubBuffer_Invoke+c6	000d52e0	000ba4e8	000bf458
oleaut32!CStubWrapper::Invoke+9f	000c14f8	000ba4e8	000bf458
ole32!SyncStubInvoke+37	000ba4e8	000ab790	000bf9d0
ole32!StubInvoke+a7	000ba4e8	000b9e48	000c14f8
ole32!CCtxComChnl::ContextInvoke+ec	000bf458	00000000	000c14f8
ole32!MTAInvoke+1a	000ba4e8	00000001	000c14f8
ole32!STAInvoke+48	000ba4e8	00000001	000c14f8
ole32!AppInvoke+a3	d0908070	000bf458	000c14f8
ole32!ComInvokeWithLockAndIPID+2c5	000ba490	000a1350	00000400
ole32!ComInvoke+ca	000ba490	00000400	000c71f8
ole32!ThreadDispatch+23	000ba490	0114fe0c	776bc27c
ole32!ThreadWndProc+fe	000b9b70	00000400	0000babe
user32!InternalCallWinProc+28	776bc27c	00550f40	00000400
user32!UserCallWinProcCheckWow+151	00000000	776bc27c	00550f40
user32!DispatchMessageWorker+327	000cbdd4	00000000	0114fecc
user32!DispatchMessageW+f	000cbdd4	000cbdd0	000cbd40
comsvcs!CSTAQueueLessMessageWork::DoWork+4e	000cbd60	000cbd40	000cbdc4
comsvcs!CSTAThread::DoWork+18	000cbdd0	00000001	000cbd40
comsvcs!CSTAThread::ProcessQueueWork+37	00000000	00037918	00037a48
comsvcs!CSTAThread::WorkerLoop+190	000cbd40	00000000	00000000
msvcrt!_endthreadex+a3	00037a48	00000000	00000000
kernel32!BaseThreadStart+34	77bcb4bc	00037a48	00000000

If you read carefully, I am even getting the line numbers in my Visual Basic .CLS file which tell me the exact line of code where the problem happened. Now isn't that cool !!! Here is the complete code of my sample component

Private Declare Function CloseHandle Lib "kernel32" (ByVal hObject As Long) As Long Public Function DoSomeCalculation(ByVal number1 As Integer, ByVal number2 As Integer, ByVal number3 As Integer) As Integer DoSomeCalculation = CallFirstFunction(number1, number2, number3) End Function Function CallFirstFunction(ByVal number1 As Integer, ByVal number2 As Integer, ByVal number3 As Integer) As Integer CallFirstFunction = CallSecondFunction(number1, number2, number3) End Function Function CallSecondFunction(ByVal number1 As Integer, ByVal number2 As Integer, ByVal number3 As Integer) As Integer CallSecondFunction = CallThirdFunction(number1, number2, number3) End Function Function CallThirdFunction(ByVal number1 As Integer, ByVal number2 As Integer, ByVal number3 As Integer) As Integer Dim sum As Integer Dim returnValue As Long returnValue = CloseHandle(number1 + number2 + number3) CallThirdFunction = returnValue End Function

So you can see that I am just passing a bad value (1 + 2 + 3 = 7) to the Kernel32's CloseHandle API which ends up throwing the "Invalid Handle" exception. The same steps can be used for troubleshooting crashes in COM+ Applications written in any programming language provided you generate the right symbols and you configure the debugger to point to the right symbols. I am attaching the complete report to the end of this blog post so that you can review what else is logged in the report. HAPPY DEBUGGING !!! CrashHang_Report__PID_6492__0716201016530581.zip

DebugDiag script to load all symbols in a dump file

PuneetG — Thu, 13 May 2010 17:52:00 GMT

At times you really need to load symbols for all modules loaded in a dump either because you want to put a breakpoint on some function (which is not exported in a module or you want) or to see the right function names in call-stacks logged in the DebugDiag's log file for debugging purposes.

The existing Crash hang Analysis scripts load symbols for modules loaded in the process but they just don't load ALL the symbols. They only load symbols for modules that required to display in the report. Also the memory analysis scripts load symbols for modules which are required for looking at the memory allocations but the below script loads modules for ALL symbols loaded in the process

<%@ Language = VBScript %>

<%@ Category = Load Symbols %>

<%@ Description = Loading symbols for all modules in the process %>

<%@ TopLevelScript = True %>

<% Option Explicit %>

<%

Const ANALYSIS_STEP_COUNT = 10

Dim g_Debugger, g_DataFile, g_Progress,g_OverallProgress,intPosition, intLen,g_ShortDumpFileName

Set g_Progress = Manager.Progress

g_Progress.SetOverallRange 0, Manager.DataFiles.Count * ANALYSIS_STEP_COUNT

g_Progress.OverallPosition = 0

g_Progress.OverallStatus = ""

g_Progress.CurrentStatus = ""

g_Progress.CurrentPosition = 0

g_OverallProgress = 0

g_Progress.OverallStatus = "Starting Analysis"

' Main analysis and reporting loop. One iteration for each dump file

Manager.ReportInformation "This script is used to load symbols for all modules loaded in the dump file"

For Each g_DataFile In Manager.DataFiles

    If (UCase(Right(g_DataFile,4)) = ".DMP") Then

        Set g_Debugger = Manager.GetDebugger(g_DataFile)

        If Not g_Debugger Is Nothing Then

            'Trim g_DataFile down to just the filename itself for later use

            intPosition = InStrRev(g_DataFile, "\", -1, 1)

            intLen = Len(g_DataFile)

            g_ShortDumpFileName = Right(g_DataFile, intLen - intPosition)

            'Go ahead and get any extension objects we'll need later

            g_Progress.OverallStatus = "Loading symbols for dump " & g_ShortDumpFileName

            g_Debugger.Execute("!symfix c:\symcache")

            g_Debugger.Execute(".reload")

            g_Debugger.Execute("ld*")

            Manager.Write " Finished Loading symbols for the dump file - " & g_DataFile & ""

        Else

            Manager.ReportError "Unable to open the file " & g_DataFile & " for analysis.", "The file is probably " &_

             "corrupt, and a new dump file of the targeted process will have to be created to do any analysis."

        End If

    Manager.CloseDebugger g_DataFile

    End If

Next

%>

Just copy the above code and create a new file in C:\Program Files\DebugDiag\Scripts folder and call it LoadSymbols.asp (or download the LoadSymbols.zip file attached to this blog). Once done, the Debug Diagnostic tool UI will show you this script and you can choose dump files whole modules you want to load in the local cache and click on Start Analysis.

Once the script finishes you should see all the symbols downloaded in the the c:\symcache folder.

Also notice that using the same way you can write whatever debug scripts you want for analyzing your dump. In the above script there is a for loop which is iterating through every dump file that the user selects in the DebugDiag tool UI.

For Each g_DataFile In Manager.DataFiles

    If (UCase(Right(g_DataFile,4)) = ".DMP") Then

        Set g_Debugger = Manager.GetDebugger(g_DataFile)

        If Not g_Debugger Is Nothing Then

            g_Debugger.Execute("vertarget")

        Else

            Manager.ReportError "Unable to open the file " & g_DataFile & " for analysis.", "The file is probably " &_

             "corrupt, and a new dump file of the targeted process will have to be created to do any analysis."

        End If

    Manager.CloseDebugger g_DataFile

    End If

Next

The light highlighted in yellow above shows a call to g_Debugger.Execute() which executes a debugger command and returns you the output of that command. For e.g. the above line will call vertarget on the dump file. For more information on this method, go through the documentation (DebugDiag.chm) that comes with the DebugDiag tool and look for the DbgObj object.

LoadSymbols.zip

Where is the Client Certificates UI in IIS 7.0 ?

PuneetG — Mon, 03 Aug 2009 09:42:00 GMT

One of the things which I found missing in the IIS 7.0 user interface was the UI which lets you configure the Client certificate mappings for IIS. In the present world, if you want to configure the client certificate mappings for a web site, you will have to use the configuration editor and specify the certificate blob and configure the client certificate as per this blog. At times, getting the certificate blog in a single line itself can get tricky and it might help to just have a UI which IIS 6.0 for this configuration.

Though the client certificate mapping UI feature was not present in IIS 7.0, the IIS 7.0 UI extensibility is so powerful that it is easy to write something like that. I went through some of the blogs that had samples for UI Extensibility and was able to create a sample UI module that adds this functionality in the UI.

Once you install the MSI (for right CPU), you will be able to see this UI module in the IIS manager. This is how it shows up in the UI.

I tried ensuring that the UI for client certificate configuration looks exactly the same as configuration interface in IIS 6.0 so if you are already familiar with configuring the client certificate mappings in IIS 6.0, you should be able to directly use this and configure the client certificate mappings quickly.

Download Location

Find the zip file attached below to this blog and run the right version (x86 or x64) on your server to install this module.

What does the installer do?

The installer copies a file ClientCertificateMapping.dll to inetsrv\ClientcertificateMapping folder on your IIS server.
It registers the ClientCertificateMapping.dll in the GAC on the server.
It modifies the application host.config and adds the two statements in there.

Under the section

And under the section

Note: This tool will just let you configure the client certificate mappings. To make sure that the client certificate mappings works as expected, you might have to review the other settings that are are mentioned in the following links.

Some of the limitations of this UI module (as I see today)

I just managed to write the functionality for one-one client certificate mapping. To be very frank, I have never seen many people using the one-to-many mapping feature anyways. But based on the user-feedback, I may spend some time writing up the UI for many-to-one mapping as well.
I wasn't able to do too much testing around this functionality. If you come across any issues, do let me know via this blog entry
There is no BROWSE button when you click the Add Mapping button. I researched a little bit and came to know that generating a object browser dialog box and using the IDsObjectPicker interface in .net requires too much of code. I found some very good sample at codeplex but realized that it is too much of code and it requires a lot of Interop and Pinvoke code. For the time being, I have not used this functionality and let the user explicitly type in the user name but if this turns out to be a requirement, I can easily add it later.
Any other issues you facing using this, please let me know :)

In the end, I just wanted to share some of the great resources that I came across while writing this module (and creating an installer for the same)

These links are specifically for creating the installer for this UI module

ClientCertificateMapping.zip

The hidden tool - MSDTC Transaction Tracing

PuneetG — Fri, 06 Feb 2009 09:16:50 GMT

If you deal with MSDTC frequently, you might have noticed an option to toggle the tracing options for transactions in the MSDTC settings but I am not sure how many are really aware of what information is logged inside the trace file and how the trace file can be converted from a binary format to a text format. The built in transaction tracing is one of the least famous (but yet a very powerful) feature of MSDTC diagnostics. This tracing logs information about various stages of transaction propagation and gives you enough information about the various outcomes of a transaction. Using this tracing, you can not only determine the various transaction managers or resource managers involved in a transaction but you can also determine the cause of a transaction abort. The transaction tracing becomes a powerful tool when the transactions are either taking too long to run or they are getting aborted intermittently and nothing inside your application is giving you any hint around why the transaction is getting aborted. If you observe one of the following errors inside your application, I recommend using the MSDTC transaction tracing to figure out more details about your transaction

The Microsoft Distributed Transaction Coordinator (MS DTC) has cancelled the Distributed Transaction
The transaction has been implicitly committed or aborted

By default, only the long-lived transactions are traced. You can turn ON tracing for all transactions or aborted transactions based on the kind of issue you are troubleshooting. You just have to select the relevant checkboxes in the trace settings and tracing will take effect (no restart of MSDTC service is required to toggle these settings) The next important thing becomes collecting the trace in the right manner. It is always better to collect a trace which just contains information about the problem that you are troubleshooting because on a busy server the Transaction trace can contain a lot of transactions and it might become very difficult to find out which transaction outcome you are interested in. So this is how we like to collect a DTC Trace.

Open up COMPONENT SERVICES and go to the properties of My Computer. Under the MSDTC tab, click on Tracing options. After that, click on Stop Session and then click on New Session button. This will generate a new file inside c:\windows\msdtc\trace folder with name like dtctrace.log.2008-12-03-11-04-52-0226-00. So this file would basically contain information about all the transactions which have already been traced up to this point and now a new session of MSDTC trace has been started which will start capturing the Tracing information as soon as you click New Session.
Reproduce the problem inside the application.
Now go back to the same dialog and click on Stop Session and New session. This will generate a trace file which should contain only the information about the transactions which were running while the issue was being reproduced. If your application is the only application using DTC, you can expect this log file to contain information about your transactions and analyzing the trace would be an easy job. If there are other transactions which were also running while the trace was turned ON, those will also be logged inside the file and looking at the transaction information you will have to figure out which transactions were initiated by your application.

The MSDTC tracing is basically built on the ETW Tracing for windows and like every other ETW trace, it is a binary file which needs to be parsed using some tools. Once you have got the DTC trace log file, you have to use two utilities inside the Windows XP Service pack 2 Support Tools (Tracefmt.exe and traceprt.dll) to parse the trace file. The sad part is that to get these utilities, you need a Windows XP machine because you can only install the Windows XP Service pack 2 Support Tools on a Windows XP machine. But after installing the setup (make sure to choose the complete version of the setup), you can copy the Tracefmt.exe and traceprt.dll from “c:\Program Files \Support Tools” directory to any operating system. To keep things easier, I always copy these two files to c:\windows\system32\msdtc\trace folder. The command used to parse the trace file is

msdtcvtr –tracelog –o OutPutFile

Running the above command will parse the binary trace file and generate a text file with the name OutputFile.csv which can be opened up in any text editor. Important things to note regarding MSDTCVTR command

MSDTCVTR batch file launches TRACEFMT.exe in the background passing the name of the DTCTRace.log file as one of the parameters. The TraceFMT.exe in turn uses tracerpt.dll to parse the trace file.
TRACEFMT uses a MOF file called msdtctr(present in the same (C:\Windows\SYSTEM32\Msdtc\Trace) folder to parse the trace. This MOF file varies from operating system to operating system. If you run the MSDTCVTR command without the MOF switch, the MSDTCVTR.BAT file automatically picks up the C:\Windows\SYSTEM32\Msdtc\Trace\msdtctr.mof file and parses the trace as per this MOF. This means that if you collect a trace from a Windows 2003 machine and try to parse it on (let’s say), a Windows XP machine, the parsed output may be blank. To fix this problem, you can copy both the MSDTCTR.MOF file along with the DTC Trace file, that you collected earlier, to the machine where you are going to parse the file and while running the MSDTCVTR command, pass a -mof switch to obtain the right output.
The time shown in the parsed output is as per the time zone of the machine where you “parsed” the trace file. This can become a bit confusing if the machine where the trace is collected and the machine where the trace is parsed are in a different time zone. So let’s say, you collected DTC trace on a machine which is in the Central Time zone and you are parsing the trace file on a machine which is in GMT time zone, the time field in the parsed output will be in the GMT format.

A Quick Walkthrough To demonstrate the right way to collect a trace and to parse the same, I created a sample application using System.Transactions in .NET 2.0 and I am running the following code The code is pretty simple. I am just running a Select statement inside a TransactionScope and when the scope is disposed, I am expecting a Transaction to commit. But what I notice is that after I run the above code, my transaction is getting ABORTED (Anyone familiar with System.Transactions should be able to figure out what’s wrong in the above code snippet which ending up in Transaction abort ) So under the transaction statistics in component services where as I expected the Transaction to commit, I can see that the transaction aborted. Now being the application developer I might think that MSDTC is doing something wrong to my transaction. Well let’s just see what is happening in the background by turning ON some tracing. So I went inside component services and pulled up the MSDTC Tracing options and configured the tracing to trace for aborted transactions in this way After doing that I clicked Stop Session and New session to ensure that we have a new tracing session which will log all the transaction activity from now onwards. The moment I did this, a new file is generated in the c:\windows\system32\msdtc\trace folder. At this point, we will just ignore this newly created file as it can contain information about other transactions which might have ran in the past in which we are not interested. After this I re-ran my application and made sure that the transaction did abort and now I come back to the tracing options tab and click on Stop Session and New Session again. At this point, this is how the c:\windows\system32\msdtc\trace folder looks like. Notice that we are interested in the second file that was generated (dtctrace.log.2009-03-21-10-14-16-0011-00) as that will contain information about the aborted transaction. Now we can parse this file using the MSDTCVTR batch file After parsing the file we will get the parsed output inside AbortedTrace.csv file and we can open it up in notepad to see what it contains. This is how the file looks like

    pid=1052       ;tid=1128       ;time=03/21/2009-10:14:05.246   ;seq=14         ;eventid=TRACING_STARTED                          ;;"MSDTC is resuming the tracing of long - lived transactions"
    pid=1052       ;tid=1128       ;time=03/21/2009-10:14:05.246   ;seq=15         ;eventid=TRANSACTION_BEGUN                        ;tx_guid=35e57abd-1782-4fe2-ae59-01ffb0e9b628     ;"transaction got begun, description : ''"
    pid=1052       ;tid=1620       ;time=03/21/2009-10:14:05.246   ;seq=16         ;eventid=RM_ENLISTED_IN_TRANSACTION               ;tx_guid=35e57abd-1782-4fe2-ae59-01ffb0e9b628     ;"resource manager #1001 enlisted as transaction enlistment #1. RM guid = '5f9d03ca-2f03-49cf-9b17-7cb7962f0ef0'"
    pid=1052       ;tid=1128       ;time=03/21/2009-10:14:05.261   ;seq=17         ;eventid=RECEIVED_ABORT_REQUEST_FROM_BEGINNER     ;tx_guid=35e57abd-1782-4fe2-ae59-01ffb0e9b628     ;"received request to abort the transaction from beginner"
    pid=1052       ;tid=1128       ;time=03/21/2009-10:14:05.261   ;seq=18         ;eventid=TRANSACTION_ABORTING                     ;tx_guid=35e57abd-1782-4fe2-ae59-01ffb0e9b628     ;"transaction is aborting"
    pid=1052       ;tid=1128       ;time=03/21/2009-10:14:05.261   ;seq=19         ;eventid=RM_ISSUED_ABORT                          ;tx_guid=35e57abd-1782-4fe2-ae59-01ffb0e9b628     ;"abort request issued to resource manager #1001 for transaction enlistment #1"
    pid=1052       ;tid=2572       ;time=03/21/2009-10:14:05.261   ;seq=20         ;eventid=RM_ACKNOWLEDGED_ABORT                    ;tx_guid=35e57abd-1782-4fe2-ae59-01ffb0e9b628     ;"received acknowledgement of abort request from the resource manager #1001 for transaction enlistment #1"
    pid=1052       ;tid=2572       ;time=03/21/2009-10:14:05.261   ;seq=21         ;eventid=TRANSACTION_ABORTED                      ;tx_guid=35e57abd-1782-4fe2-ae59-01ffb0e9b628     ;"transaction has been aborted"

A careful review of the above file gives you the following information

The transaction identifier for our transaction is 35e57abd-1782-4fe2-ae59-01ffb0e9b628.
The transaction started at 03/21/2009-10:14:05.246 and got aborted at 03/21/2009-10:14:05.261
The fifth column in the above file shows the various events that happened for this transaction and the 6th columns gives a descriptive information about that particular event. So going through the above file, we can see that the event of interest is RECEIVED_ABORT_REQUEST_FROM_BEGINNER which tells us that the application which started the transaction is the one which sent an ABORT request to MSDTC for the transaction.

So from this we can infer that we are doing something wrong in our application which is sending an ABORT request to MSDTC. Upon reviewing our code and going through the documentation of TransactionScope , we can see that we missed something very important. From this article (see TransactionScope class)

“When your application completes all work it wants to perform in a transaction, you should call the Complete method only once to inform that transaction manager that it is acceptable to commit the transaction. Failing to call this method aborts the transaction.”

And the right code becomes The above example may not be the best one to give here but I am sure it will help you understand the basics of collecting a trace and how to parse it to retrieve the right information. The MSDTC trace can contain a lot of other events as well and I am pasting here some of the most common events that we see while parsing trace files from real time customer scenarios.

ABORT_DUE_TO_TRANSACTION_TIMER_EXPIRED	The Transaction is timing out because of the configured timeout for a transaction in DTC settings.
RECEIVED_ABORT_REQUEST_FROM_BEGINNER	The application which started the transaction issued an abort request to MSDTC to ABORT the transaction. In case of a transaction originating from a COM+ component, a call to ContextUtil.SetAbort will end up sending this event to MSDTC.
TRANSACTION_PROPOGATION_FAILED_CONNECTION_DOWN_FROM_REMOTE_TM	This error is typically due to networking issues. This happens when MSDTC on one machine loses connection with MSDTC on the other machine. For more information on this event, refer this article.
TRANSACTION_PROPAGATION_FAILED_TRANSACTION_NOT_FOUND	This event happens when you try to commit or abort a transaction that has been already committed or aborted.
RM_ENLIST_FAILED_TOO_MANY_ENLISTS	DTC already has the maximum number of resources enlisted in the transaction. The maximum number of resources that can be enlisted in a transaction is 32. More information in this article.

DCOM Error 10010 in the Event logs and SLUGGISH server performance

PuneetG — Wed, 21 Jan 2009 09:22:45 GMT

We have come across this specific issue on Windows 2003 (post service pack 1) and Windows XP (post service pack 2) where the machine experiences slow performance and we observe many DCOM 10010 ERRORS in the event logs. If you are running in to this issue, you may see some of the symptoms/errors listed below.

Doing anything takes forever. You click Start button and it responds to you after a minute (or may be more than that). Switching windows in the Taskbar takes very long.
If you have IIS installed on the server, the IIS related services may not start or may hang while starting. Typically these errors appear when you try to start them.
- Trying to start the IISADMIN service throws an error - Windows Could not start the IIS Admin Service on Local Computer. For more information, review the system Event log. If this is a non-Microsoft service, contact the service vendor, and refer to service-specific error code –2147221164(0x80040154) or The IIS Admin Service service terminated with service-specific error 2147746132 (0x80040154).
- If IISADMIN service is started, trying to start the World Wide Web Publishing Service hangs and the service goes into the Starting state and after some time it may throw the error - The World Wide Web Publishing Service service terminated with service-specific error 2148007941 (0x80080005)
While expanding COM+ Applications inside component services, you may get the below error
- An error occurred while processing the last operation
- Error code 8000FFF - Catastrophic failure
You observe many DCOM Errors 10010 in the event viewer (like this)
- The server {BA126AD1-2166-11D1-B1D0-00805FC1270E} did not register with DCOM within the required timeout
- The server {8BC3F05E-D86B-11D0-A075-00C04FB68820} did not register with DCOM within the required timeout

The interesting thing to note here is that most of the components which are failing or performing slowly are the basic system components only. The GUID {BA126AD1-2166-11D1-B1D0-00805FC1270E} is the CLISID (class id) for Network Connection Manager Class and the GUID {8BC3F05E-D86B-11D0-A075-00C04FB68820} is the CLSID for WMI. The one big common thing between all these symptoms is that all the programs which exhibit the above symptoms are dependent on DCOM. If you end up with the symptoms/errors listed above, one quick thing to check would be the permissions on the HKCR\CLSID registry key. By default this is how the permission on that key should look like. If you are running into this issue, you will see that the USERS group is not listed in the ACL list for this registry key. You might see an account with the name RESTRICTED listed out there. To fix the problem, you can configure the ACLS on the HKCR\CLSID key in the default way. For Windows 2003, this is how the default permissions on the HKCR\CLSID should look like.

Administrators – FULL CONTROL

Power Users – READ

SYSTEM – FULL CONTROL
Users – READ

After making the registry change, you have to reboot the machine so that the programs can access the registry during the startup and hence function properly. At this point, we haven't got a chance to determine the root cause of this problem. In other words, we don't know what particular action ends up removing the USERS group from the HKCR\CLSID registry key and we need your help in determining root cause. If you are able to reproduce this issue at your end, we will like to know the steps you took to reproduce this issue to figure out what CAUSES this and try to avoid it from occurring in the first place. Please feel free to post your comments at the end of this blog to let us know if you have a successful repro of this situation. PLEASE NOTE : The permissions on the HKCR\CLSID may not be the only cause of the errors. The errors that are listed above are very generic errors and can come in a lot of situations. The purpose of this article is to eliminate one basic cause which we come across a lot of times. If fixing the permissions on the HKCR\CLSID and then rebooting the machine, doesn't fix the issue for you, we recommend you run the Process Monitor tool and look for any kind of registry level or file level access denied’s which are happening on the server. Otherwise check out the following list of KB articles which may point more specifically to the issue that you are facing.

You may experience various problems after you install the Microsoft Security Bulletin MS05-051 for COM+ and MS DTC - http://support.microsoft.com/kb/909444
An event ID 10010 error message may be logged many times in the event log of your Windows Server 2003-based computer that is running Terminal Services - http://support.microsoft.com/kb/873375
FIX: You experience problems after you install Windows Server 2003 Service Pack 1 or Service Pack 2 on a server that is running Application Center 2000 - http://support.microsoft.com/kb/891330
You may receive an "Error code 80080005 -- server execution failed." error message when you start many COM+ applications - http://support.microsoft.com/kb/870655
A Windows Server 2003-based computer stops responding when you perform a backup of an Exchange Server database - http://support.microsoft.com/kb/912063
PRB: ASP pages do not process and DCOM event 10010 appears in the system event log http://support.microsoft.com/kb/327153/
How To Troubleshoot ASP in IIS 5.0 http://support.microsoft.com/kb/309051