The sample is plain and simple. The program takes an argument of type string. I am instructed to run it under a debugger – WinDBG & pass a string as command line argument which is long and it should crash the application. The task is to determine why the program crashed. 

The moment the program starts running, it breaks into the debugger upon hitting the initial breakpoint. Thereafter, I just type ‘g’ and hit enter to let the program run. As indicated, the program immediately crashes due to an Access Violation and breaks into the debugger.

0:000> g
(114c.19b4): Access violation - code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
eax=000cfefc ebx=00000000 ecx=000cff3a edx=00502a36 esi=00000001 edi=0100367c
eip=010012a7 esp=000cff44 ebp=000c0000 iopl=0         nv up ei pl nz na pe nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010206
05Program!wmain+0x17:
010012a7 8b550c          mov     edx,dword ptr [ebp+0Ch] ss:002b:000c000c=????????

From this, I can tell a few things as to why we crashed. The system attempted to de-reference a pointer located at the address contained in EBP register with an offset of 0C. We attempted to dereference the pointer so that we can store its value in the register EDX. The value we got was invalid and is at location 000c00c

OK, so why is this address bad? What is the type of address we attempted to de-reference?

0:000> !address 000c000c
TEB 7efdd000 in range 7efdb000 7efde000
ProcessParametrs 00251788 in range 00250000 00256000
Environment 00250810 in range 00250000 00256000
    00090000 : 00090000 - 0003c000
                    Type     00020000 MEM_PRIVATE
                    State    00002000 MEM_RESERVE
                    Usage    RegionUsageStack
                    Pid.Tid  114c.19b4

Alright; so we know this address is on the stack and it is for the thread with the id 19b4. If I list the threads in this process, here is what I get.

0:000> ~
.  0  Id: 114c.19b4 Suspend: 1 Teb: 7efdd000 Unfrozen

So there is only one thread in this process and this address is on the stack space allocated for the crashing thread. Is this address valid?

0:000> dd 000c000c l4
000c000c  ???????? ???????? ???????? ????????

Obviously, not! The key thing to note here is that this invalid address is an offset from the EBP register, which we do not manipulate directly from our code. However, it is a key register in managing access to parameters and local variables for a call frame. An understanding of how stack works will help debug this better. Here’s a diagram.

Stack Layout	Overflow into saved EBP

Thus, it seems here that we are crashing due to a buffer overrun that corrupts the stack, specifically the EBP register. Let’s begin by looking at the callstack

0:000> kv
ChildEBP RetAddr  Args to Child             
000cff44 01001467 00000002 00502940 00503ab0 05Program!wmain+0x17 (FPO: [Non-Fpo]) (CONV: cdecl)
000cff88 76fa3677 7efde000 000cffd4 77ea9d72 05Program!__wmainCRTStartup+0x102 (FPO: [Non-Fpo]) (CONV: cdecl)
000cff94 77ea9d72 7efde000 7697df09 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo]) (CONV: fastcall)
000cffd4 77ea9d45 010015a5 7efde000 00000000 ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo]) (CONV: stdcall)
000cffec 00000000 010015a5 7efde000 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo]) (CONV: stdcall)

Now, we need to go back in time to prove our theory. Where do we begin? One theory is that we can dis-assemble backwards from the current failure spot to find out the code we executed just before we crashed. Thus:

0:000> ub . l5
05Program!wmain+0x9 :
01001299 7523            jne     05Program!wmain+0x2e (010012be)
0100129b 8b450c          mov     eax,dword ptr [ebp+0Ch]
0100129e 8b4804          mov     ecx,dword ptr [eax+4]
010012a1 51              push    ecx
010012a2 e839000000      call    05Program!HelperFunction (010012e0)

ub means un-assemble backwards and the period (.) means from the current EIP value. l5 means length; un-assemble 5 instructions. So the above command goes back 5 instructions from current value of EIP register and starts listing the instructions, which will give us the preceding 5 instructions. The highlighted one is the code that we executed just before we failed.

Similarly u means un-assemble forward from the current value of EIP. In the listing below, you can see the instruction that failed (highlighted):

0:000> u . l3
05Program!wmain+0x17 :
010012a7 8b550c          mov     edx,dword ptr [ebp+0Ch]
010012aa 8b4204          mov     eax,dword ptr [edx+4]
010012ad 50              push    eax

Alright, so now we know that just before we failed, we called another function named HelperFunction at address 010012e0. If we look back 3 instructions before this function was called (green highlight), we see an access to the EBP register where we de-reference a pointer at ebp+0ch during the execution of the wmain function. We didn’t crash there, so we can be sure that the EBP register has the correct value when executing that instruction. Further, anytime we call a function, we always save the return address plus the EBP of the current call frame so that we can restore it upon returning from the function. In this case, we save the EBP for wmain method. Thus it stands to reason that after executing the function HelperFunction, somehow we corrupted the stack and overwrote the saved EBP value, which results in an access violation when we are back in the wmain method, because, the EBP register now is pointing to an location that is not correct for the wmain method.

Next, we investigate further by completely dis-assembling the HelperFunction using the uf command

0:000> uf 010012e0
05Program!HelperFunction : 
   010012e0 8bff            mov     edi,edi 
   010012e2 55              push    ebp 
   010012e3 8bec            mov     ebp,esp 
   010012e5 83ec3c          sub     esp,3Ch 
   010012e8 8b4508          mov     eax,dword ptr [ebp+8] 
   010012eb 50              push    eax 
   010012ec 8d4dc4          lea     ecx,[ebp-3Ch] 
   010012ef 51              push    ecx 
   010012f0 ff1574100001    call    dword ptr [05Program!_imp__wcscpy (01001074)] 
   010012f6 83c408          add     esp,8 
   010012f9 8be5            mov     esp,ebp 
   010012fb 5d              pop     ebp 
   010012fc c20400          ret     4

Alright, so now we can see a few more things.

• First we see the standard function prologue. Save the current EBP (which is for wmain function & move the stack pointer).
• Allocate space for local variables. In this case we are allocating 60 bytes (3C hex – sub esp,3Ch)
• Then we move the address of first parameter passed to HelperFunction into EAX register
• Then push that address on the stack
• The LEA stands for load effective address. So we push the beginning address for the space allocated for local variable into ECX register.
• We then call the imported function WCSCPY.

The _imp__ before a function means it’s calling into another library. In this case a C runtime library. This method is used for copying a string from one variable to another using pointers. See the MSDN documentation. So essentially, the HelperFunction passes the pointers to the source and destination to the WCSCPY function.

So now our theory is getting a bit stronger. The WCSCPY method is used to copy WCHAR (Wide character unicode strings) which are terminated with a NULL. Further, from experience, a developer may know that WCSCPY method does not check for bounds. We know that the space allocated on the stack was 60 bytes, which means the code declared a WCHAR string of size 30 bytes long. Why? Since the API we are using works on WCHAR types, we need 2 bytes to store each character and thus we allocate 60 bytes.

So, if we end up writing more than 60 bytes, we will overflow into the saved EBP of the wmain function and perhaps return address & more depending on how large the string being copied is. Now our goal is to find out the length of the offending string and how much it has overflowed. Unfortunately, we are past execution of the code for HelperFunction where all of this happened and we have an EBP register value that is corrupt. So the stack has technically “unwound” since we returned from the HelperFunction. What that means is the stack pointer has unwound, but the values should still be there. So we can poke around a bit to get to that string value.

The problem is to find out what was the starting address on the stack where the allocation for this string begins. To do that, we need to try and visualize what all “existed” on the stack before we ended up in the current state. So what all gets stored on the stack?

• Parameters
• Return Addess
• Saved EBP
• Local Variables

Given this fact, we can re-construct the stack to its previous state by examining the code in the HelperFunction. Usually, a good knowledge of assembly language is a plus point when doing this. I don’t know assembly much. I just use my favourite search engine (Bing) to figure out what each instruction does and also refer the Intel x86 instruction code (available on Intel site) shared by my other colleagues.

To reconstruct the stack, we must keep in mind the current state of the stack and understand the execution path. As I mentioned before, we are back in the wmain function with a bad EBP register. Where is ESP register pointing to? At the top of the stack.

If we execute a “dps esp” command now, we can see this…

0:000> dps @esp
000cff44  000cff88
000cff48  01001467 05overrun!__wmainCRTStartup+0x102

… which means the stack pointer is pointing to saved EBP. In this case, it is for the __wmainCRTStartup method. So our stack pointer is still intact. If we look back at the dis-assembly for the HelperFunction, the last few instructions are as follows:   010012f0 ff1574100001    call    dword ptr [05Program!_imp__wcscpy

010012f0 ff1574100001    call    dword ptr [05overrun!_imp__wcscpy (01001074)]
010012f6 83c408          add     esp,8
010012f9 8be5            mov     esp,ebp
010012fb 5d              pop     ebp
010012fc c20400          ret     4

So what does this do? After we return from executing the wcscpy method, we clean up the parameters passed to that method. It is also important to remember that the wcscpy function also restores the EBP for the HelperFunction. So the EBP register is now pointing to the correct location for HelperFunction. We then copy the value in the EBP register to the ESP register. Why? Before going back to wmain method, we need to restore its EBP. So this instruction simply moves the stack pointer to that location so that the next instruction (pop ebp) can copy the saved EBP value (now corrupt), for wmain method on the stack into the EBP register. The pop instruction also moves the stack pointer by 4 bytes. Thus, the stack pointer will now point to the return address within wmain.

Returning from the HelperFunction: This is the last step and the RET instruction pops the return address from the stack into the EIP register and the stack pointer also moves to 4 bytes. This gives control back to the calling function. Only the stack pointer and instruction pointers are modified by a subroutine return.

So since we are now ready to execute wmain again, the next instruction (mov     edx,dword ptr [ebp+0Ch] ) is executed, but it fails since de-referencing EBP results in access violation.

So to reconstruct the stack and get to the local variable that overflowed, we factor in the following:

• Parameter passed to the HelperFunction (4 bytes)
• The space required for saving of return address + EBP for the wmain method (4 bytes each)
• The local variable of 60 bytes.

All of this adds up to 72 bytes.

NOTE: I am not factoring in what happens on the stack when 05Program!_imp__wcscpy function executes since locals variables and register values for that function will be saved in stack locations beyond what area we need to look up. Further the cleanup of the stack space allocated by that method will be done by the callee itself. (CDECL calling convention. In the CDECL calling convention the called function is responsible for cleanup of the stack).

So now, for the math – Since the stack grows in the reverse order, we need to subtract 72 bytes.

0:000> ?esp-0n72
Evaluate expression: 851708 = 000cfefc

So now, we can use the dps command to dump the DWORDS and symbols as follows:

0:000> dps 000cfefc
000cfefc  00520041 //
000cff00  00610065 //
000cff04  006c006c //
000cff08  004c0079 //
000cff0c  00720061 //
000cff10  00650067 //
000cff14  00420044 //
000cff18  006f0043 // local variables
000cff1c  006e006e //
000cff20  00630065 //
000cff24  00690074 //
000cff28  006e006f //
000cff2c  00740053 //
000cff30  00690072 //
000cff34  0067006e //
000cff38  000c0000 //Saved EBP
000cff3c  010012a7 05Program!wmain+0x17 //return address 
000cff40  005029f8
000cff44  000cff88
000cff48  01001467 05Program!__wmainCRTStartup+0x102
000cff4c  00000002
000cff50  00502940
000cff54  00503ab0
000cff58  98f01d46
000cff5c  00000000
000cff60  00000000
000cff64  7efde000
000cff68  000cff74
000cff6c  00000000
000cff70  000cff58
000cff74  8a754a32
000cff78  000cffc4

The values on top of the stack now look like some text characters represented in hexadecimal values. For example, if we take the value 00520041 and start disassembling it from right to left in groups of 4 bits, we get 0041 and 0052. We can then run the .formats command

0:000> .formats 0041
Evaluate expression:
  Hex:     00000041
  Decimal: 65
  Octal:   00000000101
  Binary:  00000000 00000000 00000000 01000001
  Chars:   ...A
  Time:    Thu Jan 01 05:31:05 1970
  Float:   low 9.10844e-044 high 0
  Double:  3.21143e-322

0:000> .formats 0052
Evaluate expression:
  Hex:     00000052
  Decimal: 82
  Octal:   00000000122
  Binary:  00000000 00000000 00000000 01010010
  Chars:   ...R
  Time:    Thu Jan 01 05:31:22 1970
  Float:   low 1.14906e-043 high 0
  Double:  4.05134e-322

From the highlighted section above, we can see that the first 2 are characters as we thought, since it was a string copy operation. So let’s dump the entire string. We can use the du command since it is a unicode string

0:000> du 000cfefc
000cfefc  "AReallyLargeDBConnectionString"

Alright, so now we have the offending string. The length of this string is 30 characters long. We allocated 60 bytes since this is a WCHAR string & each character needs 2 bytes. So then why are we access violating? Let’s look at the dps command output again, specifically, the following lines.

000cff34  0067006e
000cff38  000c0000

The values in blue highlight are the last 2 digits of the string, which is “ng” and the values in cyan highlight is where we terminate the string. So we overflow into the Saved EBP register space of the stack by 2 bytes when we terminate the string, thus corrupting the saved EBP value on the stack. When we return from the function, this value is now stored into the EBP register. So now when we try to dereference locals and parameter values from the wmain function, we end up looking in the wrong memory location since the de-referencing is based on the relative address from the EBP register. In short, we have a stack corruption caused by a buffer overrun which is causing this crash.

SOLUTION: Most debugging books always mention this – Use only how much you allocate & always check what you are writing fits into the allocated space. Since we allocated space for 30 characters, we can technically write only 29 characters and leave one byte for the string termination. The developer must validate the input string length to ensure it fits in the space allocated or throw an error message to gracefully handle the situation to avoid stack corruption. Even better, follow the recommendation/security note from the MSDN Documentation for this function which is:

Security Note  Because strcpy does not check for sufficient space in strDestination before copying strSource, it is a potential cause of buffer overruns. Consider using strncpy instead.

The problem we had at hand had 2 particular symptoms

a. If we tried to access a SharePoint document library using UNC path, we will get back an error: <path> is not accessible. You might not have permission to use this resource.

b. If we browse to the document library using a browser, then select Explorer view, right click, we will get only the option New > Folder. In other words, the options to create office documents and other types of files will not be present. A couple of screen shots of what is expected vs. What is available is given below

Expected	Actual

Scoping Step 1: One of the things to remember is that IIS 7 comes with a new WebDav module. It can be downloaded from www.iis.net or installed along with adding the IIS web server role. That WebDav module does not work well with SharePoint sites. You can remove it specifically for SharePoint sites using the steps mentioned in this previous blog post of mine. After I confirmed that the new WebDav module was not installed, we proceed to the next step.

Scoping Step 2: If you see only New > Folder option, that is a behaviour that usually occurs when MS FPRPC protocol is in use. WebClient service in prior versions of Windows does not work with ports other than 80. It is important to understand that this is a limitation of the WebClient service and not the WebDav protocol. So in such cases the option is to use FPRPC protocol. FPRPC does not allow the rich experience that WebDAV allows. By default, WebDAV is preferred over FPRPC so that the end user has a richer experience when interacting with the SharePoint system. Thus if you see the reduced functionality, it means one of the following:

• WebClient service is disabled on the computer which forces the client to use FPRPC
• The site you are accessing is over a non standard protocol. (Limitation of the WebClient service, not protocol)

In this particular case the configuration was correct. WebClient service was started and further proof to this was that when accessing a SharePoint document library in another farm via Explorer View/UNC path, the full functionality was available. So there was no reason to suspect the client configuration.

Scoping Step 3: At this point, I had verified the basics and got enough evidence to prove this is a server side problem. So the next logical step was to capture a network trace to view the underlying traffic. The steps on how to capture a network trace are available in an earlier blog post of mine.

Below is how the traffic looks like when I filter traffic by WebDAV protocol. I have highlighted in yellow, the things I look at when reading the trace.

As you can see, the response the server is sending for an OPTIONS request is an HTTP 404. Unfortunately, there will be nothing in the ULS logs that you can use to further debug this. We debugged this problem and found the root cause. So I am just going to give you the answer now. After all blog is a way to share our domain knowledge, right?

You will see this behaviour when accessing SharePoint document libraries if the root site collection does not exist. The behaviour is that the server will send back an HTTP 404 response to the client. This can be easily reproduced by deleting the root site collection. Thus, if you see the above symptoms and have gone through the basics, capture a network trace and check if you see the response pattern as 401, 401, 404. If so, use the Central Administration website to check if the root site collection exists. If it doesn’t, then you will see an option to select the root site “/” against the Web Site Address option in the Create Site Collection page.

RESOLUTION – Create the root site collection with a blank site. If you do not wish to use it, just restrict access to the farm administrator using permissions.

One of the first things to understand this behaviour is to understand what causes Request Timed Out Exception. If you turn off custom errors in SharePoint and set the CallStack attribute to true in the web.config file for the problem application, you will notice that the exception is of type System.Web.HttpWebException. This exception is thrown by the .NET CLR. The CLR provides a quantum of time to the managed thread that is executing the request. By default, this is 110 seconds in .NET CLR 2.0. If this threshold is reached, the request is aborted and this particular exception is thrown. Why do we do this? To prevent runaway threads. So this is actually a fail safe mechanism. However there are cases where you might need to adjust it to a higher value. Be careful when you do this because if you have lots of requests that are allowed to run for a long time, then you will end up exhausting the CLR threadpool causing more problems such as queuing of requests which appear as a hang. Unfortunately, there are a few Microsoft KB articles that recommend increasing this value to as high as 999999. My opinion is – Do not do this on production servers (atleast)

So now that we know what causes this exception, let me explain how I went about getting to the root cause. First – We scoping the problem area. This takes up 90% of the time. Since this is a performance problem, there are a few possibilities

• Possible performance issues with web server or SQL Server.
• Possible performance issue with a single site or site collection.

I am not including the network piece here – Why? Because this is a server side behaviour not network latency. The server is aborting the request because it has been running for too long.

The problem did not have specific symptoms. It could occur while browsing any site within the site collection or performing administrative tasks such as adding users, creating sites, adding items to a list. So basically any activity could trigger this behaviour.

Scoping Step 1: I created a new site collection in the problematic web application and checked what the performance is like. It was fast and we never experienced any timeout errors. So the problem was specific to a particular site collection.

Scoping Step 2: I looked about in the UI to see how different it is from the default SharePoint sites. I immediately realized that there were some customizations in place, mostly cosmetic. So that didn’t look like a problem to me.

Scoping Step 3: I needed some data while running into this problem to see what SQL queries and stored procedures are being called. Based on our possibilities listed above, I also wanted to see performance data from the web and SQL Servers. So I setup the SQL Server to capture a profiler trace with capturing data only from the PID of the IIS worker process hosting this web application. I also setup the web server to capture performance monitor log including all counters for the processor, process, thread, .NET CLR exceptions and Web Service counters. There is a reason for using these specific counters. These counters will provide me with the following information:

• The process that is taking up CPU cycles.
• The overall processor usage.
• The thread that is taking the most CPU cycles
• Exceptions are expensive. .NET CLR Exceptions counter will give me information if there are too many exceptions occurring in the IIS worker process hosting this application.
• Web Service Counters will give me information about the load on the web server

Once I got this information, I quickly realized that with a single request, we had a processor usage in excess of 60% which was sustained and lasted well over 100 seconds. I then looked at the SQL Profiler traces. When I look at the SQL profiler traces, I am usually trying to figure out

• If any particular query is taking up too much time to execute. (This can be figured out by using the Duration column)
• If there are queries that are invoked a lot of times

You can easily do this using the Organize Columns button in the Events Selection tab of the SQL profiler trace to group by either duration or ObjectName. Grouping by duration can quickly tell you the long running queries and grouping by ObjectName will give you the stored procedures invoked the most number of times. Here’s a screenshot

Both of the above scenarios could result in performance problems. In this case, I had data like the following from the SQL Profiler trace after grouping by ObjectName column.

Needless to say there is a pattern in which these stored procedures are being called. For example proc_SplitUrl invokes fn_GetLeafName and fn_GetDirName the same number of times. proc_SplitUrl is used by many stored procedures and let us see if we can match it up with something else over here that executes a large number of times... and we have proc_UrlToWebUrlOutput. This in turn maps to proc_GetTpWebMetaDataAndListMetaData. Thus the chain of execution is as follows:

proc_SplitUrl > proc_UrlToWebUrlOutput > proc_GetTpWebMetaDataAndListMetaData
     |------------ fn_GetDirName
     |------------ fn_GetLeafName

If we lookup the documentation for these stored procedures on MSDN, you will notice that these stored procedures are invoked anytime SharePoint needs to enumerate the metadata for the site/list for whatever activity it is doing. The information we have so far isn’t enough to determine what the cause is. We know that some stored procedures are being invoked a very large number of times for a single request and the stored procedures being invoked are primarily to get metadata for the site/list. So this forms one part. The other unanswered part is – What user activity is causing us to enumerate the metadata? We already know that the problem occurs on any type of activity. So are all activities enumerating metadata?

Recollect that I also mentioned that my observations from perfmon collected from web servers shows sustained CPU utilization in excess of 60% for a single request. That is very bad. Imagine what could happen on a busy production server. I collected multiple hang dumps using Debugging tools for Windows to see what the thread(s) are doing? Maybe that has a direct relation on the SQL activity as well? Clearly, reviewing memory dumps require some advanced skill sets. There are many blogs out there that can get you started on managed debugging. Here’s a link to the most popular one. What I observed was quite interesting. All of the busy threads had a managed callstack where this portion was common:

Microsoft.SharePoint.Publishing.CachedObjectFactory.CreateWebFromUrl(System.String)
Microsoft.SharePoint.Publishing.Navigation.SPNavigationSiteMapNode..ctor(Microsoft.SharePoint.Publishing.Navigation.PortalWebSiteMapNode, ...
Microsoft.SharePoint.Publishing.Navigation.SPNavigationSiteMapNode.CreateSPNavigationSiteMapNode
Microsoft.SharePoint.Publishing.Navigation.PortalWebSiteMapNode.ProcessSPNavigationNode(Microsoft.SharePoint.Navigation.SPNavigationNode, ...
Microsoft.SharePoint.Publishing.Navigation.PortalWebSiteMapNode.PopulateNavigationChildren(Microsoft.SharePoint.Publishing.NodeTypes)
Microsoft.SharePoint.Publishing.Navigation.PortalSiteMapNode.GetNavigationChildren(Microsoft.SharePoint.Publishing.NodeTypes,
Microsoft.SharePoint.Publishing.Navigation.PortalSiteMapNode.GetNavigationChildren(Microsoft.SharePoint.Publishing.NodeTypes)
Microsoft.SharePoint.Publishing.Navigation.PortalSiteMapProvider.GetChildNodes(Microsoft.SharePoint.Publishing.Navigation.PortalSiteMapNode, ...
Microsoft.SharePoint.Publishing.Navigation.PortalSiteMapProvider.GetChildNodes(System.Web.SiteMapNode)
Microsoft.SharePoint.Publishing.Navigation.PortalHierarchicalEnumerable.System.Collections.IEnumerable.GetEnumerator()
System.Web.UI.WebControls.Menu.DataBindRecursive(System.Web.UI.WebControls.MenuItem, System.Web.UI.IHierarchicalEnumerable)
System.Web.UI.WebControls.Menu.DataBindRecursive(System.Web.UI.WebControls.MenuItem, System.Web.UI.IHierarchicalEnumerable)
System.Web.UI.WebControls.Menu.DataBindRecursive(System.Web.UI.WebControls.MenuItem, System.Web.UI.IHierarchicalEnumerable)
System.Web.UI.WebControls.Menu.DataBindRecursive(System.Web.UI.WebControls.MenuItem, System.Web.UI.IHierarchicalEnumerable)
System.Web.UI.WebControls.Menu.DataBindRecursive(System.Web.UI.WebControls.MenuItem, System.Web.UI.IHierarchicalEnumerable)
System.Web.UI.WebControls.Menu.DataBindItem(System.Web.UI.WebControls.MenuItem)
System.Web.UI.WebControls.Menu.PerformDataBinding()
System.Web.UI.WebControls.HierarchicalDataBoundControl.PerformSelect()
System.Web.UI.WebControls.BaseDataBoundControl.EnsureDataBound()
System.Web.UI.WebControls.Menu.EnsureDataBound()
System.Web.UI.WebControls.Menu.OnPreRender(System.EventArgs, Boolean)
Microsoft.SharePoint.WebControls.AspMenu.OnPreRender(System.EventArgs)

From my domain knowledge of SharePoint and developer background, I immediately realized that all of this was being trigged by the use of a menu control. Check the highlighted portion above.

OK, so there are umpteen menus in the UI. Which one are we talking about? I poked to find this information from the memory dump. The Microsoft.SharePoint.WebControls.AspMenu on the managed stack has a property named _id that holds the name of the menu. When I dumped it out, the name of the menu was: TopNavigationMenu

So now, we are ready to put our story together backed up by data.

CONCLUSION: While building the user interface for any request, we need to build the top navigation menu since it is part of every page. While doing so, we end up enumerating a lot of metadata for many different sites. We looked into the number of top level sites and sub sites. There were few top level sites and many child & sub child sites. So why are we enumerating all of them? When I looked at the top menu, I found that just hovering over a menu item will throw open the list of sub sites and hovering over the sub menu will throw open the sub sites list for that site and so forth. In short, the nest level for showing the sub sites was more than 1 (which is the default). We figured that using SharePoint designer, a developer had changed the nest level to 4. Thus we end up enumerating the metadata for all of the sub sites – 4 levels deep, just so that we can use that data for security trimming.

We turned off the “Show subsites” option using the top level site settings: Portal > Site Settings > Modify Navigation

This immediately fixed the Request Timed out exception since we are no longer displaying the top level menu structure. So we proved our findings by making this change. The end resolution for this issue was to change the nest level of the top menu bar to the default of 1 or whatever value is appropriate for you without running into the Request Timed out exception.

Many of you may be wondering… such a trivial thing created such a huge performance problem? Well, yeah. Think about what impact any customization (even the smallest change) can have. The developer wouldn’t have noticed this in the beginning because the number of sites in the hierarchy may have been relative small. As the site structure grows, this quickly becomes a performance problem.

The other scenarios that I am aware of and by the way, a very common one that we see in our support cases, are large lists that even our customers were not aware of existed in their environment. My first suggestion is to click on the View All Site Content link in the quick Launch bar and see if you can find any large lists using the Items column. Keep in mind that there may still be large lists that are not visible here, such as hidden lists. How Microsoft engineers figure out large lists will probably be the next blog post. :)

Scenario: I created a new web application on port 100 with no host headers & then a site collection based on collaboration portal. On one of my client machines, I had a HOSTS file entry named NorthWindWeb that mapped to my Web Front End server. Thus I would access the application as http://northwindweb:100

Later, I uploaded some documents using the browser interface and ran a Full Crawl. The first thing I observed was that my crawling failed. I have written a separate blog post here on how I went about troubleshooting the problem. Anyways, so after fixing the crawl issues, I observed a peculiar behaviour – In the Search Centre of the collaboration portal, if I click on Advanced Search, I’d get back the following error: The Search Application is not available for this site.

My first instincts are to look at the Event logs on the Web Front end. I check the event logs on the web front end because that is where my request is sent to. Nothing in there. As always, the first thing in any troubleshooting is Scoping the problem area. Scoping really helps narrow down on the problem area.

In this case, it had to be some sort of configuration problem since I can get back a page which says “search application is not available for this site”. I always say this – pay attention to the error message. Although many times errors can be misleading, there might still be some clues to begin with when you have nothing to start with.

So let me split the error - “The search application” “is not available” “for this site”. Hmmm.. for this site? Why? So does it work for other sites? I decided to use the default URL with which I created the web application to begin with – which was http://servername:100. I did so because when you create applications with a specific URL, I’d at least expect the system to get it configured properly using the defaults. So in this case I opened a new browser and navigated to http://servername:100, clicked on the search centre and then Advanced Search. Cool! I was able to browse to the page without errors and search worked as expected! Now I figure that there is something wrong with the URL that I am using – which is http://northwindweb:100. What is wrong? I am able to browse perfectly!

So here’s where some domain knowledge can help you move forward. In Office SharePoint Server, the Alternate Access Mappings (AAM) has a huge significance. AAM plays a vital role in accessing content, especially shared services. If you find one URL works and another doesn’t in MOSS apps, think AAM. So my first instinct was – Is the URL that I am using configured in the alternate access mappings configuration for that web application? Turns out that I didn’t have it! Although my HOSTS file entry was routing the traffic to the web front end correctly, the MOSS system doesn’t recognize the incoming URL HOST value. It checks its database and does not find this site/URL information and therefore returns the error. It works with the server name because it created the AAMs when I created the web application.

So I am sure you all know the resolution by now. :)

1. Open Central Administration Site.

2. Select the Alternate Access Mapping for that web application.

3. Click on Edit Public URLs link.

4. In the Intranet text box, add the value http://northwindweb:100

5. Click OK and test.

The destination farm is running on a Windows Server 2008 computer. When deploying to the destination, the content deployment fails with the following error:

The remote server returned an error: (404)

The first thing to remember is that any errors related to content deployment will be reported to the application event log. This is the default diagnostic logging setting for Content Deployment category. Any medium events will also be reported to the ULS logs. Therefore the first place to find out why the deployment is failing is to look at the Application event log of the export server.

In my case, here’s what I found from application event logs:

Failed to transfer files to destination server for Content Deployment job '80 - 500 Full Deployment'. Exception was: 'System.Net.WebException: The remote server returned an error: (404) Not Found. at System.Net.HttpWebRequest.GetResponse()
at Microsoft.SharePoint.Publishing.Internal.Administration.HttpDataTransfer.UploadFile(String sourceFilePath, String postToUrl, String& responseXml)
at Microsoft.SharePoint.Publishing.Administration.ContentDeploymentJob.UploadFilesToRemoteServer(Guid remoteJobId, String adminPortUrl, ArrayList dataFiles)
at Microsoft.SharePoint.Publishing.Administration.ContentDeploymentJob.DoServerToServer()'

Publishing: Content deployment job failed. Error: 'System.Net.WebException: The remote server returned an error: (404) Not Found.
at System.Net.HttpWebRequest.GetResponse()
at Microsoft.SharePoint.Publishing.Internal.Administration.HttpDataTransfer.UploadFile(String sourceFilePath, String postToUrl, String& responseXml)
at Microsoft.SharePoint.Publishing.Administration.ContentDeploymentJob.UploadFilesToRemoteServer(Guid remoteJobId, String adminPortUrl, ArrayList dataFiles)
at Microsoft.SharePoint.Publishing.Administration.ContentDeploymentJob.DoServerToServer()
at Microsoft.SharePoint.Publishing.Administration.ContentDeploymentJob.ExecuteJob()
at Microsoft.SharePoint.Publishing.Administration.ContentDeploymentJob.Run(Boolean runAsynchronously)'

Publishing: Content deployment job failed. Error: 'System.Net.WebException: The remote server returned an error: (404) Not Found.
at System.Net.HttpWebRequest.GetResponse()
at Microsoft.SharePoint.Publishing.Internal.Administration.HttpDataTransfer.UploadFile(String sourceFilePath, String postToUrl, String& responseXml)
at Microsoft.SharePoint.Publishing.Administration.ContentDeploymentJob.UploadFilesToRemoteServer(Guid remoteJobId, String adminPortUrl, ArrayList dataFiles)
at Microsoft.SharePoint.Publishing.Administration.ContentDeploymentJob.DoServerToServer()
at Microsoft.SharePoint.Publishing.Administration.ContentDeploymentJob.ExecuteJob()
at Microsoft.SharePoint.Publishing.Administration.ContentDeploymentJob.Run(Boolean runAsynchronously)
at Microsoft.SharePoint.Publishing.Administration.ContentDeploymentJobDefinition.Execute(Guid targetInstanceId)'

The Execute method of job definition Microsoft.SharePoint.Publishing.Administration.ContentDeploymentJobDefinition (ID 70d6c4ef-bda5-4b5e-84a0-8c07e4ec3e42) threw an exception. More information is included below.

So as we can see, the Exception was - The remote server returned an error: (404) Not Found. Where do we start troubleshooting? We should start by understanding what status code 404 means and what type of operation we were in when this failure occurred. The second part of the question can be easily answered by looking at the above callstack. At the top of the callstack is HttpDataTransfer.UploadFile() method. This means were were uploading a file to the target farm when the error occurred.

Info: Content deployment transfers contents to the destination by splitting them into multiple cab files and then uploading those files to a page in the target central administration site. Thus if we open the IIS log for the target central administration website, we should see the failure and more specific information about HTTP 404 response. Here’s what I found

POST /_admin/Content+Deployment/DeploymentUpload.aspx filename=%22ExportedFiles1.cab%22&remoteJobId=%223c392e22-67d9-46cb-b6c3-225cade93479%22 81 - 404 13 0 20

The salient parts in this IIS log entry are highlighted in yellow colour. The value 13 right after 404 is a sub-status code that gives more information about the type of failure. The IIS status codes and their meanings can be found here

So from this article, we end up with: 404.13 - Content length too large

So what is the size of the file ExportedFiles1.cab that we are trying to upload to the destination? We can find this out easily by opening the path to the temporary files on the source (Export Server) as configured in the content deployment settings within Central Administration. By default, this path is %windir%\TEMP\ContentDeployment. Now there is a catch here. By default, content deployment removes the temporary files. You must first configure content deployment to keep the temporary files in the event of a failure. This setting cannot be changed from the User Interface. The correct procedure to change the setting is to use STSADM command line

STSADM.EXE -o editcontentdeploymentpath -pathname "MyPath" -keeptemporaryfiles Failure

After making this change in setting, on the next run, the temporary files will not be removed and you can easily find out the size of the file it was trying to upload by opening this folder. In my case, it turned out to be 61 MB. So why can’t I upload a file of this size? It was possible to do this in Windows Server 2003! Also, who is throwing this error? These are the next set of questions we need to find answers to.

As I just mentioned, it is very important to figure out who is throwing this error – Is it IIS or Is it SharePoint? We know this behaviour was not present in Windows 2003 and therefore a likely chance that IIS 7, that ships with Windows 2008 is a cause here. Also, a little bit of IIS domain knowledge will help you very easily conclude where the problem is. If you do not have the domain knowledge, you can still troubleshoot using the Failed Request Tracing feature of IIS to pin point where exactly the failure occurred. Failed request tracing is a different subject altogether and if you are interested to know how to use this feature, here’s the official documentation.

So for folks without IIS domain knowledge, throw in the returned error code and associated message from the KB above into a search engine of your choice and you will get several articles about what setting you need to change to fix this. I’d suggest always using official documentation. I found this.

For myself, I know from experience that IIS 7 has a change where by we limit the max size of a request. The feature implements something like what URLScan did for us in the previous versions. Here is the documentation on requestLimits on IIS 7. So reading through, I can see that this module in IIS 7 can return 404.13 status and the size of request allowed is specified by maxAllowedContentLength attribute with a default value of 30000000 bytes, which is approximately 28.6MB.

So now we have identified why the server returned this status code and the solution is to simply increase this limit. Here’s how to do that using the new command line feature of IIS 7

On the destination farm, Open a command prompt and switch to C:\Windows\System32\inetsrv folder
Execute: appcmd set config "SharePoint Central Administration v3" -section:system.webserver/security/requestFiltering -requestLimits.MaxAllowedContentLength:524288000

I logged onto the machine using my domain account. Then created local user accounts for use with my standalone farm on this machine and then ran SharePoint Post Installation Configuration Wizard. After creating a new farm with a new configuration database, it also created the SharePoint Central Administration site for me automatically. I then attempted to provision my Shared Services Provider and guess what I run into? After several minutes, I get a page with the following error:

Provisioning for Shared Services Provider 'My SSP' has failed and will be retried.  Reason: User cannot be found. 

The error message is very odd! It doesn’t tell us which user it cannot find and what it wants to do. So I turn on our friendly diagnostic logging from the Operations tab in SharePoint Central Administration to get a verbose output of the actions. What I ended up finding was amusing. Although I didn’t completely understand why it works this way, here’s what I found in the ULS logs. I am including only the data in the message column for brevity:

Retrieved central administration site 'http://CASite:81'.    
Central administration site owner is 'Domain\UserName'.    
Creating shared services administration site 'My SSP'.    
Creating site ssp/admin in content database MySite_Content_82    
Error in resolving user 'Domain\UserName' :

System.ComponentModel.Win32Exception: Unable to contact the global catalog server    

at Microsoft.SharePoint.Utilities.SPActiveDirectoryDomain.GetDirectorySearcher()    

at Microsoft.SharePoint.WebControls.PeopleEditor.SearchFromGC(SPActiveDirectoryDomain domain, String strFilter, String[] rgstrProp, Int32 nTimeout, Int32 nSizeLimit, SPUserCollection spUsers, ArrayList& rgResults)    

at Microsoft.SharePoint.Utilities.SPUserUtility.ResolveAgainstAD(String input, Boolean inputIsEmailOnly, SPActiveDirectoryDomain globalCatalog, SPPrincipalType scopes, SPUserCollection usersContainer, TimeSpan searchTimeout, String customFilter)    

at Microsoft.SharePoint.Utilities.SPActiveDirectoryPrincipalResolver.ResolvePrincipal(String input, Boolean inputIsEm...    

Microsoft.SharePoint.SPException: User cannot be found.    

at Microsoft.SharePoint.Administration.SPSiteCollection.Add(SPContentDatabase database, String siteUrl, String title, String description, UInt32 nLCID, String webTemplate, String ownerLogin, String ownerName, String ownerEmail, String secondaryContactLogin, String secondaryContactName, String secondaryContactEmail, String quotaTemplate, String sscRootWebUrl, Boolean useHostHeaderAsSiteName)    

at Microsoft.SharePoint.Administration.SPSiteCollection.Add(String siteUrl, String title, String description, UInt32 nLCID, String webTemplate, String ownerLogin, String ownerName, String ownerEmail, String secondaryContactLogin, String secondaryContactName, String secondaryContactEmail, Boolean useHostHeaderAsSiteName)    

at Microsoft.Sh...    

So it turns out that while provisioning the Shared Services, there is a lookup on who the Site collection administrator is for the Central Administration Website. This makes sense because it needs to update all these settings for SSP database. While provisioning the Central Administration site, it used my domain account for the site collection administrator. Now, during the provisioning of the Shared Services, it was trying to resolve the account specified in the Central Administration site’s site collection administrators list, but failed to do so as it could not contact the global catalog server, resulting in this behaviour.

Resolution:

1. Open Central Administration and then click on Site Actions in top right corner, then site settings.
2. Under users and permissions column, click on Site Collection Administrators.
3. Removed my domain account from here and added the local Administrator account which I wanted to use for administering the local farm.
4. Click OK and wait for the Shared Services to be provisioned automatically (SharePoint will periodically re-attempt a failed provisioning of SSP)

As you can see from the event captured, the iexplore.exe process is running with Low Integrity level. Integrity levels are a new feature within Windows Vista and upwards. So if a process has a low integrity level, irrespective of whether the identity is part of local administrators group, operations like these on file system will still fail. Understandably, this is a security feature that I like and dislike at the same time. I like it because it prevents any website that I browse to from running code on my system that may be harmful. On the other hand I cannot save links to my favourites, which is not good either.

Weighing both options, I think I will stick to running iexplore process in low integrity level and save the links manually. The other option is to run iexplore.exe process as Administrator by right clicking on the shortcut and selecting “Run as Administrator”.

CAUTION: I do not recommend you do this at all because this runs the browser with elevated privileges and therefore it will be very easy for any malicious code to infect the machine.

Ever run into a situation where the people picker in SharePoint will fail to resolve usernames that are within a domain in a totally different forest? Assuming the trust relationships are setup properly? Well think again. Here is a quick check list:

• Ensure your people picker property is configured correctly in SharePoint.
• Configure your trust relationships properly.
• Ensure the ports required for inter server communication is opened. A list can be found <insert hyperlink>
• Ensure your DNS configuration is correct. This is specifically important because the web server will need to locate the Global Catalog Servers and the Domain Controllers in the source & target domains.

Each of the above points is in itself a big task. A failure in any one of these dependent components will cause people picker to fail. To allow SharePoint to query AD of a different domain, you need to configure it to use a specific account from the trusted domain. Here’s how you do that using STSADM command line

• stsadm –o setapppassword –password ********
  stsadm –o setproperty –pn peoplepicker-searchadforests –pv “domain:FQDN of trusted domain,Account in trusted domain,Password” –url URL of App
  Recycle AppPool for changes to take effect (Optional)

Alright. So now we know the basic check lists, we need to know how this works normally before you can troubleshoot any issues. In other words, unless you know what is normal, you cannot spot the abnormal?

Here’s a description of how PeoplePicker works in SharePoint.

Web server contacts one of the DCs in its domain and requests a SID lookup using the Windows API LsarLookupNames4. The LsarLookupNames4 method translates a batch of security principal names to their SID form. This traffic is encrypted and the Web server and domain controller talks via RPC. The RPC end point mapper is a UUID: E1AF8308-5D1F-11C9-91A4-08002B14A0FA. Now because this is initiated from LSASS, the LSARpc identifier is 12345778-1234-ABCD-EF00-0123456789AB. You should see both of these in a network trace. A successful request/response indicates that that RPC communication is successful.

So with LsarLookupNames4 API, we should get a SID. The next thing that happens is an LDAP query trying to lookup this SID and see if the name matches with what the user entered. To perform this, you need to have Kerberos traffic flowing properly. If Kerberos is working properly, you should also see that traffic just before the LDAP query with the username that you configured within SharePoint. After Kerberos authentication, SharePoint server then sends the LDAP query to one of the DCs in the trusted domains and does a search - something like:

LDAP:Search Request, MessageID: 26, BaseObject: DC=SharePoint,DC=com, SearchScope: WholeSubtree, SearchAlias: neverDerefAliases
LDAP:Search Result Entry, MessageID: 26, Status: Success

A filter is also passed based to indicate search based on the SID. Filter: (&(objectSID=))

The search result contains the properties requested for the user including the user’s SID. If everything matches, then we are done and the user’s full name should be displayed.

So that’s how it is “expected” to work. But most of the times when a support engineer is looking at the problem, he will not find the above traffic. Instead he is looking at the traffic in the broken scenario and there may be several reasons why the feature is not able to find the user. For eg:

• What if the trust relationship is not setup properly? Can we verify that using a network trace? Is it possible?
• What if the MSRPC is broken? Can we determine that using a network trace?
• What if the DNS entries are not setup properly? Can you determine that using the network trace?

So well, the answer is “depends”. A lot of times you can make a good conclusion depending on what you see in the network trace if you have domain specific knowledge.

• If you never see the MSRPC bind requests getting a success response, chances are that the trust is not setup properly.
• If you do not see Kerberos traffic or connecting with the username specified in SharePoint, then your SharePoint configuration is probably not correct.
• If you see DNS related errors in the network trace (filter by DNS traffic), then your DNS is probably broken and needs to be fixed.

Obviously, what needs to be fixed depends on what his broken. No matter what works and what does not, at the end of the day, if you performed a Check Name operation within People Picker, we must match the user with a SID. To do that, SharePoint goes to great lengths. SharePoint will attempt a query based on Person or Group and also perform a wild card search. Here’s an example of filters used that you may see in the network trace for LDAP queries:

First Attempt:

filter: (objectCategory=person)
filter: (objectClass=user)
filter: (!(BIT_AND: (userAccountControl)&2))
filter: (|(name=Sharepoint\Skumar)(displayName=Sharepoint\Skumar)(cn=Sharepoint\Skumar)(mail=Sharepoint\Skumar)(samAccountName=Sharepoint\Skumar)(proxyAddresses=SMTP:Sharepoint\Skumar)(proxyAddresses=sip:Sharepoint\Skumar))

filter: (objectCategory=group)
filter: (BIT_AND: (groupType)&2147483648)
filter: (|(name=Sharepoint\Skumar)(displayName=Sharepoint\Skumar)(cn=Sharepoint\Skumar)(samAccountName=Sharepoint\Skumar))

Result: None

Second Attempt

filter: (objectCategory=person)
filter: (objectClass=user)
filter: (!(BIT_AND: (userAccountControl)&2))
filter: (|(name=Sharepoint\Skumar*)(displayName=Sharepoint\Skumar*)(cn=Sharepoint\Skumar*)(mail=Sharepoint\Skumar*)(sn=Sharepoint\Skumar*)(SamAccountName=Skumar*)(proxyAddresses=SMTP:Sharepoint\Skumar)(proxyAddresses=sip:Sharepoint\Skumar))

filter: (objectCategory=group)
filter: (BIT_AND: (groupType)&2147483648)
filter: (|(name=Sharepoint\Skumar*)(displayname=Sharepoint\Skumar*)(cn=Sharepoint\Skumar*)(SamAccountName=Skumar*))

Chances are that you may get back a response with a wild card search – as in the second case on my machine, because an OR search on SamAccountName=SKumar* found a record but not with SamAccountName=SharePoint\SKumar as in the first case. However, what happens right after that is, the system will pick the SID from the response (if any) and attempt a match with the SID. If that fails, Check Name operation will throw an error that it could not find the user. So the key to getting that to work is ensure that we can perform a SID lookup successfully.

So what tools are there to check if we can resolve the SIDs?

Microsoft Support uses from PSGetSID from SysInternals. It is one of the tools you can use to verify if the SID lookup is working properly in your environment. & it is really easy to use. From a command line, run: PSGetSid <domain\username>.

If this tool fails to get the SID, ignore the SharePoint part and focus on fixing your environment first. 

If you attempt copying a document from one document library to another and the source and target libraries are in different web applications, then this feature won’t work with Office 2003 clients and MOSS 2007. My guess is that this behaviour occurs because, when Office 2003 was released, it wasn’t built to support features that MOSS 2007 provides. MOSS 2007 was released 4 years later!

This feature will work with Office 2007 clients, though. However if you are copying from one library to another which are located in different site collections AND there does not exist a root site collection at the target, then this feature won’t work. If you see these symptoms, creating an empty root site collection should resolve the problem. It appears to me that it may have something to do with not being able to find the path to the child if the root site collection does not exist.

The way the Send to... Other location works is by using a ActiveX Control (Multiple Document Upload Control) installed with Microsoft Office 2007. After you provide the destination URL (BTW, the destination URL must end with the library name and not the view name such as AllForms.aspx) and click OK, the Jscript on the web page invokes this control with a call to new ActiveXObject("STSUpld.CopyCtl");. To successfully instantiate the control, the following registry keys must be present.

• HKCR\STSUpld.CopyCtl
• HKCR\STSUpld.CopyCtl.CLSID\(Default)
• HKCR\STSUpld.CopyCtl.CLSID

When instantiated, this control contacts a web service on the SharePoint server to copy the contents from source to destination. It first makes a request to the source to get the document and its properties and then POSTs the data to the target location. Now, if the Multiple Upload control cannot be loaded for some reason, SharePoint will attempt to perform this operation via server side by sending a POST to CopyResults.aspx. This is what you see in the browser address bar. Now, the CopyResults.aspx page is capable of copying of documents only if the source and destination are in the same web application. The functionality appears to be by design – In other words, the page restricts copying across application boundaries. The API that causes this behaviour is also documented on MSDN.

ValidateDomainCompatibility Method

Also, the posting and responses from the Program Manager @ the SharePoint Designer Team blog has mentioned that calling Cross applications is not allowed. Here’s the link. FYI, there isn’t a setting to change this behaviour. This is probably something that the development team knows about and may fix in the later versions of MOSS, hopefully.

When application domain boundaries need to be crossed, only the web service - Copy.asmx works because it completely extracts all the contents (encrypted of course!) and the properties of the document before posting it to the target document library. I am still not sure why this functionality cannot work from server side code though; but that appears to be what is going on.

Coming back to problems instantiating the Multiple Upload control installed with Office 2007 clients, the best way to fix it will be to run a repair installation of Microsoft Office because the DLL in question is not something you can register using regsvr command. Also, when running the Office Setup, ensure Windows SharePoint Services support component is selected.

I hope the information shared above was helpful to people who have/are running into this behaviour.

"Copying of this file failed. This operation uses the SharePoint Administration service (spadmin), which could not be contacted. If the service is stopped or disabled,  start it and try the operation again."

So it turns out that the internal exception that causes this problem is:

Failed to connect to an IPC Port: The system cannot find the file specified. 

There is a new hotfix available for this which updates a .NET Framework DLL, which resolves this issue. The specific KB article that talks about this is

A System.Runtime.Remoting.RemotingException exception is thrown when you deploy a SharePoint solution on a SharePoint Web server that is running the .NET Framework 2.0 SP2

FYI: You need to have .NET Framework 2.0 Service Pack 2 installed before applying this fix.

If you want to debug this and confirm that the internal exception is actually the one noted above, there are 2 ways to use this.

Easy way: Look at the application event logs to see if this exception is logged.

Hard way: !!! Do not try this in Production !!! Read on…

1. Download and install Debugging tools for Windows for the platform on which you are running.
2. Using ADPlus, log all first chance .NET CLR exceptions.
3. Note down the IIS worker process (w3wp.exe) PID corresponding to your app using using task manager
4. Save the below script to Debugging tools for windows folder as LogExceptions.cfg
5. Execute the command as: CScript adplus.vbs –p <PID> –c LogException.cfg
6. Reproduce the problem.
7. Inside the debugging tools for Windows folder there is another folder that is created that begins with Crash…The exceptions are logged in a text file that is named as PID-<PID>.exe…

Here is the script:

<ADPlus>
<Settings>
<RunMode>Crash</RunMode>
<Option>Quiet</Option>
</Settings>

<PreCommands>
<Cmd>.loadby sos mscorwks</Cmd>
</PreCommands>

<Exceptions>
<Option>NoDumpOnFirstChance</Option>
<Option>NoDumpOnSecondChance</Option>
<Config>
<Code>clr</Code>
<Actions1>Log;Stack</Actions1>
<CustomActions1>.time;!pe-nested;!clrstack;!threads</CustomActions1>
<ReturnAction1>gn</ReturnAction1>
</Config>
</Exceptions>
</ADPlus>