Background

I frequently use my Windows Phone (currently a Nokia Lumia 800) for listening to audio books. I’ve found that the best way to do so is to change the category of the files to “podcast” and they’ll show up as podcasts and I won’t have to keep track of where I was, etc. since Zune will do that for me. Zune uses the creation date to order the files in a podcast, and some times the creation dates of my files may not be entirely correct. There are a lot of shareware programs that will allow you to do batch updates on files and change these properties, but it’s actually a lot easier to do it yourself in PowerShell

How to fix this

If you have the files sorted the way you want alphabetically and want to order them by creation date in the same way, then this is a pretty easy way to do it.

Naturally you could alter other properties as well simply pass one of the objects into get-method to see what’s available:

Well, that’s basically it. Enjoy!

/ Johan

Hello According to this article, http://blogs.msdn.com/b/johan/archive/2008/01/31/using-windbg-hunting-exceptions.aspx written by you, you write that there are three "ever-present exceptions", I can see why they have to be on the heap, but why is 790ff624 3 36 System.Text.DecoderExceptionFallback 790ff5d8 3 36 System.Text.EncoderExceptionFallback Also always there? I always see them on the heep when I do a debug session. I have always discarded them. According to Tess, they are "not real exceptions", refering to that a print of all exceptions on the heap throws an error when the are printed. I think that the are put on the heap when the process is created, like the other three. But why? Why are they so important? I DEMAND an answer, because I love you and your blog :) I am just joking. I have written to you because I think that you know your ways around the CLR and the heap.

I thought the answer might interest others as well, so I decided to answer the e-mail here.

This is not an Exception

The reason why these are on the heap and are not possible to examine using !PrintException (!pe) is because they’re actually not Exceptions at all. They show up when you run !dumpheap -type Exception because they contain the word “Exception” in their name, but that’s it. The System.Text.DecoderExceptionFallback object for example derives from the DecoderFallback object. It’s a fail-safe mechanism that is used when you can’t class that is used then you can’t complete the decoding conversion.

So what is a Decoder Fallback?

There are different fallbacks you can use. You can use the DecoderReplacementFallback that will replace any byte sequence that it can’t decode with a predefined string. For example; the decoder comes across a segment it can’t convert. You’ve defined a DecoderReplacementFallback and it replaces the segment with “[unknown]”. Execution will then continue.

If you had instead defined a DecoderExceptionFallback a DecoderFallbackException (notice the difference?) would be thrown when the decoder came across the segment that it couldn’t convert and execution would stop after that.

Cheers! / Johan

He wanted to change the “Allow rules to specify another site as a target location”-setting (seen below) through code and needed to know if this could be done.
I hadn’t tried this before, but thought it’d be fun to try.

Finding the correct key

I suspected that the property could be found in Web.AllProperties, but I had no clue as to what the key might be called, so I simply iterated through all the keys twice. First with the setting enabled and secondly with the setting disabled. I wrote the keys to a label, and simply pasted them into Notepad so I could compare the two listings. The code I used to do this was quite simple:

foreach (String key in Web.AllProperties.Keys)
{     lblKeys.Text += key + "<BR>";
}

Having done this it was quite easy to identify the key I needed. It turned out it was called “_routerenablecrosssiterouting”. I recommend using the same approach if there is another setting you wish to change.

The final code

Now that I had the key name I was basically done. I used the following line to read the setting:

myString = (Web.AllProperties["_routerenablecrosssiterouting"]).ToString();

This is the code I used to change it:

Web.AllowUnsafeUpdates = true;
Web.AllProperties["_routerenablecrosssiterouting"] = "False";
Web.Update();

Since I was using a GET-request in my little sample I had to set AllowUnsafeUpdates = true

I’ve seen other people asking how to accomplish this on the net, so hopefully this post will be of use to you.

/ Johan

0:000> !dumpheap -stat
The garbage collector data structures are not in a valid state for traversal.
It is either in the "plan phase," where objects are being moved around, or
we are at the initialization or shutdown of the gc heap. Commands related to 
displaying, finding or traversing objects as well as gc heap segments may not 
work properly. !dumpheap and !verifyheap may incorrectly complain of heap 
consistency errors.
Error requesting GC Heap data
Unable to build snapshot of the garbage collector state

This can happen if both the .NET 4.0 and the .NET 2.0 runtime are loaded in the same process. I actually tried using .loadby on both Framework versions and neither of them returned with an error

0:000> .loadby sos mscorwks
0:000> .loadby sos clr

The easiest way to resolve this is to run the following command:

0:000> .cordll -I mscorwks.dll -N -ve

After that, simply reload the extensions you want to use and you should be fine.

The problem

I came across the following scenario a while back: A customer was creating a custom WebControl in the Visual Web Designer. The control had a smart tag panel with an option that they wanted to behave like a radio button. I.e. The text says “Enabled”, you click it and it changes to “Disabled”. The problem was that the panel didn’t refresh. Even if they called DesignerActionUIService.Refresh(). They had to close the panel and reopen it for the change to become visible.

If they instead hosted the same code in a WinForms control then the panel was refreshed immediately. So in effect the code below would update immediately if designing a WinForms component, and require closing and reopening the panel if designing a WebForms panel.

public bool LockColors
{
    get
    {
        return colLabel.ColorLocked;
    }
    set
    {
        GetPropertyByName("ColorLocked").SetValue(colLabel, value);

        // Refresh the list.
        this.designerActionUISvc.Refresh(this.Component);
    }
}

Cause

Simply put, the Web Developer doesn’t subscribe to the DesignerActionUIStateChange event, so this is currently by design.

Resolution

You can fix this by implementing your own event using code like this:

public UserControlDesignerActionList(ISourceControlledItem scItem)
        : base(((IDesigner)scItem).Component)
    {
        this.designerActionUISvc = GetService(typeof(DesignerActionUIService))
as DesignerActionUIService;
 
        if (this.Component is Control) {
            this.designerActionUISvc.DesignerActionUIStateChange += 
new DesignerActionUIStateChangeEventHandler(OnDesignerActionUIStateChange);
        }
    }
 
    void OnDesignerActionUIStateChange(object sender, DesignerActionUIStateChangeEventArgs e) {
        if (e.ChangeType == DesignerActionUIStateChangeType.Refresh) {
            ElementDesigner ed = ElementDesigner.GetElementDesigner((Control)this.Component);
            IWebElementDesigner wed = (IWebElementDesigner)ed;
            if (wed != null) {
                wed.UpdateView();
            }
        }
    }

In order for this to work you need to add a reference to Microsoft.Web.Design.Client.dll, which is located at C:\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\IDE\Microsoft.Web.Design.Client.dll as well as in the Global Assembly Cache.

Licensing

The main issue with this approach is that of licencing. If you intend to redistribute this component to a customer you may redistribute the .dll to that customer if they already own a valid license of that same .dll. Any component you design that need this change to work would require a copy of Visual Studio to be at all usefull, but you never know...

The best approach would probably be to rely on the version in the GAC and not distribute the .dll at all.

If you have a web application that uses SSL and this application, for some reason, is hosted on an IIS that doesn’t have Internet access you will eventually find that any call to the server over SSL will take a lot of time to reply. It doesn’t matter if it’s ASP.NET, classic ASP or even CSS. As long as the request is made via SSL the response-time will skyrocket.

This isn’t a very common situation, since normally the IIS is connected to the Internet. HoweverI’ve come across this “in the field” where clients were routed internally to the IIS through a proxy, so it does happen.

Cause

This is a variation of the problem described in this article. The IIS will want to go on-line and verify that the certificate has not been revoked. In order to do this is requires a working Internet connection. Otherwise it will wait until the request times out and then service the request. When the next SSL-request comes along it will attempt to go on-line again, time out, and so on.

If you've had your server connected to the Internet during testing, then you will most likely not come across this problem straight away, since the response from the lookup in the certificate revocation list (CRL) will be cached. This makes it harder to pinpoint the root cause, unless you know where to look.

Resolution

The easiest way to resolve this is to skip CRL checking. This is done by setting the metabase property CertCheckMode to 1.

Create a blank .txt file on the server and paste the following code into it:

Set oWeb = GetObject("IIS://localhost/W3SVC")
oWeb.CertCheckMode = 1
oWeb.SetInfo
Set oWeb = Nothing

Running this script will change the property so that revocation checking is no longer done. If you want to switch back to the default value, then you should set CertCheckMode to 0 instead.

/ Johan

(What to do about the slow startup of web services)

As I learned more about his problem I found that this was more than just a simple matter of pre-compilation and keeping your w3wp.exe alive.

The Cause

The customer was using authenticode signed assemblies in his web application. When that application calls the assembly for the first time it will want to go on-line and check that the certificate is still valid. So, what if this is an internal server with no Internet access? Well, then we have a problem. The process will spend quite some time trying to check the certificate revocation lists. Eventually it will give up, but for my customer this meant that it hung for at least 20 seconds upon startup.

The Resolution

To resolve this I’d chose one of the following three options:

Grant Internet access to the server

This is will obviously resolve the problem, but may not always be possible.

Review the need for Authenticode signing

If you own the assemblies then you might want to consider not using Authenticode signing at all. A Strong Named Assembly might be sufficient. For more information on Strong Named Assemblies and how to create them, please see the following article on MSDN: How to: Sign an Assembly with a Strong Name

Disable Signature Verification

If all else fails you can disable Signature Verification by adding the following to your machine.config or application.exe.config:

Identifying the problem using windbg

Okay, so if we suspect that this is the problem, how can we verify this?

I got a memory dump from my customer, so I’ll use that to demonstrate.

First of all you want to make sure the dump is taken as the application pool hangs upon the first request. Getting a dump right after the problem has occurred will usually do us no good at all. We need to get the dump as the problem is actually ocurring. Having made sure the dump is fine I open up the dump in windbg and load sos.dll. I then use the !aspxpages command to see the pending requests currently on the server. !aspxpages dumps all the HttpContexts found on the heap so we get a list of all currently executing requests, as well as recently finished ones that haven’t been garbage collected yet.

So, the request on thread 19 is the one we want. As we can see in the ThreadId column it is the only active request. Let’s jump to that thread and investigate what it is doing.

So we’re in the Page_Load event of the page in question, but apart from that !clrstack doesn’t tell us much. Let’s take a look at the native callstack instead using the kb-command.

Here we see that we’re currently trying to check the certificate revocation list. This is done on a separate thread so this thread is currently doing nothing but waiting. Our next step would be to find this thread, but first we can find out the name of the .dll that we’re trying to verify. Look at the first argument to GetPublisher.

So it’s a Microsoft .dll that is part of the Enterprise Library. Good to know. Now let’s find the thread that is attempting to retrieve the revocation list by running ~*kb and scanning through the threads until we find this one:

This is the thread. To finally verify that it is in fact attempting to download the Certificate Revocation List we can check the value of the first argument to cryptnet!CInetSynchronousRetriever, InetRetrieveEncodedObject and CObjectRetrievalManager::RetrieveObjectByUrl or the second argument to cryptnet!InetSendReceiveUrlRequest as I did above.

Conclusion

In this case the customer chose to disable Signature Verification completely rather than granting Internet access to the server.

If you found the windbg-portion of this post hard to understand I can recommend a closer look at my Debugging School (In the menu to the left). Begin with Getting started with windbg - Part I and you should be set to go in no time.

/ Johan

You have an ASP.NET application which intermittently responds sluggishly. As the problem occurs memory usage is about average, as is CPU usage, but still certain pages respond slower and slower. The machine acts just as if it is under heavy load, but judging from the CPU it isn't. In fact CPU usage might even decrease rather than increase. When trying to access non .NET content such as plain text/html or images you usually find that response times are just as fast as ever.

When you find that your application has hung in a low CPU state it is usually because of one of the following reasons (in order of likelihood based on my experience) :

1. External resources responding slowly. E.g. a database query or AD lookup is taking a long time to complete and so the application is simply waiting for another server to respond.
2. One of the threads of the application has entered a critical section and now several other threads are waiting to enter the same critical section.
3. A classic deadlock. Thread A waits for thread B which waits for thread C which waits for thread A.

Quite often you'll see a mixture of items 1 and 2. A thread enters a critical section, calls a remote database, or similar, and all other threads attempting to enter the same critical section will have to nicely stay in line until the struggling database has responded. Still, it is quite possible that the critical section is waiting for a local event to finish, but then you'd usually see a higher CPU load on the machine. (Though not necessarily in the w3wp.exe process.)

What are Critical Sections?

In case you've forgotten or simply not worked that much with critical sections it might be necessary to refresh your memory on this.

A Critical Section is a segment of code that you do not want to be executed more than one at a time. In order to ensure this you use the lock keyword to ensure that a certain block of code is able to complete with no interruption from any other threads. To illustrate this I've created the following code sample:

public class FileIO
{
    private static object myLock = new object();

	public FileIO()
	{
	}

    public string performLockOperation()
    {
        string strResult = "The Lock Operation started on " + System.DateTime.Now.ToLongTimeString();
        lock (myLock)
        {
            // Let the thread sleep for 10 seconds
            // This way we simulate a File IO operation that takes some time to complete.
            System.Threading.Thread.Sleep(10000);
        }
        strResult += " and ended on " + System.DateTime.Now.ToLongTimeString();
        return strResult;
    }

    public string performNoLockOperation()
    {
        string strResult = "The No Lock Operation started on " + System.DateTime.Now.ToLongTimeString();
        System.Threading.Thread.Sleep(10000);
        strResult += " and ended on " + System.DateTime.Now.ToLongTimeString();
        return strResult;
    }

}

Let's take a quick look at the parts of this class that I've written, called FileIO:

myLock

The class has a static variable called myLock that I will be using to identify the lock I'm about to create. This is why the variable is static. Otherwise I'd be creating new locks for each instance of the FileIO class leaving the lock operation completely redundant. Now that I use a static variable the lock spans all instances of the FileIO class.

constructor

The constructor is empty. Move along. Nothing to see here.

performLockOperation

This is the most interesting part of the class. It is the function we'll be using to actually reproduce the potential problems. The function returns a string which will contain information about when the call to the function was made and when it completed. We store this data in a string called strResult which we will eventually be returning as the result of the function call.

Having declared this variable and put the initial time of the function call in it we then create our lock. This is simply done with the following syntax:

lock (object)
{
}

Any code within this block will first make sure that there is not already a lock taken on this object. If there is, then they will pause execution and wait for the lock to become available before continuing execution.

Inside our lock we then simulate a slow File IO operation by letting the thread sleep for 10 seconds.

After that we append the current time to the result and return it.

performNoLockOperation

This function is identical to the previous one, except this time we do not have a critical section. This function is included just for comparison.

The potential problem

Now, let's get down to business. If we have a web page that makes a call to the performLockOperation-function and that page is requested at the same time by four different clients. What would be the response times?

The answer is 10, 20, 30 & 40 seconds.

Here's why:

1. The thread handling the first request enters the critical section, effectively locking it for everyone else. When it is complete 10 seconds have passed due to the Sleep-call.
2. One of the other threads enters the critical section once the first request is finished. It has already waited for 10 seconds, and the thread will now sleep for an additional 10 seconds.
3. As the second thread finishes one of the remaining two enters the critical section. It has now waited for 20 seconds and will sleep for an additional 10.
4. The final request is able to enter the critical section after having waited for 30 seconds. Once it is complete 40 seconds have passed.

If we'd been using the performNoLockOperation instead then all requests would have responded after 10 seconds.

Now, I know that 10 seconds is an exceptionally long time to let the thread sleep, but this was all for illustration purposes. Imagine instead that the sleeping time was 1 second and the number of requests was 200...

Troubleshooting this issue

Okay, so what would it look like when this problem occurs? Let's have a look. If you haven't looked at my posts on getting started with Windbg, then I suggest reading them before continuing. (Part I & Part II)

First of all I fire up WinDbg, then I attach to the w3wp.exe. I then use "~*e!clrstack" to look at the managed threads of the w3wp.exe application. Here is a snippet of what I get:

OS Thread Id: 0xce0 (24)
Child-SP         RetAddr          Call Site
000000000447e0f0 000007ff001e13d2 FileIO.performLockOperation()
000000000447e170 000007feee323ec9 _Default.Page_Load(System.Object, System.EventArgs)
000000000447e1b0 000007fee7956aea System.Web.Util.CalliHelper.EventArgFunctionCaller(IntPtr, System.Object, System.Object, System.EventArgs)
000000000447e1e0 000007fee794d2c4 System.Web.Util.CalliEventHandlerDelegateProxy.Callback(System.Object, System.EventArgs)
000000000447e210 000007fee794d322 System.Web.UI.Control.OnLoad(System.EventArgs)
000000000447e250 000007fee79498ac System.Web.UI.Control.LoadRecursive()
000000000447e2a0 000007fee7948db0 System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)
000000000447e370 000007fee7948cdb System.Web.UI.Page.ProcessRequest(Boolean, Boolean)
000000000447e3d0 000007fee7948c70 System.Web.UI.Page.ProcessRequest()
000000000447e430 000007ff001e0c09 System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)
000000000447e490 000007fee7950117 ASP.default_aspx.ProcessRequest(System.Web.HttpContext)
000000000447e4c0 000007fee791449b System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
000000000447e570 000007fee7ffbd41 System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)
000000000447e610 000007fee7fed132 System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception)
000000000447e7a0 000007fee7fcf599 System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, System.AsyncCallback)
000000000447e7f0 000007fee80f5344 System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, System.Web.HttpContext)
000000000447e910 000007fee80f65bb System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
000000000447ea80 000007fee80f4994 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
000000000447eae0 000007fef72701ea DomainNeutralILStubClass.IL_STUB(Int64, Int64, Int64, Int32)
000000000447f310 000007fee80f5424 DomainNeutralILStubClass.IL_STUB(IntPtr, System.Web.RequestNotificationStatus ByRef)
000000000447f3f0 000007fee80f65bb System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
000000000447f560 000007fee80f4994 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
000000000447f5c0 000007fef727043b DomainNeutralILStubClass.IL_STUB(Int64, Int64, Int64, Int32)
OS Thread Id: 0x15ec (25)
Unable to walk the managed stack. The current thread is likely not a 
managed thread. You can run !threads to get a list of managed threads in
the process
OS Thread Id: 0x1958 (26)
Child-SP         RetAddr          Call Site
0000000004c1dd10 000007ff001e13d2 FileIO.performLockOperation()
0000000004c1dd90 000007feee323ec9 _Default.Page_Load(System.Object, System.EventArgs)
0000000004c1ddd0 000007fee7956aea System.Web.Util.CalliHelper.EventArgFunctionCaller(IntPtr, System.Object, System.Object, System.EventArgs)
0000000004c1de00 000007fee794d2c4 System.Web.Util.CalliEventHandlerDelegateProxy.Callback(System.Object, System.EventArgs)
0000000004c1de30 000007fee794d322 System.Web.UI.Control.OnLoad(System.EventArgs)
0000000004c1de70 000007fee79498ac System.Web.UI.Control.LoadRecursive()
0000000004c1dec0 000007fee7948db0 System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)
0000000004c1df90 000007fee7948cdb System.Web.UI.Page.ProcessRequest(Boolean, Boolean)
0000000004c1dff0 000007fee7948c70 System.Web.UI.Page.ProcessRequest()
0000000004c1e050 000007ff001e0c09 System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)
0000000004c1e0b0 000007fee7950117 ASP.default_aspx.ProcessRequest(System.Web.HttpContext)
0000000004c1e0e0 000007fee791449b System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
0000000004c1e190 000007fee7ffbd41 System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)
0000000004c1e230 000007fee7fed132 System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception)
0000000004c1e3c0 000007fee7fcf599 System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, System.AsyncCallback)
0000000004c1e410 000007fee80f5344 System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, System.Web.HttpContext)
0000000004c1e530 000007fee80f65bb System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
0000000004c1e6a0 000007fee80f4994 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
0000000004c1e700 000007fef72701ea DomainNeutralILStubClass.IL_STUB(Int64, Int64, Int64, Int32)
0000000004c1ef30 000007fee80f5424 DomainNeutralILStubClass.IL_STUB(IntPtr, System.Web.RequestNotificationStatus ByRef)
0000000004c1f010 000007fee80f65bb System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
0000000004c1f180 000007fee80f4994 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
0000000004c1f1e0 000007fef727043b DomainNeutralILStubClass.IL_STUB(Int64, Int64, Int64, Int32)
OS Thread Id: 0x1e28 (27)
Child-SP         RetAddr          Call Site
000000000492dcf0 000007ff001e13d2 FileIO.performLockOperation()
000000000492dd70 000007feee323ec9 _Default.Page_Load(System.Object, System.EventArgs)
000000000492ddb0 000007fee7956aea System.Web.Util.CalliHelper.EventArgFunctionCaller(IntPtr, System.Object, System.Object, System.EventArgs)
000000000492dde0 000007fee794d2c4 System.Web.Util.CalliEventHandlerDelegateProxy.Callback(System.Object, System.EventArgs)
000000000492de10 000007fee794d322 System.Web.UI.Control.OnLoad(System.EventArgs)
000000000492de50 000007fee79498ac System.Web.UI.Control.LoadRecursive()
000000000492dea0 000007fee7948db0 System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)
000000000492df70 000007fee7948cdb System.Web.UI.Page.ProcessRequest(Boolean, Boolean)
000000000492dfd0 000007fee7948c70 System.Web.UI.Page.ProcessRequest()
000000000492e030 000007ff001e0c09 System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)
000000000492e090 000007fee7950117 ASP.default_aspx.ProcessRequest(System.Web.HttpContext)
000000000492e0c0 000007fee791449b System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
000000000492e170 000007fee7ffbd41 System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)
000000000492e210 000007fee7fed132 System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception)
000000000492e3a0 000007fee7fcf599 System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, System.AsyncCallback)
000000000492e3f0 000007fee80f5344 System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, System.Web.HttpContext)
000000000492e510 000007fee80f65bb System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
000000000492e680 000007fee80f4994 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
000000000492e6e0 000007fef72701ea DomainNeutralILStubClass.IL_STUB(Int64, Int64, Int64, Int32)
000000000492ef10 000007fee80f5424 DomainNeutralILStubClass.IL_STUB(IntPtr, System.Web.RequestNotificationStatus ByRef)
000000000492eff0 000007fee80f65bb System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
000000000492f160 000007fee80f4994 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
000000000492f1c0 000007fef727043b DomainNeutralILStubClass.IL_STUB(Int64, Int64, Int64, Int32)
OS Thread Id: 0x1c38 (28)
Child-SP         RetAddr          Call Site
0000000004d0e0b0 000007ff001e13d2 FileIO.performLockOperation()
0000000004d0e130 000007feee323ec9 _Default.Page_Load(System.Object, System.EventArgs)
0000000004d0e170 000007fee7956aea System.Web.Util.CalliHelper.EventArgFunctionCaller(IntPtr, System.Object, System.Object, System.EventArgs)
0000000004d0e1a0 000007fee794d2c4 System.Web.Util.CalliEventHandlerDelegateProxy.Callback(System.Object, System.EventArgs)
0000000004d0e1d0 000007fee794d322 System.Web.UI.Control.OnLoad(System.EventArgs)
0000000004d0e210 000007fee79498ac System.Web.UI.Control.LoadRecursive()
0000000004d0e260 000007fee7948db0 System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)
0000000004d0e330 000007fee7948cdb System.Web.UI.Page.ProcessRequest(Boolean, Boolean)
0000000004d0e390 000007fee7948c70 System.Web.UI.Page.ProcessRequest()
0000000004d0e3f0 000007ff001e0c09 System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)
0000000004d0e450 000007fee7950117 ASP.default_aspx.ProcessRequest(System.Web.HttpContext)
0000000004d0e480 000007fee791449b System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
0000000004d0e530 000007fee7ffbd41 System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)
0000000004d0e5d0 000007fee7fed132 System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception)
0000000004d0e760 000007fee7fcf599 System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, System.AsyncCallback)
0000000004d0e7b0 000007fee80f5344 System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, System.Web.HttpContext)
0000000004d0e8d0 000007fee80f65bb System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
0000000004d0ea40 000007fee80f4994 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
0000000004d0eaa0 000007fef72701ea DomainNeutralILStubClass.IL_STUB(Int64, Int64, Int64, Int32)
0000000004d0f2d0 000007fee80f5424 DomainNeutralILStubClass.IL_STUB(IntPtr, System.Web.RequestNotificationStatus ByRef)
0000000004d0f3b0 000007fee80f65bb System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
0000000004d0f520 000007fee80f4994 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
0000000004d0f580 000007fef727043b DomainNeutralILStubClass.IL_STUB(Int64, Int64, Int64, Int32

So, what does this tell us?

Well, obviously we have four threads that call my function FileIO.performLockOperation() from Page_Load.

Please note that we don't have any additional callstack, and since the performLockOperation-function was named by me it could just as well have been named "foo". In order to find out that these threads are actually waiting for a lock we need to look at the native callstack using the kb-command. Let's take a look at thread 24:

0:029> ~24kb
ntdll!NtDelayExecution+0xa
kernel32!SleepEx+0x84
mscorwks!EESleepEx+0x2d
mscorwks!Thread::UserSleep+0x71
mscorwks!ThreadNative::Sleep+0xf9
App_Code_tvh3usvo!FileIO.performLockOperation()+0x8a
App_Web_uywrbugj!_Default.Page_Load(System.Object, System.EventArgs)+0x32
System_Web_RegularExpressions_ni!System.Web.Util.CalliHelper.EventArgFunctionCaller(IntPtr, System.Object, System.Object, System.EventArgs)+0x19
System_Web_ni!System.Web.Util.CalliEventHandlerDelegateProxy.Callback(System.Object, System.EventArgs)+0x2a
System_Web_ni!System.Web.UI.Control.OnLoad(System.EventArgs)+0x84
System_Web_ni!System.Web.UI.Control.LoadRecursive()+0x42
System_Web_ni!System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)+0x97c
System_Web_ni!System.Web.UI.Page.ProcessRequest(Boolean, Boolean)+0xa0
System_Web_ni!System.Web.UI.Page.ProcessRequest()+0x5b
System_Web_ni!System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)+0xf0
App_Web_uywrbugj!ASP.default_aspx.ProcessRequest(System.Web.HttpContext)+0x9
System_Web_ni!System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()+0x257
System_Web_ni!System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)+0xab
System_Web_ni!System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception)+0x501
System_Web_ni!System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, System.AsyncCallback)+0x72

Apparently we entered FileIO.performLockOperation, and then called sleep, (which is what we're doing in this repro in order to simulate a slow IO operation.) So, what do the other threads look like? Here's thread 26:

0:029> ~26kb
ntdll!ZwWaitForMultipleObjects+0xa
kernel32!WaitForMultipleObjectsEx+0x10b
mscorwks!WaitForMultipleObjectsEx_SO_TOLERANT+0xc1
mscorwks!Thread::DoAppropriateAptStateWait+0x41
mscorwks!Thread::DoAppropriateWaitWorker+0x191
mscorwks!Thread::DoAppropriateWait+0x5c
mscorwks!CLREvent::WaitEx+0xbe
mscorwks!AwareLock::EnterEpilog+0xc9
mscorwks!AwareLock::Enter+0x72
mscorwks!AwareLock::Contention+0x1fb
mscorwks!JITutil_MonContention+0xdf
App_Code_tvh3usvo!FileIO.performLockOperation()+0x7f
App_Web_uywrbugj!_Default.Page_Load(System.Object, System.EventArgs)+0x32
System_Web_RegularExpressions_ni!System.Web.Util.CalliHelper.EventArgFunctionCaller(IntPtr, System.Object, System.Object, System.EventArgs)+0x19
System_Web_ni!System.Web.Util.CalliEventHandlerDelegateProxy.Callback(System.Object, System.EventArgs)+0x2a
System_Web_ni!System.Web.UI.Control.OnLoad(System.EventArgs)+0x84
System_Web_ni!System.Web.UI.Control.LoadRecursive()+0x42
System_Web_ni!System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)+0x97c
System_Web_ni!System.Web.UI.Page.ProcessRequest(Boolean, Boolean)+0xa0
System_Web_ni!System.Web.UI.Page.ProcessRequest()+0x5b

This is one of the threads waiting for the critical section. It then found that it was unable to get a lock and so began waiting for the critical section on thread 24 to end.

Resolving the problem

This is the tricky part. There is no quick-fix for this. We can use the debugger to identify the bottleneck, but after that we'll need to review our design in order to get this working smoothly. A couple of generic things to try would be the following:

• Review what you're doing inside the critical section. Is it really necessary to perform all of the operations you're doing, or would it be possible to do some/all of them outside the scope of the critical section?
• Identify the reason for the lock taking so much time. Can it be resolved by load-balancing or clustering the resource we're waiting for?
• If we're waiting for another server, make sure the physical infrastructure is okay. Are network response times within the expected parameters or do you need to fix your wiring, router, etc?
• Are all calls to the critical section taking the same amount of time, or are there certain calls that take an exceptional amount of time? Perhaps most calls to the function are just fine, but intermittently a call comes along that tries to get everything from the AD, database, etc. In order to investigate this I'd use the commands outlined in Getting started with windbg -part II. Identify the thread owning the lock and investigate the callstack and stack objects using !clrstack and !dso. Use !dso and !do to query the stack objects and try to find the SQL query or AD query that is causing the thread to take such a long time to finish. Is the query significantly different from the ones waiting to execute, then this is most likely our culprit.

Post Script

A quick thought on locks and their limits:

Locks do not scale well. A lock can only be made within a specific process, so if you're using a web farm, or in any other way more than one w3wp.exe, then you will not be able to lock all processes. In other words. Server 1 can get a lock, and so can Server 2. Both will happily execute their code independent of each other. For products such as SQL server you'd deal with this in your SQL query, but if you're performing other operations, such as File IO you might have to work around this by using the same approach as Ms Word. Create a lock file in a shared space, and before you enter your critical section you simply check if the file exists:

1. You are entering your critical section. Look for lock file in shared location
2. If file exists, then wait
3. If/when file doesn't/no longer exist, create file and enter critical section
4. Make sure you have solid exception handling within the critical section so that you under no circumstances leave the critical section without removing the lock file.
5. When done, remove lock file

/ Johan

Problem description

The client was building a kiosk-like application. I.e. a WinForms application with the WebBrowser control slapped onto it. The application worked fine for the most part, but they'd found that when viewing video via facebook they were only able to view the video the first time. Any additional attempt to view the video would fail with a custom error from the facebook application stating that the video was either restricted or no longer available.

This only happened when viewing the page via the WinForms application. Using IE directly worked just fine. We also found that this problem occurred with both IE7 and IE8 installed, so it seemed not to be related to the browser version.

Other sites using the Adobe Flash player to display video, such as MSN or Youtube worked just fine.

Troubleshooting

Using the Developer tools included in IE8 I found that the facebook application displays a placeholder image with a play-button. Once that button is clicked the .swf player is loaded via client-side script. The .swf player then downloads the .mp4 that is to be displayed and divides it into suitable chunks.

Fiddler traces revealed that the mp4 was downloaded when using IE but not when using the WinForms application. No errors were reported, instead the request was simply never made.

We removed all code behind. All that remained was the On_Load event that set a starting page for the WebBrowser control, still we were unable to successfully load the movie.

Adding facebook to the trusted zone changed nothing.

Using the COM-version of the WebBrowser control still reproduced the problem.

I then suspected restrictions in cross-domain access or similar, (facebook uses a lot of different domains,) but couldn't find anything conclusive.

Finally I stumbled upon the root cause.

Cause

The problem is due to a bug within the Adobe Flash-player: http://bugs.adobe.com/jira/browse/FP-256

The problem occurs when dynamically trying to display a cached .swf application with an external interface through script. So, another quick repro would be the following (Copied from the Adobe bug report):

1. Create a C# application with an embedded WebBrowser control.
2. Load a HTML page with no content.
3. Via JavaScript, add a SWF with external interface to the page via an object tag. (At this point I see server traffic, requesting the SWF.)
4. From JavaScript, access the external interface of the SWF. Works OK.
5. Browse to a second web page.
6. Browse back to the empty one.
7. Via JavaScript, add the SWF again. Cannot contact the external interface. (There is no server traffic, indicating that the SWF is already being cached.)

Solution

While waiting for a fix from Adobe the only way to circumvent this is to manually remove the cached .swf. Article 262110 describes in detail how you can do this. Unfortunately this can be a bit tricky some times. In the sample repro above you'd resolve it by simply removing all .swf items from the cache upon page-load, but this wouldn't work in the facebook situation, since it might use the same control repeatedly without refreshing the page in between. Depending on your level of control over the target platform it might be easier to simply turn of caching.

/ Johan

The reason for this is that the RODC is quite picky about what server roles and features it will allow you to install. I recommend looking at the following articles, but they still don't cover everything.

Applications That Are Known to Work with RODCs
http://technet.microsoft.com/en-us/library/cc732790.aspx

Planning for Application Compatibility with RODCs
http://technet.microsoft.com/en-us/library/cc731746.aspx

The main lesson to be learned is that although the application is listed as known to work with RODCs that doesn't necessarily mean it installs well once the server has been promoted to RODC. When it comes to FTP you have two valid options to get this working:

1. Install the FTP server before promoting the server to RODC
2. Install the FTP Service 7.5 for IIS7. This is a stand-alone installer and works just fine.

/ Johan