Parvez's Blog https://docs.microsoft.com/archive/blogs/parvez/feed.xml Tue, 16 Apr 2019 06:20:05 GMT en-US hourly 1 IIS Application Pool Crash and Debug Diag https://docs.microsoft.com/archive/blogs/parvez/iis-application-pool-crash-and-debug-diag Sat, 06 Aug 2016 22:08:58 GMT https://blogs.msdn.microsoft.com/parvez/?p=115 Introduction If you observe either of the following events under the system logs in Event Viewer, then this article is meant for you. A process serving application pool '%1' suffered a fatal communication error with the Windows Process Activation Service. The process id was '%2'. The data field contains the error number. A process serving application pool '%1' terminated unexpectedly. The process id was '%2'. The process exit code was '%3'. This is a clear indication of an application pool crash issue wherein something went wrong within the application and the application pool had to be terminated. Once the application pool is terminated, its corresponding w3wp process is also terminated. Hence, if you have saved any cache/sesson based information within the w3wp process, it gets completely wiped out. Ideally, when an application pool crashes, in order to honor the incoming requests, a new w3wp process gets spawned up automatically. However, if the application pool crashes more than 5 times within a period of 5 minutes, then the application pool goes into a stopped state. You will have to restart the application pool manually to get it up and running. You will observe the following event under system logs in Event Viewer if something similar happens. Application pool '%1' is being automatically disabled due to a series of failures in the process(es) serving that application pool. You can modify this setting under "Rapid Fail Protection" section present under Advanced settings of the respective application pool. It is "Enabled" by Default.

1

Capture Dumps Now that we know that our application pool has crashed, the next step is to determine why exactly it crashed. In order to determine this, we will have to capture a dump of the w3wp process just before it terminates. There are multiple ways to do this. You can set up WER, procdump and Debug Diag to capture a crash dump. In this article, I will discuss only the Debug Diag  way of capturing data. Debug Diag This is by far the easiest and the most sophisticated tool to capture a crash dump of an application pool. This is how we set it up. You can download it from here. Launch debug diag and select “crash” rule.

2

Select “A specific IIS web application pool”

3

Select the specific application pool which is crashing.

7

You might also receive a pop up indicating that the IIS management compatibility feature is not installed and that the application pools will not be listed. Not to worry ! Just click on OK and type in the application name manually in the above window.

Select “Breakpoint”

4

Click on “Add breakpoint”

5

Click on “Ntdll!ZwTerminateProcess” under Breakpoint such that it is selected as the Breakpoint Expression. Change Action Type to “Full userdump” and set Action Limit to 10 and click on OK. 

6

Action Type indicates the type of dump that would be captured. A mini user dump will not provide complete information of the issue as it captures very minimal data of the w3wp process.

Action Limit indicates the number of dumps that this rule will capture.

Click on Save and close

13

Click on Next to activate the breakpoint

12

Click on "Next"

 21

In the above section, you can change the path where the dumps will be stored. The size of each dump file will be the same as that consumed by w3wp process in the memory. So, keeping that in mind, place the dumps in a drive that has enough free disk space.

Click on "Finish" to activate the Rule.

20

You will now see that the crash rule is in active state and the "Userdump Count" is 0. As soon as the issue occurs, the dump count will increase and a corresponding dump file will be generated.

PS: A normal recycle of the application pool will also trigger a dump. This is because when the application pool recycles, its corresponding w3wp process's PID changes and hence a dump gets generated. This is a false positive dump and will not help in the analysis of the application pool crash issue. So whenever you see an increment in the userdump count, check the event logs to see if the above mentioned crash events have occurred. If yes, then the dump that has been captured is correct.

Dump Analysis

Once the dump is captured, you can use the Debug Diag Analysis executable to analyse the captured crash dump. Just a "double click" on the dump will also work. Make sure you have the symbol path set right. To check this, select Tools --> Options and settings and make sure your "Symbol Search Path for Analysis" and "Symbol Search path for Debugging" are set to "srv*c:\symcache*https://msdl.microsoft.com/download/symbols".

14

Once the report is generated, you can open it in IE and understand the call stack of the thread that triggered the Breakpoint Expression. Read the call stack from bottom to top and figure out which method or component triggered the application pool to crash

Hope this helps :)

   ]]> IIS Bindings https://docs.microsoft.com/archive/blogs/parvez/iis-bindings Wed, 27 Jul 2016 18:11:18 GMT https://blogs.msdn.microsoft.com/parvez/?p=75 Introduction A binding is an attribute of a website which is used to uniquely identify itself and consists of a combination of IP, Port and the Hostname. Basically Binding = IP + Port + Hostname Confusion So if its that simple, where and when does the confusion arise? Consider a situation where you have multiple websites hosted on an IIS machine. When a request reaches this web server, how does it decide as to which website this request should go to? This is where the knowledge of Bindings can be useful. For two or more websites to run successfully on IIS, their Bindings have to be unique. It basically means that either one of the following attributes - IP, Port or Hostname has to be different. HTTP The default port that HTTP requests use is port 80.  Consider an example where website A has the following Binding. IP: 10.10.10.1 Port:80 Hostname: contosso.com If another website website B has to be hosted on the same server, atleast one of the above attributes has to be different. So, Website B can have either of the following Bindings.
  • IP: 10.10.10.1 Port:80 Hostname: Test.com
  • IP: 10.10.10.1 Port:8080 Hostname: Contosso.com
  • IP: 20.20.20.2 Port:80 Hostname: Contosso.com
Option 1 is something that is feasible, scalable and easier to manage. Since port 80 is the default port of HTTP protocol,  if a website is listening on any other port other than 80 , it has to be specified by the user in the browser which becomes a little tedious. Ideally, each website will be accessed using its Hostname. So having a different Hostname rather than a different Port makes more sense. This way, you can host multiple websites without any difficulty. Option 2 is generally used when you have website B as the backup for website A. Website A usually contains the latest code release and website B contains the last working code. If, for some reason website A goes down, the administrator can immediately stop website A, modify the port number of website B and get the last working code up and running. Option 3 is used if you have an additional IP address added to the machine. HTTPS - Windows server 2008 R2 The default port that HTTPS requests use is port 443. There is a small twist in the way requests are routed to individual websites in case of websites running over HTTPS. As we discussed earlier, to uniquely establish two websites having HTTP bindings, we used the combination of IP, Port and Hostname. However, in case of two or more websites having HTTPS binding, they can be established using the combination of IP and Port only. This is because, during the SSL handshake, only IP and Port details are available. Hence, hostname does not play any part in determining the website that the request should go to. If you observe the Bindings section while adding an HTTPS binding, the Hostname section will be greyed out. 1 Consider an example where website A has the following Binding. IP: 10.10.10.1 Port:443 If another website website B has to be hosted on the same server, either of the above attributes has to be different. So, Website B can have either of the following Bindings.
  • IP: 20.20.20.2 Port:443
  • IP: 10.10.10.1 Port:445
Option 1 is used if you have an additional IP address added to the machine. This is the most feasible option for HTTPS websites on a windows server 2008 R2 machine. Option 2 can be used but is not feasible since it uses a port number that is different than the default HTTPS port(443). The disadvantage is that the users will have to specifically mention the port number in the URL of the website that they intend to access. Example: https://www.contosso.com:445 Special case of using wildcard certificates - Windows server 2008 R2 Consider an example where website A has the following Binding - IP: 10.10.10.1 Port: 443 and is supposed to be accessed using the Hostname: website1.contosso.com If another website website B has to be hosted on the same server with the Hostname: website2.contosso.com, then we can assign the same IP and Port combination to both the websites in the IIS manager. But two websites running with the same IP and Port combination will result in a conflict. So, in order to uniquely identify the websites and make sure that the request reaches the correct website, the "Bindings" section under "sites" attribute in the applicationHost.config file will have to be modified in the following manner. For website A, <binding protocol="https" bindingInformation="10.10.10.1:443:website1.contosso.com" /> For website B, <binding protocol="https" bindingInformation="10.10.10.1:443:website2.contosso.com" /> HTTPS - Windows server 2012 R2 The major disadvantage with windows server 2008 R2 machine was Scalability. Hosting multiple websites required users to configure multiple IP addresses on the machine or have the users access the website using an unconventional port. In order to rectify this problem, a very important feature called "Server name Indication(SNI)" was introduced. 2 SNI allows us to utilize all three attributes of a binding - IP , Port and Hostname that we used previously with HTTP based websites to uniquely identify themselves. Most of the new browsers support the SNI feature. You can refer to this blog for more information on SNI and how it can be configured. Summary HTTP = IP + Port + hostname HTTPS [windows server 2008 R2] = IP + Port HTTPS [windows server 2012 R2] = IP + Port + Hostname Hope this helps :)        ]]> Authentication Precedence and Behavior in IIS https://docs.microsoft.com/archive/blogs/parvez/authentication-precedence-in-iis Sat, 21 Sep 2013 10:48:00 GMT https://blogs.msdn.microsoft.com/parvez/2013/09/21/authentication-precedence-in-iis/ Introduction

This blog seeks to explain which authentication mode will take precedence when multiple authentication mechanisms like Anonymous, Basic, Windows are enabled. Please note that the following instructions are applicable only to ASP.Net applications and Internet Explorer.

Authentication Precedence

The best way to find out which authentication mode will take precedence is to revisit the IIS 6 Directory security Tab. The order in which authentication modes are presented from top to bottom indicates the authentication precedence. For example, Anonymous Authentication will take precedence over Windows, Digest and Basic authentication. However, if the client Certificate Authentication is configured, then it will take precedence above all other authentication mechanisms since the SSL handshake takes place even before the HTTP request comes into picture.

To Summarize everything, this is how the authentication precedence looks like.

  • Client Certificate Authentication
  • Anonymous authentication
  • Windows authentication
  • Basic authentication

Authentication Behavior

Scenario 1 - Only Anonymous authentication enabled

Under any given circumstances, when a client makes the first request to the server, it will be anonymous. On the server side, if the application is also configured to run under anonymous authentication, then the server will process the request and send the desired response back to the client without requesting any further validation from the client.

Scenario 2 - Multiple authentication modes enabled

Let us assume that both Basic and Windows authentication have been enabled on the server at the application level. When the first anonymous request from the client reaches the server, the server checks the application's configuration and understands that it is configured to run under Basic/Windows authentication. Since these authentication mechanisms need further validation to deliver the required page, the server sends the authentication details to the client and requests for further validation. The client picks up the most secure authentication mechanism(windows) from the list provided by the server and takes necessary steps to provide the required information to the server. It is incorrect to assume that if two authentication modes are configured and if one fails, the request will fall back to other authentication mode. If you consider my previous example, I stated that if Windows Authentication and Basic Authentication are enabled, Windows authentication will take precedence. Now, if Windows authentication fails, it will never fall back to Basic and take it forward from there.

A fall back would mostly occur only in following scenarios.

  • Scenario 1: Under Windows Authentication, if Kerberos fails, it will fall back to NTLM
  • Scenario 2: If client Certificate Authentication and Anonymous authentication are enabled on the server at the application and if client Certificate Authentication fails, then the request will fall back to Anonymous authentication. If you're setting up client certificate authentication in your environment, please make sure you have anonymous authentication disabled so that you don't end up receiving false positive answers.
Hope this helps :)]]>