When you have enabled client certificate authentication either using AD client certificate authentication or IIS client certificate mapping authentication you can follow the below steps in general.

The most common error or status code you would face with respect to client certificate authentication is 403.7 which means the server is expecting a client certificate but the client is not sending it.

Follow the steps in order to check what the issue might be

1) Make sure you have set Require for client certificate in SSL settings in IIS

2) When using IIS client certificate mapping we either use one to one mapping where we map individual certificate to an user or many to one mapping where we map all the certificates matching a criteria to a single user. The one to one mapping or the many to one mapping should be created at the site level. The most common mistake, I have seen is if people want to use client certificate authentication to a specific application under a site, they create the mappings at the application level which is wrong. You will have to create the mappings at the site level and enable client certificate authentication at the required level. The mappings will only be read at the site level and any mapping at the application level will be ignored. This is not mentioned in most of the internet articles.

3) On the client machine in IE make sure you go to Internet Options->Security->Intranet or Internet(Based on the type of site)->Miscellaneous->Don't prompt for client certificate when only one certificate is disabled. We need to make sure it is disabled so that at least we get to know if we get a prompt for certificate selection and if this fails out after clicking on certificate or without even sending a certificate.

4) In the client machine, confirm that the client certificate has a private key. You can check this by going to mmc->Add or remove snapin->Certificates(My User Account)->Personal->Certificates and then select the client certificate which you have installed for authentication. If the certificate is proper then you should be able to see the private key as below. ALso you can run the below command to check the validity of the certificate

certutil –verifystore my <Thumbprint of the certificate>

5)  Also go to details tab of the certificate and click on Enhanced Key usage where you should see that this certificate is intended for client certificate authentication which also validates the certificate to be a genuine client certificate.

6) If you go to the Certification Path tab in the certificate, you will be able to see the certificate chain. Double Click on each certificate in the chain and make a note of it.

7) In the certification path the certificate at the top is called the Root Certificate and the certificates in between the Root CA and the actual certificate in the chain are called Intermediate CA’s. In the above image there is a single intermediary CA but there can be more than one intermediary CA as well. In the server machine and as well as the client machine go to mmc->Add or Remove Snapin->Certificates Local Computer->Trusted Root Certification Authorities->Certificates, make sure the Root CA is present in this location. Next go to Intermediate Certification Authorities->Certificates and make sure the Intermediate CA’s are present there. This step is important because during SSL handshake when the server requests for the client certificate it sends the list of Trusted Issuer List which will contain the list of the Trusted Root CA’s installed on the server to the client and based on the that the client certificate to be sent is chosen.

8) The feature of sending the CTL or trusted list can also be controlled by setting the below registry key which can be used from windows 2008 and onwards. If the value is set to 1 then the list is send and if the value is 0 then the list will not be sent. In windows 2012 and onwards the default value is 0 and in the lower versions the default value is 1 if the key is not present.

9) There is one more thing you need to cross check if this is a windows 2012 (IIS 8) and higher machine. By default the CTL is off, if the CTL is on and if in the server mmc->Add or Remove Snapin->Certificates Local Computer->Trusted Root Certification Authorities->Certificates you have a non-self signed certificate then you might receive a 403.7 error as well. More details on the issue is logged in the below kb. The solution is to delete the non-self signed certificate from the Trusted Root Certification Authority

https://support.microsoft.com/en-in/kb/2802568

10) Thanks to Daniel Stolt, i had missed out a point. If the client is a web application or a web service and you are sending the client certificate from the code then the additional thing you will have to check is if the identity of the client application has permissions on the machine key for the client certificate. There are 2 ways you can do it.

Method A: Go to mmc->Add or Remove Snapin->Certificates Local Computer or Current User->Personal->Certificates. Right Click on the correct client certificate->Manage Private Keys and give permissions to the identity or user under which the client application is running.

Method B: This is bit complicated but interesting method. In this method we find out the machine key linked with the client certificate and then give the proper permissions required.

a. Download and install "find private key" application
http://blog.rhysgoodwin.com/windows-admin/findprivatekey-exe-pre-compiled/

b.  Find the Thumbprint of the client certificate
Run -> MMC -> File-> ADD or REMOVE SNAP IN->certificates-> Local Computer or Current User->Ok
Expand Personal->Certificates->Choose the appropriate certificate and open it
In the certificate->details tab->Thumbprint->Copy it

c. find the private key path for the certificate suing the thumbprint copied
Syntax:
FindprivateKey.exe My LocalMachine -t "THumbprint of the certificate"

d. Next, we need to provide permissions to application pool account to certificate

syntax (account to use : app pool identity - in our case it is NetworkService):
icacls "private key file path" /grant "Account":R

11) Also several times i have seen that 403.7 is just masking the actual error. On the client you get a 403.7 but there might be a different error before that. For this please check the logs. One common scenario i have seen is as below. We see a 500 just before the 403.7 error and this is visible in logs as below

If we take Failed request traces for the 500 we see below

In this scenario normally you would have got a prompt for the certificate in IE but you end you getting 403.7 still in the end because the certificate was not read completely. Normally when we just set certificate to Require in SSL settings, the client certificate negotiation happens in later part of the request. To avoid the above error we have to enable clientcertificatenegotiation so that the negotiation happens during ssl handshake during the server hello is sent. To enable client cert negotiation you can follow the below blog

http://blogs.iis.net/rickbarber/require-client-certificates-in-iis-7-5

The blog talks on how to enable by deleting and recreating the bindings from command prompt. There is also a hack which we can follow as below.

For example if you have a SSL binding for IP: PORT 10.10.10.10:443 then traverse to the registry path HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\HTTP\Parameters\SslBindingInfo\10.10.10.10:443

Create a DWORD called DefaultFlags and set the value to 2. The commands mentioned in the blog also does the same

The above changes require a machine reboot to take effect.

12) The above steps sums up the configuration part from the client and server perspective. But i have seen 403.7 error pop because of network issues as well. I have seen the issue happen If there is some intermediary device or firewall or proxy which doesn’t support client certificate authentication or negotiation. If we have set clientcertificatenegotitation as above then we will be able to see the client certificate request in the earlier stage of the handshake itself as below

if we don’t have clientcertificatenegotiation then we might not be able to see the certificate request(Server Hello) in the handshake as it might happen later and it will be encrypted.

If you have the setting enabled and you still don’t see the client certificate request and see something like below then this is a possible issue with a network device

As one more final step you can also try enabling SCHANNEL logging. Steps are in the below link

https://blogs.msdn.microsoft.com/chiranth/2014/02/18/ssl-troubleshooting-troubleshooting-steps-for-server-side-ssl-problems/

Hope this helps

1) ISAPI 'C:\Windows\Microsoft.NET\Framework\v2.0.50727\aspnet_isapi.dll' reported itself as unhealthy for the following reason: 'Deadlock detected'.

2) ISAPI’C:\Windows\Microsoft.NET\Framework64\v4.0.30319\aspnet_isapi.dll’ reported itself as unhealthy for the following reason: ‘Deadlock detected’”

3) Description: ISAPI 'C:\Windows\system32\inetsrv\asp.dll' reported itself as unhealthy for the following reason: 'ASP unhealthy because 100% of executing requests are hung and 0% of the request queue is full.'

In the below blog or link i have discussed as to why we see this error and what are the steps we need to follow to automate the data collection in case we see this issue to find out the root cause.

https://blogs.msdn.microsoft.com/webtopics/2016/04/05/data-collection-for-deadlock-detected-issue-of-a-iis-based-web-application/

Hope this helps

When we try to change the application pool identity of an application pool to a domain account we get the below error while committing the changes.

The error code might change accordingly but you will get the text as keyset does not exist while trying to update the application pool identity with a custom account or while trying to specify a connect as user in the basic settings of a site.

You might also get the below error codes

0x80090016

0x80090005

Why am I seeing this issue?

This issue happens when your IIS specific machine keys are corrupt or missing.

Your machine keys might go corrupt if there was an improper shutdown of the machine or the machine was cloned from an existing image or if there was a system crash.

Its always a good idea to have a backup of the machine keys.

Before I explain in detail what goes wrong internally first lets discuss about the different machine keys required in IIS for encryption. The location of the key depends on the operating system is basically C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys from windows 2008 R2 and onwards

The above 3 keys are used by IIS 7 and higher versions for encryption of the configuration.

The pertaining configuration info having information about these keys in IIS config looks something like below and will be present in C:\Windows\System32\inetsrv\config\applicationHost.config

IIS 6.0 uses a particular key c2319c42033a5ca7f44e731bfd3fa2b5_GUID to do the encryption. If you are seeing IIS admin service start up issue on an IIS 7 machine or IIS 6 machine, first thing we would check is if the metabase.xml is corrupted or if c2319c42033a5ca7f44e731bfd3fa2b5_GUID is corrupted.

In IIS any sensitive data like passwords won’t be stored in plain text and will have to be encrypted before it is stored in the config and get decrypted accordingly when needed.

So in case if the above machine keys are corrupted the configuration in IIS and the keys go out of sync and any encryption or decryption task will start failing. That is the reason you might see the above error while changing the application pool identity to a domain or custom account. If you change the application pool identity to an account like local system or network service or service accounts which doesn’t need password and hence no encryption the task won’t fail.

To confirm that this is the issue with machine key corruption you can try changing connect-as account for any website or virtual directory in IIS to a custom account and you will see the same error.

How to fix the issue?

Most of the times we suggest to rebuild or reinstall IIS. But what if reinstalling IIS is not an option?

Let’s see if there is something else we can do before we end up reinstalling IIS.

Before you proceed with the below steps it’s better to take backup of the IIS configuration so that you need not configure everything from the scratch in case you have to reinstall IIS.

To take backup of the IIS configuration install msdeploy from http://www.iis.net/downloads/microsoft/web-deploy

Once you install web deploy and reopen the inetmgr console you will see the option to export a server package and the root node as below.

Click on Export server package.

You would want to encrypt the configuration with your own password. To do that Click on Advanced Settings.

Enter the encryption password. You would want to remember this password while importing the package.

Click on Ok and Next.

Save the package or zip file which you can use to import later if needed

To import the package you will have to follow similar steps by clicking on Import package at the root node of IIS and then entering the encryption password.

How to work around the machine key corruption?

So enough of why and Now coming back to the main issue as to how we can fix the machine key corruption and the below errors.

We can regenerate these machine keys without reinstalling IIS completely. To do that follow the below steps.

The above steps should regenerate the machine keys and also the configprotecteddata section in applicationhost.config

If this doesn’t help I guess I think most preferable option would be to reinstall IIS to fix the corruption.

Hope this helps J

Most common issues I have seen I  Windows Azure pack is the admin and tenant portal errors. When we see any runtime errors in the portals. Either the error might be because of configuration within the Admin/Tenant Site or it might be because the Admin API/Tenant API is throwing an error when a call is being made to them.

Things to Check:

First thing we do is check the event logs to see if there any errors in Application and Services Logs->Microsoft->Windows Azure Pack and check the logs for Admin/Tenant Site and the Admin/Tenant API sites. Most of the times this gives us insight on what might be happening

Second thing we do is we want to identify if the issue is in API or the site itself. To isolate this normally we want to keep a track of all the calls going from the site to the API internally. To keep a track of the out going calls we can actually hook fiddler to the admin site or tenant site.

Steps to capture fiddler for outgoing calls.

There are two methods we can use to achieve this. but there are some steps common to both.

Common Steps:

1. This is the most common method used. you can download and install Fiddler from http://www.telerik.com/download/fiddler

2. Before you start capturing the requests you need to make sure that fiddler is able to decrypt the https traffic.

3. To be able to decrypt the https traffic in fiddler start fiddler and go to Tools->Fiddler Options and make sure highlighted check boxes are selected. When prompted to install the fiddler certificate click on Yes.

Method 1:

1. Here you can fiddler as a proxy in the site web.config. But as we know that we cant make any changes directly to the WAP sites config as it is encrypted.

2. So before adding fiddler as a proxy, you need to decrypt the config by running below commands in the WAP PowerShell command let as below.

To decrypt the configuration of Admin Site

To decrypt the configuration of Tenant Site

Reference: https://technet.microsoft.com/en-us/library/dn520996%28v=sc.20%29.aspx?f=255&MSPPError=-2147217396

3. Take a backup of your original web.config before proceeding with any changes. Once you are done decrypting the web.config file, you can add the below entries in your web.config and make sure fiddler is running.

4. Now fiddler will start listening to traffic going outside of Admin Site/Tenant Site

5. After you are done capturing the data, revert the changes and encrypt the web.config again by running the below PowerShell command lets.

To encrypt the configuration of Admin Site

To encrypt the configuration of Tenant Site

I personally prefer method 2 as method 1 is a little tedious.

Method 2:

1. Create a file named setproxy.aspx page within admin site/ tenant site physical folder location depending for which site you want to capture the traffic.

2. Copy the below snippet to your setproxy.aspx page

3. Make sure fiddler is running and now browse the setproxy.aspx page. Now fiddler will be automatically attached to your process and will keep listening to the outgoing requests.

4. To revert the changes and stop fiddler from listening to your requests is very simple, you just need to recycle the application pool for the site and to enable it again just browse the page.

Hope this helps

The normal way we actually follow is as below.

1. Open the Debug Diagnostic tool.

2. Go to the Processes tab, right-click the worker process corresponding to the right application pool and select “Create Full Userdump”.

3. Repeat Step 2. Three times with a gap of 5-10 seconds in between and collect 3 sets of dumps for us.

But the above plan always doesn’t work if the issue is very intermittent and resolves on its own.

Another way to automate the collection as specified in the below blog.

http://blogs.msdn.com/b/benjaminperkins/archive/2013/12/02/using-procdump-and-failed-request-tracing-to-capture-a-memory-dump.aspx

http://blogs.msdn.com/b/webtopics/archive/2009/08/04/iis7-and-above-using-freb-to-capture-dumps-for-a-long-running-request.aspx

Over here I will be discussing how we can automate this collection to capture multiple dumps in a different way.

Note: Test all the below rules in your QA and testing server before porting the changes to your production box.

Create Freb rule:

1. Go to intemgr.exe, click on the site or application for which you are experiencing slowness or hang and go to Failed Request Tracing rules.

2. Click on Add rule and follow the below steps.

a. Select “All content” and click next. If we know that this is for a specific page, we can mention the page name as well by selecting Custom.

b. Select Time-Taken in seconds option and specify the time (if the page is taking around 30 seconds, you can specify 10 or 15 so that we start capturing the freb and dumps 15 seconds past execution till the end). Click Next

c. Click Finish

3. On the right hand pane at the site level under Configure go to Failed Request Tracing and enable it. You can configure it to the required location or leave it to the default.

Steps to Create Debug Diag Rule:

1. We will use Debug Diagnostic 2.0 update 2 version to troubleshoot this issue further. Please install the Debug Diagnostic 2.0 update 2 tool from https://www.microsoft.com/en-us/download/details.aspx?id=49924 (Please un-install all other versions of Debug Diagnostic tool before installing the latest version)

2. Before proceeding with the rule creation we need to download the symbols needed for the automated rule creation. Take the below steps to download the symbols for your w3wp process.

3. Create a Crash Rule in Debug Diagnostic 2.0 with the following steps:

v Open Debug Diag (Start -> Programs -> Debug Diagnostic Tool 1.2)

v Select "Crash" and click Next

v Choose the option “A specific IIS application pool” and click next and choose the application pool that is crashing from the list and click next. (Note: The Application pool list may be empty if the IIS 6 Metabase Compatibility is not installed. In such a case you can always type the name of the application pool and click Next)

v Click Next in Advanced Configuration (Optional), click on Breakpoints and then click on Add Breakpoint

v Add iisfreb!FREB_LOG_FILE_MANAGER::WriteLogFile and change the Action Type to Custom and add the below script in the script window and click on Ok.

v Set Action Limit to 5 and click on Ok.

v Click on Save and Close

v Click Next. Here you can set your "Rule Name" and also the "User dump Location" can be changed here.

v Select "Activate the rule now" and click Finish

v Notice the Status is Active. The User dump Count will increase each time a dump file is created.

v Wait for the issue to happen.

Once the issue occurs

1. The user dump count field in the Crash Rule will increment whenever the tool generates a dump. The dump will be generated in the User Dump Path

2. You can also analyze the Dump by selecting the Debug Diag Analysis tool and clicking Add Data Files. When the .dmp is added, select the Crash/Hang Analyzers script and click Start Analysis. When finished, a report (.mht) will be created in C:\Program Files\DebugDiag\Reports and displayed in Internet Explorer with the results and recommendations. If using custom DLL’s, the Symbol path (Tools menu -> Options and Settings -> Symbol Search Path) to the custom PDB files can be added.

Below are some of the data that we need to collect.

Steps to collect Procmon:

1. Download process monitor from https://technet.microsoft.com/en-us/library/bb896645.aspx

2. Run procmon.exe

3. Wait till the issue reoccurs and after the issue reoccurrence please stop the trace.

Note: Process monitor collects all the system level events and a lot of data. Even though this is a very good trace, if the issue you are facing is very intermittent avoid going for procmon data collection as this might consume more space and cause disk space issues.

Steps to Create Debug Diag Rule:

1. We will use Debug Diagnostic 2.0 update 2 version to troubleshoot this issue further. Please install the Debug Diagnostic 2.0 update 2 tool from https://www.microsoft.com/en-us/download/details.aspx?id=49924 (Please un-install all other versions of Debug Diagnostic tool before installing the latest version)

2. Before proceeding with the rule creation we need to download the symbols needed for the automated rule creation. Take the below steps to download the symbols for your w3wp process.

a. Download PDBDownloader.exe from https://github.com/rajkumar-rangaraj/PDB-Downloader/releases b. Run the PDBDownloader.exe and click on open files c. If the application we are creating the rule for is a .Net 2.0 64 bit app, Navigate to the path C:\Windows\Microsoft.NET\Framework64\v2.0.50727\ and select mscorwks.dll. Click on Start d. If the application we are creating the rule for is a .Net 2.0 32 bit app, Navigate to the path C:\Windows\Microsoft.NET\Framework\v2.0.50727\ and select mscorwks.dll. Click on Start e. If the application we are creating the rule for is a .Net 4.0 64 bit app, Navigate to the path C:\Windows\Microsoft.NET\Framework64\v4.0.30319\ and select clr.dll. Click on Start f. If the application we are creating the rule for is a .Net 4.0 32 bit app, Navigate to the path C:\Windows\Microsoft.NET\Framework\v4.0.30319\ and select clr.dll. Click on Start g. The tool will download the public symbols of mscorwks.dll or clr.dll to C:\Symbols folder and we are good to proceed with the rule creation

Note: The executable requires internet connection to download the symbols. If you don’t have internet connection on the server, then copy the dll specific to your framework and bitness to the machine which has internet connection and run the tool over there. Once you have the symbols downloaded, copy PDB file to the server’s symbol folder. In case of Debug Diag 1.2 or Debug Diag 2.0 Update 1 copy it to C:\Symcache, else use C:\Symbols

3. Create a Crash Rule in Debug Diagnostic 2.0 with the following steps:

a. Open DebugDiag (Start -> Programs -> Debug Diagnostic Tool 2.0 Collection)

b. Select "Crash" and click Next

c. Choose the option A specific IIS application pool and click Next and choose the application pool that is crashing from the list and click Next. (Note: The Application pool list may be empty if the IIS 6 Metabase Compatibility is not installed. In such a case you can always type the name of the application pool and click Next)

e. Click Next in Advanced Configuration (Optional), click on Breakpoints and then click on Add Breakpoint

f. If it is a .net 4.0/4.5 app then Add clr!AppDomainNative::Unload and change the Action Type to Full User Dump and Action Limit to 5 and click on Ok. If it is a .net 2.0/3.5 app then Add mscorwks!AppDomain::UnloadWait and change the Action Type to Full User Dump and Action Limit to 5 and click on Ok.

g. Select Action Type as Full User Dump.

h. Click on Save and Close

i. Click Next for "Rule Name". The "User dump Location" can be changed here.

j. Select "Activate the rule now" and click Finish

3. Notice the Status is Active. The User dump Count will increase each time a dump file is created.

4. Wait for the issue to happen.

If everything goes well you will process monitor trace and the process dump at the time of the issue.

How to analyse process monitor trace?

As we had discussed earlier we know that application domain recycles whenever there is some change to either the framework directory or to your application directory. So we will be interested to look what exactly is getting changed at the time of the issue and who is the culprit doing it.

I will show you some of the shortcuts that you can use to analyse this trace.

As I said most likely we will be interested in the writes to any directory.

Step 1: Go to Tools->Count Occurrences->Select Operation and then click on Count.

Step 2: We will be interested in WriteFile as below. Click on Filter. Click on close.

Step 3: Next we are interested in changes to either the framework directory or application directory. Go to Filter->Filter.
Step 4: In the first drop down select Path and in second drop down select contains and in the 3 text box, type in your application directory. For example in my case it would be C:\inetpub\wwwroot\TestUserIdentity452. Click on Add

Step 5: You will get a list of write file attempts as below.

Now you can double click on any one of the above by matching the time stamp of the app domain recycle in the event log. Here you will see which process is trying to write the file and the account used by the process.
Repeat from step 4 and try the above steps for the path contains “Framework” and see if there are any writes happening.

How to analyse the dumps captured using debug diag?

Sometimes I have seen that just running debug diag analysis will give you information on the app domain unload exception. If not I will discuss some tricks as to how we can debug this using windbg.

WinDBG basic setup instructions:

If you don’t have windbg you can go ahead and install windbg from https://msdn.microsoft.com/en-us/windows/hardware/hh852365.aspx

Select and download Standalone debugging tools for windows specific to your operating systems.

You can setup the symbols for your debugger or windbg by opening windbg and going to File->Symbol File Path and entering the below path

SRV*C:\symcache* http://msdl.microsoft.com/download/symbols

Prerequisites:

Download the netext.dll from http://netext.codeplex.com/ which will be using for analysis.

Copy the appropriate version dll’s inside your debugger installation folder.

How to analyse the dumps?

Open the dump in debugger.

Load the netext.dll in the debugger by typing in the below command.

.load netext

Before you can run any commands you need to run indexing for the dumps you have.

!windex

I have worked on 3 of the scenarios which might cause application domain recycles and below are the commands and the outputs you might get

Scenario 1: configuration change

0:035> !wfrom -type System.Web.FileMonitor where ($enumname(_lastAction) == "Modified") $a("==================\nPath",DirectoryMonitor.Directory+"\\"+_fileNameLong),$a("Change Time:",$tickstodatetime(_utcLastCompletion.dateData)) ================== Path: C:\inetpub\wwwroot\web.config Change Time:: 06-10-2015 20:20:39 1 Object(s) listed 14 Object(s) skipped by filter

Scenario 2: Bin directory change

0:036> !wruntime Runtime Settings per Application Pool ========================================================================= Address         : 000000F585279150 First Request   : 07-10-2015 01:25:51 App Pool User   : IIS APPPOOL\DefaultAppPool Trust Level     : Full App Domnain Id  : /LM/W3SVC/1/ROOT/TestUserIdentity452-1-130886547510004207 Debug Enabled   : False Active Requests : 0n0 Path            : C:\inetpub\wwwroot\TestUserIdentity452\ (local disk) Temp Folder     : C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files Compiling Folder: C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\testuseridentity452\cd5dfb1f\82daaaa9 Shutdown Reason : BinDirChangeOrDirectoryRename at 07-10-2015 01:26:49 Change Notification for critical directories. bin dir change or directory rename HostingEnvironment initiated shutdown Change Notification for critical directories. bin dir change or directory rename Change Notification for critical directories. bin dir change or directory rename HostingEnvironment caused shutdownn   at System.Environment.GetStackTrace(Exception e, Boolean needFileInfo)    at System.Environment.get_StackTrace()    at System.Web.Hosting.HostingEnvironment.InitiateShutdownInternal()    at System.Web.HttpRuntime.ShutdownAppDomain(String stackTrace)    at System.Web.HttpRuntime.OnCriticalDirectoryChange(Object sender, FileChangeEvent e)    at System.Web.FileChangesMonitor.OnCriticaldirChange(Object sender, FileChangeEvent e)    at System.Web.DirectoryMonitor.FireNotifications()    at System.Web.Util.WorkItem.CallCallbackWithAssert(WorkItemCallback callback)    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)    at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)    at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()    at System.Threading.ThreadPoolWorkQueue.Dispatch()

Scenario 3: Page change

0:033> !wfrom -type System.Web.FileMonitor where ($enumname(_lastAction) == "Modified") $a("==================\nPath",DirectoryMonitor.Directory+"\\"+_fileNameLong),$a("Change Time:",$tickstodatetime(_utcLastCompletion.dateData)) ================== Path: C:\inetpub\wwwroot\TestUserIdentity452\Default.aspx Change Time:: 09-10-2015 00:11:40 ================== Path: C:\inetpub\wwwroot\TestUserIdentity452\Default.aspx Change Time:: 09-10-2015 00:11:17 ================== Path: C:\inetpub\wwwroot\TestUserIdentity452\Default.aspx Change Time:: 09-10-2015 00:11:34 3 Object(s) listed 15 Object(s) skipped by filter

Reference: http://netext.codeplex.com/#wfrom

Hope this helps J

You might have observed sometimes that the application seems to be working fine and you observe one of the below 2 outcomes.

Ideally the first request for an asp.net application takes more time compared to the upcoming requests as I need to create an application domain within the process. Application domain is a processing environment within the process where your code is loaded and request is processed. This domain is the same place where your process specific objects are also stored. If you are using in-proc mode for session state, then those session objects also reside in this part of the process. So in case even your process is intact and just application domain within the process recycles or goes down, you might observe session loss as mentioned above

How to detect if the application domain/app domain is getting recycled?

You can follow one of the below steps

Step 1:

Enable health monitoring using the below steps.

Note: x is ‘32’ if 32 bit app and '64' if 64 bit app and y is ‘4.0.30319’ if .net 4.0 app and ‘2.0.50727’ if it is .net 2.0 app

Reference:

http://blogs.msdn.com/b/tess/archive/2006/08/02/asp-net-case-study-lost-session-variables-and-appdomain-recycles.aspx

Step 2:

Also you can enable the event logging on app domain recycling in code as suggested the below blog

http://weblogs.asp.net/scottgu/433194

Step 3:

You can add a Perfmon counter and monitor your server as below
1) Click Windows Start button -> Run. Type Perfmon and click Ok.
2) Expand Performance Logs and Alerts.
3) Right click Counter Logs and select New Log settings.
4) Type a name and click Ok.
5) Click on Add Counters button.
6) From the Performance object select ASP.NET v.1.1.4322 or the ASP.NET version of your application.
7) Choose Select counters from list and select Application Restarts.
8) Click Add, Close, and Ok.
9) You will notice that the Perfmon will start logging data into this log file.

Once the issue happens you would observe the counter increasing.

Most common reasons for App domain recycling:

1) Antivirus Scans: You can get rid by excluding your framework, Application directories from scanning

More info on the directories to be excluded from antivirus scan can be found in the below blog

http://blogs.msdn.com/b/amol/archive/2015/12/10/asp-net-anti-virus-exclusion-list.aspx
2) Group policies changing or modifying framework folders or application folders either the permissions or the folder contents
3) Machine.Config, Web.Config or Global.asax are modified
4) The bin directory or its contents is modified
5) Exceeding the number of recompilation.

6) The physical path of the virtual directory is modified
7) The CAS policy is modified.
8) The web service is restarted.
9) Application Sub-Directories are deleted.

10) Writing files within web application or website directory.

These are some of the basic checks we can do to identify if the issue we are facing is because of Application Domain Recycling.

In the next series of blogs I am going discuss how to collect much more advanced data and how to look into these data to identify the root cause.

Hope this helps

The issue may surface and we may receive different errors. Below are some of such errors

1.

2.

In one of the cases it complains about permissions and in the other it complains about the start url.

in both the cases I observed that the configuration file(applicationhost.config) was being read from a network share.

On further investigation I found out that we had the My Documents folder point to a shared drive or network share.

in 2 out of 3 instances I have seen this issue appear when the network share being pointed to is a MAC share.

Also when we have this in a network share I also observered that in one of the machine the applicationhost.config was going into corrupt state every time as there were “NUL NUL” being inserted into the applicationhost.config file.

I suspect the issue either because how the data is written over network for different operating systems. And in other scenarios it was because of the network share going down.

So the million dollar question is “How to change the custom user home directory for IISExpress?”.

If you research over internet on how to change the custom user home directory for IISExpress you will see in various references that you can change it by following the below steps.

But what if you don’t find the key being set at all , which was the same case which happened with me.

So after lot of research I identified the below registries where the location of My documents will be changed and we need to revert these back.

We need to change the default path of the My Documents directory from network share to the local or you can try pointing to a different network share if you don’t want to choose the local option. There are options in the UI to achieve this but it didn’t work for me.

So fail proof method is to modify the below registries which will be pointing to the network share. Take a backup of the registry keys before modifying it

These will have the value of my docs in network share. Change this to local

For eg:

Old value: \\chiranth8r2\c$\Users\chiranth\Documents

New value: C:\Users\chiranth\Documents

Hope this helps

References:

http://www.iis.net/learn/extensions/introduction-to-iis-express/iis-80-express-readme

Windows 20081. Start the Tracing. From a command prompt run the following command: netsh winhttp set tracing trace-file-prefix="C:\TEMP\WinHttpLog" level=verbose format=hex state=enabled max-trace-file-size=1048576000 2. Recycle the IIS Application Pool. 3. Reproduce the issue. 4. Stop the Tracing. From a command prompt run the following command: netsh winhttp set tracing state=disabled 5. Review the trace with Notepad or any Text editor. NOTE: The Identity of the IIS application pool will require write access to the  log location  c:\Temp in this example: This type of tracing is process bitness specific, so if you are looking at a 32 bit process running from 64 bit OS, you need to use: c:\windows\syswow64\cmd.exe, rather than using the regular 64 bit cmd.exe (start a run a cmd.exe) Windows 2008 R2 Method 1 This method will output the Winhttp API calls , but not raw data for network communication. From a command prompt run the following command: 1. Start the tracing netsh winhttp set tracing trace-file-prefix="C:\Temp\Test3" level=verbose format=hex netsh winhttp set tracing output=file max-trace-file-size=512000 state=enabled 2. Recycle the IIS Application Pool. 3. Reproduce the issue. 4. Stop the Tracing. From a command prompt run the following command: netsh winhttp set tracing state=disabled 5. Review the trace with Notepad or any Text editor. NOTE: The Identity of the IIS application pool will require write access to the log location c:\Temp in this example: This type of tracing is process bitness specific, so if you are looking at a 32 bit process running from 64 bit OS, you need to use: c:\windows\syswow64\cmd.exe, rather than using the regular 64 bit cmd.exe (start a run a cmd.exe) Method 2 To get the raw data communication at network layer and the Winhttp  Api calls. 1. Start the tracing: From a command prompt run the following command: netsh trace start scenario=InternetClient capture=yes report=yes Note the etl file location for example:   Trace File:         C:\Users\<your user name>\AppData\Local\Temp\NetTraces\NetTrace.etl 2.Recycle the IIS Application Pool. 3. Reproduce the issue. 4. Stop the tracing: From a command prompt run the following command: netsh trace stop 5. Read the Trace by opening it in Netmon 3.4. Method 3 The ETW format for winhttp API is available on windows 2008 R2 and win7 via the Event Viewer 1.  Open event viewer. Go to “View” menu --> make sure “Show Analytic and debug logs” is checked. 2. Open “Applications and Services logs” -- > Open “Microsoft” -- > Open “Windows –> Winhttp –> Diagnostic. 3. Highlight “Diagnostic” under Winhttp tree and right click mouse, then click “enable log”. 4.  Reproduce the issue then you can review the logs.

15:57:26.662 ::Completing WinHttpReadData() with error ERROR_WINHTTP_TIMEOUT (12002); Request

Handle = 1C8457F0, Context = 1C5F0E70

15:57:26.662 ::sys-recver processing WebReceiveHttpResponseEntity completion

(error-cdoe = ? (0x5b4), #bytes = 0, overlapped = 1B775260)

15:57:26.662 ::sys-recver failed to read data; error = ? (1460)

15:57:26.662 ::ERROR_WINHTTP_FROM_WIN32 mapped (?) 1460 to (ERROR_WINHTTP_TIMEOUT) 12002

15:57:26.662 ::sys-req completing a read-data call(error = ERROR_WINHTTP_TIMEOUT(12002),cbRead =0)

15:57:26.662 ::usr-req 1C9B4F80 received OnReadData() callback; error = ERROR_WINHTTP_TIMEOUT 

(12002), ulNumberOfBytesTransferred = 0, dwptrContext = 1C380160

Hope this one helps

References:

http://blogs.iis.net/richma/archive/2012/08/24/winhttp-tracing-options-for-troubleshooting-with-application-request-routing.aspx

http://technet.microsoft.com/en-us/library/ee683908(WS.10).aspx

http://technet.microsoft.com/en-us/library/ee683951(WS.10).aspx

http://technet.microsoft.com/en-us/library/ee683923(WS.10).aspx

http://technet.microsoft.com/en-us/library/ee683943(WS.10).aspx

http://technet.microsoft.com/en-us/library/dd443533.aspx

ARR is an out of band module available for IIS 7.x and onwards.

ARR can be used for below purposes.

1) As a load balancer when you want to load balance requests between multiple servers.

2) As a reverse proxy where your web servers are hidden from the internet world.

3) It can also be used for platform provisioning and application provisioning(by this I mean keeping configuration and features installed on top of IIS in sync between multiple servers) with the help of web deploy(which is famously know as web farm framework WFF). WFF is not tested completely from IIS 8 and onwards.

In this series we will discuss 1) and 2). First we will concentrate on how we can use ARR as a load balancer.

ARR as a load balancer

Before going ahead and using ARR as a load balancer. Let’s get familiarized with the options present in the ARR server farms UI which will help us during our discussion.

Below are the important options which we need to familiarize ourselves with when we want to use server farms or ARR as a load balancer.

1) Caching

2) Health Test

3) Load Balance

4) Monitoring and Management

5) Proxy

6) Routing rules

7) Server Affinity

Caching:

This is similar to your IIS caching where you have the above mentioned options. This option will help you in caching content received from the backend server when the content has not been changed.

http://technet.microsoft.com/en-us/library/ee731895(WS.10).aspx

Load Balance:

Over here you can specify the algorithm you want to use to perform load balancing and based on what criteria the incoming requests should be split between your web servers. The options present are all standard algorithms and you can find info about these algorithms over internet about how they work. Below are the available algorithm options about load balancing and a brief introduction about them.

• Weighted round robin – Distributes traffic based on the number of incoming requests and their normalized weight. Each server can receive the same distribution of requests or a custom distribution.
• Weighted total traffic – Distributes traffic based on the size of the requests and responses in bytes. Requests are routed so that the amount of data is load balanced. In an even distribution, the server with the least amount of data will receive the next request.
• Least current request – Distributes traffic based on the current number of HTTP requests between ARR and each of the application servers. Requests are routed to the server with the least number of current HTTP requests.
• Least response time – Distributes traffic based on the fastest response time from the servers, which enables the server to respond most quickly.
• Server variable hash – Distributes traffic based on a hashed value of a server variable.
• Query string hash – Distributes traffic based on the hashed value of the query string value. When more than one query string name is specified, the concatenated string of the corresponding query string value is used for the hash.

· Request hash – Distributes traffic based on the hashed value of the configured server variable or URL. For example, if the server variable is QUERY_STRING, the hashed value is based on the names in the request query string.

Proxy:

I forgot to include explanation for Reverse rewrite host in response headers in the image while creating it.

Reverse rewrite host in response headers: This option might not be of much value over here but its a very important setting while having reverse proxy. Imagine the internet exposed url is http://contoso.com and you have the backend servers contoso1 and contoso2. consider an example where you have a redirect status set and in the response location tag is set to http://contoso1/redirectedpage.aspx in web server in respect to the request forwarded from ARR server and this has to be notified to the end client. the client doesn't know who contoso1 is. so before sending the complete response to the client, the ARR server rewrites the host name in the location tag as http://contoso.com/redirectedpage.aspx

Routing Rules:

This is pretty much self-explanatory and you will select “Use URL Rewrite to inspect incoming requests” option to use URL rewrite rules to evaluate requests that come through your server.

Another important option over here is Enable SSL offloading- if you select this option even though the requests from client to ARR server will be over https but while forwarding the requests to backend servers this will happen over http.

Requests with the following extensions are not forwarded: This is a very rarely used but a very good feature which helps you in improving performance. if you specify *.jpg, then it means that any request for .jpg image file will not be sent to the backend server. you can keep the static content like images on ARR server itself. This will add a condition in your rule. you can add this condition manually in your inbound rule as well. the condition will look like below.

Requests with the following patterns are not forwarded: This one is similar to the one mentioned above. when you have contents like templates, images or static content which wont change often then you can keep the copy of it in ARR server itself and then choose not to forward these request and process these request locally on ARR server. I can choose to ignore all the requests containing images in the path from being forwarded. you can manually add such a condition in your inbound rule as well and it will look like below.

Server Affinity:

Client affinity: People might be familiar with the name “sticky sessions” available in most of the hardware load balancers. This option provides the same feature in IIS. When you want the requests to be load balanced to the same server where the first request from the client was routed to for the whole session, you can use the client affinity feature over here.

This feature is advantageous where in your session state is ‘in-proc’ and not available outside the worker process.

Host Name affinity: imagine you have multiple hostnames for a site. based on the hostname used you can decide the affinity. Also if you have 4 servers for load balancing you can control the number of servers the requests can be routed if it comes with a specific hostname.

Monitoring and Management:

This will be your load balancer dashboard where you can monitor the health status and the request statistics, also how the load balancing is happening.

Creating sample to demonstrate load balancing by creating server farm in ARR

Scenario: Let’s consider the scenario where in we have two backend web servers’ contoso1.chiranth.com and contoso2.chiranth.com and one ARR server where we want to load balance the requests for the site “contoso.com” between the two servers.

To achieve the above requirement we have two follow two simple steps.

1) Creating a server farm which will be a container for these two servers

2) Creating URL rewrite rules to monitor the incoming requests and route it accordingly

Creating a server farm:

Creating a server farm is a two-step simple procedure as below.

1) Right click on server farms-> Create new server farm

2) Enter a friendly farm name

3) Add the server which you want to include for load balancing. Over here in our scenario we have 2 servers contoso1 and contoso2

Creating URL rewrite rules for your farm:

After creating the farm 50% of the task is done. Now we will have to configure URL rewrite rules to monitor the requests and route it accordingly. When you create a server farm ARR will prompt you asking if it should create a URL rewrite rule for you. Let’s click on No for that. I will take you through the manual steps for creating the rule.

1) Make sure under routing rules at the server farm level you have the “USE URL REWRITE to inspect incoming requests” option checked.

2) When you are using ARR for either Load balancing or as reverse proxy the URL rewrite rules should be created at the server level. Go to URL Rewrite at the server level.

3) Add rule-> Inbound rules ->Blank rule.

4) I have already explained about the sections in inbound rules and regular expressions in my previous blog. So over here pattern would be ‘.*’ i.e, anything and a condition stating that this rule should be executed only when the hostname or HTTP_HOST is contoso.com, action would be “Route to server farm” and under action properties select the appropriate server farm and specify the REQUEST_URI which we have stored as {R:0} from the pattern section in the URL.

The rule will look like below.

5) Click on Apply and you are good to go for testing.

Hope this helps. In next part of my upcoming blog I will discuss how we can use ARR as reverse proxy and how we can go through the troubleshooting if we are facing any issues and the tools that we can make use of.

References:

http://technet.microsoft.com/en-us/library/ee731895(WS.10).aspx

http://technet.microsoft.com/en-us/library/dd443524.aspx

http://technet.microsoft.com/en-us/library/dd443526.aspx

http://technet.microsoft.com/en-us/library/dd443532.aspx

http://technet.microsoft.com/en-us/library/dd443543.aspx