Tips and tricks: ASP.NET, IIS and .NET development...

.NET Core & ASP.NET Core 1.0 is being released at Red Hat DevNation.

Ahmet Mithat Bostanci — Mon, 27 Jun 2016 16:15:09 GMT

Today is another milestone on developers' world: .NET Core & ASP.NET Core 1.0 is being released at Red Hat DevNation. If you want to eyewitness that moment you can watch it here. If you want to get started with .NET Core & ASP.NET Core 1.0 (if you haven't done yet), you can start with the following resources: .NET Home Page https://dot.net/ .NET Documentation https://docs.microsoft.com/en-us/dotnet/ .NET Community https://www.microsoft.com/net/community .NET Blog https://blogs.msdn.microsoft.com/dotnet/ Channel 9 .NET Channel https://channel9.msdn.com/Blogs/dotnet .NET Team Twitter Account https://twitter.com/DotNet .NET Facebook Account https://www.facebook.com/Dotnet -- AMB

Enlightening a mystery with Failed Request Tracing: does IIS not respect the minFileSizeForComp setting for static compression?

Ahmet Mithat Bostanci — Mon, 23 May 2016 16:47:47 GMT

I have been working with one of my customers to fix some static compression issues on IIS 8.5. During our work I have been asked that if IIS is not respecting the minFileSizeForComp settings and if it is compressing the static files that are smaller than the limit configured in the IIS settings. Before proceeding with the situation and the solution, I would like to explain what that minFileSizeForComp setting is used for first:

minFileSizeForComp Specifies the minimum number of ~~kilobytes~~ bytes a file must contain in order to use on-demand compression. Ref.: https://www.iis.net/configreference/system.webserver/httpcompression

As explained, if the file size is smaller than the limit configured, then the static file compression will not happen. Note that the default limit is 2700 bytes for IIS 7.5 and the never versions. This means that, a static file (for example a CSS file) will not be compressed if it is smaller than 2700 bytes with the default configuration. Another note is that this setting will be ignored for the dynamic compression as a by-design behavior. This configuration is quite useful because compressing the small files may produce larger files so it may cause more bandwidth usage. For example, if you compress a CSS file which is 7 bytes, then you may have a compressed one which is 130 bytes as a result, which is larger than the uncompressed version of the same file. Obviously this will waste the CPU time on the server and will consume more bandwidth. PROBLEM As I explained in the beginning of this article, my customer asked if this setting is ignored for static files and if the small static files are compressed as an unexpected behavior. When I searched the internet for the minFileSizeForComp setting, I have found that several discussions have been made around the same topic and the same behavior is reported with no solution or workaround. For example the following forum thread shows the same discussion:

IIS7 Compression not honoring minfilesizeforcomp setting https://forums.iis.net/t/1192378.aspx?IIS7+Compression+not+honoring+minfilesizeforcomp+setting

REPRO To understand if IIS really ignores this setting I decided to set up a repro with the following configuration on my test machine:

I set up a web site on my IIS 8.5 machine.
I created the web site with two CSS files referenced:
- /Content/Site.css: a css file which is 7 bytes
- /Content/Bootstrap.css: a css which is 117 KB (120.502 bytes)
I enabled both Dynamic Compression and Static Compression for my site since I would like to have both dynamic and static files to be compressed:

Below is the compression settings for my web site: Please note the followings:

minFileSizeForComp is 2700 bytes, which is the default value.
I changed the staticCompressionIgnoreHitFrequency to true so static compression will happen for every requests (please see https://www.iis.net/configreference/system.webserver/httpcompression for the details).

When I request my site with this configuration I end up with both CSS files are compressed. Please see the Fiddler log below: As seen in the screenshot above, the 7 bytes /Content/Site.css file is served as 130 bytes after compressed with gzip, although I have minFileSizeForComp is set 2700 bytes. Expectation was to serve it uncompressed. TROUBLESHOOTING THIS BEHAVIOR IIS comes with a great built-in troubleshooting tool called Failed Request Tracing (a.k.a. FREB). FREB can provide a lot of useful information about what happens in the IIS pipeline, either within the Integrated or Classic pipeline. I set up a FREB rule in my web site for /Content/* path for troubleshooting: Since I am interested in only successful responses from IIS, I set up the rule for HTTP 200 status code. This makes sure to generate FREB logs for every requests made for a file under /Content/ path for HTTP 200 response: I am only interested in seeing compression related data in the IIS pipeline so I select only the WWW Server Providers and choose the Compression property. This will log only the compression related processing from IIS pipeline and it creates the minimum data for analyzing: After reproducing the same behavior FREB logs will be created in the FREB logging directory. REVIEWING THE DATA When I open the FREB log for Site.css request, I confirm that IIS is actually respecting the minFileSizeForComp setting and the static compression does not happen for that request. The reason is shown as FILE_TOO_SMALL: This proves that IIS respects the minFileSizeForComp setting. But why is the small CSS file compressed then? As seen in the same FREB results above, the CSS file is compressed because of Dynamic Compression and it also shows that the original size is 7 bytes and the result is 130 bytes! Well, obviously this is an undesirable behavior. So the small file is not compressed because of static compression, instead it is compressed because of dynamic compression settings. Note that as I expressed at the beginning of this article minFileSizeForComp is ignored for dynamic compression which is a by-design behavior. That is why dynamic compression did not check if the file is smaller than the minFileSizeForComp value. There should be some questions arising here:

If a file is compressed by static compression, then is the same file compressed by dynamic compression again?
How to avoid dynamic compression for this particular file?

The answer for the first question is NO, if a file is compressed by static compression then it will not be compressed by dynamic compression. Remember that I had another CSS file which was 117 KB (bootstrap.css) and it was larger than the 2700 bytes limit so it should have been compressed by static compression. If I check the FREB log for that file I can confirm that the static compression happened but dynamic compression did not: As seen in the FREB log results above, the static compression is successful so IIS does not try to compress the same file with the dynamic compression and the ALREADY_CONTENT_ENCODING information is logged. For the answer to the second question, let's please welcome the solution below. SOLUTION If you do not need dynamic compression then you can just disable it. This will avoid dynamic compression to happen at all. Please note that the dynamic compression should be used carefully since it may cause higher CPU usage on the server since it will not create the compressed version on the temporary folder on the disk (by default the folder is C:\inetpub\temp\IIS Temporary Compressed Files), meaning that the dynamic compression will happen for each request for the same file by default. This causes the CPU work for every request for the same file. However, if you need Dynamic Compression, then you can tell IIS to not compress a certain file type with dynamic compression. You can do that at the site level with IIS Configuration Editor easily. To do so please follow the steps below: 1) Select your web site and choose IIS Configuration Editor under Management section and then choose system.webServer/httpCompression. Then click ... to open the Dynamic Types settings: 2) Then add the “text/css” as mimeType and set enabled=False, as seen in the screenshot below: 3) Do not forget to click Apply after closing the window above. It will make the necessary changes in the web.config file for your web site. Now you can test it again and this time you should be able to see the small static file is not compressed. You can see a sample Fiddler output in the screenshot below. Note that the /Content/Bootstrap.css file is compressed because it is a larger file than the minFileSizeForComp value while the /Content/Site.css is served as uncompressed since the file size is 7 bytes and it is smaller than the minFileSizeForComp value: SOME WORDS ABOUT BEST PRACTICES I was discussing this issue with my friend and my colleauge Paul Cociuba who works as an Escalation Engineer in Microsoft IIS Support Team. We agreed that it is a best practice not to serve several small CSS or JS files separately. Instead, using the Bundling and Minification techniques will improve the request times. The following article is a good read for those techniques:

Bundling and Minification https://www.asp.net/mvc/overview/performance/bundling-and-minification

REFERENCES HTTP Compression https://www.iis.net/configreference/system.webserver/httpcompression Troubleshooting Failed Requests Using Tracing in IIS 7 https://www.iis.net/learn/troubleshoot/using-failed-request-tracing/troubleshooting-failed-requests-using-tracing-in-iis Editing Collections with Configuration Editor https://www.iis.net/learn/manage/managing-your-configuration-settings/editing-collections-with-configuration-editor APPLIES TO IIS 7.x and newer versions. -- AMB

Some IIS administration cmdlets run slowly in PowerShell 4.0

Ahmet Mithat Bostanci — Mon, 02 May 2016 09:56:01 GMT

I wrote a KB article about PowerShell IIS administration cmdlets when you run in Win 2008 or Win 2008 R2. If you are running New-WebApplication or ConvertTo-WebApplication cmdlets in Win 2008 or Win 2008 R2 then you may face with some performance issues. For the details and the workaround please see the following article: Some IIS administration cmdlets run slowly in PowerShell 4.0 https://support.microsoft.com/en-us/kb/3144309 Applies To Powershell 4.0 on Win 2008 or Win 2008 R2 -- AMB

AppDomain is still unloaded when web.config file is changed even File Change Notifications (FCN) are disabled

Ahmet Mithat Bostanci — Tue, 09 Feb 2016 10:36:00 GMT

Let's assume that you are running an ASP.NET application and you disable the File Change Notifications in web.config file (for ASP.NET 4.5 or later) or in the registry as explained in https://support.microsoft.com/kb/911272.

However the appDomains are still unloaded and reloaded when you change a web.config file.

--
AMB

IIS 8.5 - How to log client port number

Ahmet Mithat Bostanci — Thu, 12 Nov 2015 07:05:00 GMT

When an HTTP request is made from a client machine to a web server, a TCP connection is established between the client and the server machines. That TCP connection is built on the port numbers on both server and client sides. A web server, for example IIS, listens on port 80 for HTTP and on port 443 for SSL requests. Those are default ports of course and can be configured.

On the client side, operating system uses a random port number from an available port list, between 1024 – 65535.

You can read the following article for the TCP / IP details:

How TCP/IP Works
https://technet.microsoft.com/en-us/library/cc786128(v=ws.10).aspx

Getting the client port information

IIS writes the server side port configured for the web site request on the IIS logs. However it does not write the client port and you may need to get that client port information for some reason.

Luckily that information is available on Server Variables. From Microsoft’s explanation, the Server Variables means the following:

IIS server variables provide information about the server, the connection with the client, and the current request on the connection. Ref.: https://msdn.microsoft.com/en-us/library/ms524602(v=vs.90).aspx

Enhanced Logging feature of IIS 8.5

In IIS 8.5, the administrator has the option of logging additional custom fields from request or response headers, or from server variables. Ref.: https://www.iis.net/learn/get-started/whats-new-in-iis-85/enhanced-logging-for-iis85

That means that we can log any information available in request or response headers, or in server variables.

Logging the client port number

Client Port information is stored as REMOTE_PORT in server variables. So you can follow the instructions below:

Open IIS 8.5 manager and choose the site you would like to configure logging with client port. You can do that on server level as well so all of the sites can log the client port.
Open Logging and choose Select Fields.
On Custom Fields section click Add Field.
Give a name for Field Name, note that you cannot use space.
Choose Server Variables from Source Type.
Type REMOTE_PORT in Source text box.
Click OK.

You should be seeing something like below:

From that point on, you should be seeing the client TCP port logged in the IIS logs. A sample output is below. For readability reasons I disabled most of the logging fields. I also disabled the HTTP Keep-Alive for my test web site so each request I made from the same client just created a new TCP/IP connection:

Applies To:

IIS 8.5 and onwards.

For IIS 7.0, 7.5 and 8.0 you can use Advanced Logging module (https://www.iis.net/downloads/microsoft/advanced-logging).

--
AMB

Collecting a memory dump with ProcDump when ASP.NET ISAPI is reported unhealthy or deadlock detected in an ASP.NET application

Ahmet Mithat Bostanci — Fri, 06 Nov 2015 06:47:00 GMT

PROBLEM

You are running an ASP.NET application in IIS.
You noticed that the worker process shuts down and restarts.
At the same time you see one or both of the following events in the event logs:

Event ID 2262: ISAPI 'C:\Windows\Microsoft.NET\Framework\\aspnet_isapi.dll' reported itself as unhealthy for the following reason: 'Deadlock detected'. Event ID 5013: A process serving application pool 'app_pool name' exceeded time limits during shut down. The process id was ''. ACTION PLAN Memory dump analysis is required to find out the root cause most of the times. You can follow the steps below to collect memory dumps automatically: 1) Download ProcDump from https://technet.microsoft.com/en-us/sysinternals/dd996900.aspx on your server and extract it. 2) Copy the ProcDump.exe, ProcDump64.exe and Eula.txt files to both C:\Windows\System32 and C:\Windows\SysWOW64 folders. 3) Create C:\Dumps folder where the dump files are written. You can use another local disk or folder but you should change it in the step 5. 4) Open IIS manager, find your application pool which is reported unhealthy and open its advanced settings. 5) Find Process Orphaning settings and configure it as below, as seen in the screenshot:

Enabled: True
Executable: c:\windows\system32\procdump.exe
Executable Parameters (if your application pool is 32bit): -accepteula -ma %1% c:\Dumps
Executable Parameters (if your application pool is 64bit): -accepteula -64 -ma %1% c:\Dumps

6) Click OK to close the dialog box. When the same issue happens the dump(s) should be generated in the c:\Dumps (or where you configured) folder. NOTES 1) Do not forget to disable Process Orphaning when the dumps are collected. 2) A memory dump is a snapshot of a process memory so the size of a dump file will be approximately the same as the size of the process in the memory. For example, if a process consumes 2 GB of memory when taking the dump, then the dump file will take approximately 2 GB in the disk. Please set a location where you have enough free disk space. 3) Avoid writing dumps to a network share or a slow disk (is there any slow disk left?) because when a dump is being written, all threads of the process will be suspended. Any requests arrive while the dump is written will be queued until the dump file is completed. TIP FOR ANALYZING THE MEMORY DUMP The issue described in the PROBLEM section may be mostly related with some requests waiting to acquire a lock which is owned by another thread. If a lock is not relased for enough time, IIS assumes that the process is deadlocked and WAS terminates the worker process, reports it as unhealthy and restarts a new one. You can open the dump in WinDBG and use SOS extension's SyncBlk command to analyze the locks. Try to find out which thread owns the lock and why it does not release that (e.g.: thread may be doing a long running task in a lock). You can also use Debug Diagnostic's Auto Analyse feature and review the report generated. MORE INFORMATION Process Orphaning feature of IIS allows you to avoid terminating the “unhealthy” or “deadlocked” process so the “orphaned” process could live in the memory, without accepting any requests, for troubleshooting purposes (for example attaching a debugger to it or taking the dumps of it). REFERENCES Managing, Tuning, and Configuring Application Pools in IIS 7.0 https://technet.microsoft.com/en-us/library/cc745955.aspx APPLIES TO All supported IIS versions All ASP.NET versions

IIS application pool may be terminated while collecting dump

Ahmet Mithat Bostanci — Thu, 26 Mar 2015 05:04:00 GMT

Consider the following scenario:

You are running a web application on IIS and you need to collect a memory dump of the application pool process (w3wp.exe).
W3wp.exe in subject consumes huge memory, for example, around 40 GB, and it takes some time to write the dump file.
When you try to collect a memory dump of the worker process, it is terminated and the memory dump is corrupted.

This may be happening because of the “health monitoring” configuration of the application pool. For example the following screenshot shows the default settings for health monitoring: The following happens with the default configuration seen in the screenshot above:

WAS will monitor application pool by pinging it every 30 seconds (Ping Period configuration).
If the process does not respond in 90 seconds (Ping Maximum Response Time) then WAS will terminate the process.

Remember that the process is suspended when a dump is written to the disk, which means that it will not respond to WAS’ ping request and if that exceeds the ping maximum response times then WAS will terminate the process. If you see the similar behavior and you need to collect a useful dump then you may need to increase the ping timeout or set “ping enabled” to false until you collect the memory dump. Please note that changing these settings will cause application pool recycle. -- AMB

Free e-book - Windows 8.1 Universal application development

Ahmet Mithat Bostanci — Tue, 24 Mar 2015 05:11:02 GMT

A colleague of mine, Matteo Pagani, has published a free e-book about Windows 8.1 Universal Application Development, it is a second part of the same topic. Below is what he says about it:

More Windows 8.1 Succinctly

Move beyond the development stage and present your apps to the world. In this final volume, author Matteo Pagani guides you through the process of readying your Windows 8.1 and Windows Phone 8.1 apps for the Windows Store. You’ll learn how to integrate your app into Microsoft’s infrastructure to maximize its functionality and appeal.

Table of Contents

Publishing an Application on the Store
Interacting with the Network
Interacting with the Real World
Contracts and Extensions
Creating Multimedia Applications
Tiles and Notifications
Supporting Background Operations

You can get a copy of the e-book from the following site:

https://www.syncfusion.com/resources/techportal/ebooks/morewindows8.1

--
AMB

Everyone can build an app – introducing Windows Phone App Studio beta

Ahmet Mithat Bostanci — Tue, 06 Aug 2013 14:28:41 GMT

"Windows Phone App Studio is about giving everyone the ability to create an app, regardless of experience. It also can radically accelerate workflow for all developers."

Read the full story at Windows Phone Developer Blog.

--
AMB

Free e-book collection

Ahmet Mithat Bostanci — Fri, 21 Jun 2013 00:59:35 GMT

Hello,

one of my colleauges shared the following blog post which provides a list for free e-books for different technologies including web and mobile development. Definetly it is worth to check:

Huge collection of Free Microsoft eBooks for you, including: Office, Office 365, SharePoint, SQL Server, System Center, Visual Studio, Web Development, Windows, Windows Azure, and Windows Server
https://blogs.msdn.com/b/mssmallbiz/archive/2013/06/18/huge-collection-of-free-microsoft-ebooks-for-you-including-office-office-365-sharepoint-sql-server-system-center-visual-studio-web-development-windows-windows-azure-and-windows-server.aspx

Enjoy :)

--
AMB