Inside Microsoft.comManagement and Delegation of ASP.NET
Jeff Toews
The microsoft.com Web infrastructure runs almost entirely on the .NET Framework 2.0. One of the microsoft.com operations team’s key Web administration challenges is the proper configuration of ASP.NET, and we’ve learned a lot while trying to tweak our settings to perfection. Getting
configuration settings right requires a good working knowledge of the various configuration sections in web.config and machine.config files and an understanding of what those settings mean. A good way for you to get a handle on the settings and their significance is to look at examples. In this column I’ll present some configuration tips derived from configuring the servers running microsoft.com so that you have the benefit of learning from our experience.
1. Set the Compilation Switch Appropriately
When deploying ASP.NET-based applications in a production environment, it is critical to make sure that nobody accidentally (or deliberately) leaves the compilation debug attribute set to true in any application web.config files, as it is here:
<compilation debug="true" />
In busy environments with many Web applications to manage, you will want to use the ASP.NET configuration control mechanisms to block this from occurring. (I’ll give more details on this in a bit.)
It is also important to ensure that the page-specific debugging attribute is not set to true in individual .aspx as it looks like here:
<%@ Page debug="true" %>
Again, in busy environments with high volumes of publishing, it’s often not realistic to expect you’ll be able to ensure that this setting is removed from all .aspx pages prior to their being published. You’ll need a global means to prevent this from occurring.
Compiling your Web application with this setting will result in debug binaries rather than retail binaries. Plus your code will not be optimized, resulting in slower performance. In addition, your ASP.NET requests will not time out because the debug settings prevent timeouts. Using a debug version in production environment is like opening a door and inviting hackers!
Fortunately, the Microsoft® .NET Framework 2.0 has a new deployment setting for machine.config that tells ASP.NET to disable the following: debug capabilities, trace output, and display of ASP.NET error messages (both on localhost and remotely) regardless of the instructions in your web.config file or specific page attributes. It looks like this:
<configuration>
<system.web>
<deployment retail="true"/>
</system.web>
</configuration>
Note that these last two benefits (disabling trace output and the disabling of detailed ASP.NET error messages remotely) are security best practices you really should adopt. If you don’t, you’re exposing the internal workings of your application for all the world to exploit.
While I’m on the subject, here’s another important fact: locking the <system.web><compilation> debug attribute in the root web.config file inside <location allowOverride="false"> or using the lockItems attribute will prevent web.config files lower in the application configuration hierarchy from turning on debug settings. But this will not prevent individual .aspx pages from enabling debug mode in their page attributes. The deployment retail switch is the only way to fully disable ASP.NET debug capabilities at all levels.
Setting the deployment retail switch to true is probably a best practice that any company with formal production servers should follow to ensure that an application always runs with the best possible performance and no security information leaks. As I said, this switch is brand new in ASP.NET 2.0; it was the direct result of feedback to the ASP.NET team.
Conversely, for internal-facing preproduction environments where devel-opers need to debug their Web applications, do not use the deployment retail setting. Simply set <compilation debug="false"> in the preproduction root web.config file and allow this value to be overridden by individual applications’ web.config files or .aspx page attributes.
2. Use Medium Trust in ASP.NET 2.0
If, as was the case for many sites on microsoft.com, you were still using a Full or High trust level even after migrating your site or applications to ASP.NET 2.0, take another look at what is now possible under Medium trust. For example, restricted WebPermission constrains an application’s communications to only one address or range of addresses that you define in the <trust> element. This lets you control and maintain a list of approved external sites and address ranges that can be called remotely. This is a big security boon.
FileIOPermission is also restricted. This means an application’s code can only access files in its virtual directory hierarchy. By default under Medium trust, each application is granted Read, Write, Append, and PathDiscovery permissions for its virtual directory hierarchy only. This puts an end to random file I/O access—quite important in a shared Web environment in which many applications are hosted.
Another advantage is that unmanaged code privileges are removed. What this means is that you can prevent use of legacy components, the easiest way we’ve yet found to disable use of the Aspcompat page attribute. (Setting Aspcompat to true can cause a page’s performance to degrade.)
Medium trust under ASP.NET 2.0 is quite flexible in terms of the ability it gives the administrator to create custom exceptions to each of the previous default restrictions. This flexibility was lacking in ASP.NET 1.1. Another reason running at Medium trust with ASP.NET 2.0 is easier than with ASP.NET 1.1 is that you have access to Microsoft SQL Server™ databases.
Thus, if you host multiple applications on the same server, you can use code access security and the Medium trust level to provide application isolation. By setting and locking the trust level (using <location allowOverride="false"> tags) in the root web.config file, you can establish security policies for all Web applications on the server. By setting allowOverride="false", as you see in Figure 1, an individual developer is unable to override the Medium trust policy setting in his application’s web.config file.
Figure 1 Trust Settings
<configuration>
<location allowOverride="false">
<system.web>
<securityPolicy>
<trustLevel name="Full" policyFile="internal" />
<trustLevel name="High" policyFile="web_hightrust.config" />
<trustLevel name="Medium" policyFile="web_mediumtrust.config" />
<trustLevel name="Low" policyFile="web_lowtrust.config" />
<trustLevel name="Minimal" policyFile="web_minimaltrust.config" />
</securityPolicy>
<trust level="Medium" originUrl="" />
</system.web>
</location>
</configuration>
More information regarding the use of Medium trust in ASP.NET 2.0 can be found in the following article: "How To: Use Medium Trust in ASP.NET 2.0".
3. Restrict Download of Specified File Types
There are file types on your server that you undoubtedly do not want to fall into the wrong hands. Fortunately, by default, ASP.NET is configured to intercept and stop requests for several different file types that are used in ASP.NET applications. They include .config files and .cs files that store the source code of the application. ASP.NET ensures the privacy of these files by associating both file types with System.Web.HttpForbiddenHandler. This handler, when invoked, returns an error to the user who requests the file. In fact, you can use this method to restrict any file type.
For example, on the microsoft.com site, the file type .asmx is disallowed by adding the following entry to the <system.web><httpHandlers> section of the root web.config file:
<add path="*.asmx" verb="*" type=
"System.Web.HttpForbiddenHandler" />
As you can see, you can use <add> sub tags in the <httpHandlers> element to specify additional file types that you want blocked. Set the verb attribute equal to "*". When you do this, you specify that all types of HTTP requests are blocked for that file type. Define the path attribute as a wildcard character that matches the types of files you want to block. For example, you may specify "*.mdb". Finally, set the type attribute to "System.Web.HttpForbiddenHandler".
4. Be Careful When Adding Assembly References
Each Web server that runs the .NET Framework has a machine-wide code cache called the Global Assembly Cache (GAC). The GAC stores assemblies specifically designated to be shared by multiple applications on the computer. It makes sense to add assemblies to the GAC if several different applications need to reference them, even if the majority of the sites on the Web server don’t access them. This does not result in a significant performance/resource penalty but gives the benefit of maintaining centralized version control rather than having shared assemblies scattered across the server in individual application’s /bin folders.
The criteria for when an assembly reference should be added to the root web.config, however, needs to be much stricter than the criteria dictating when an assembly is placed in the GAC. Having individual applications add assembly references in their applications’ web.config file for components that aren’t truly global yields much better performance than declaring these assembly references in the root web.config file. This reduces page load time significantly for all applications on a Web server that don’t use these assemblies as the compiler doesn’t need to spend time loading unnecessary assemblies. The ASP.NET compiler won’t load an assembly for an application simply because the assembly resides in the GAC; it only loads it if an assembly reference exists in its appdomain or in a higher app scope. As a result, at microsoft.com we allow application developers to override the <configuration><system.web><compilation><assemblies> element in their applications’ web.config file for all Microsoft.com environments.
Specifically, in the microsoft.com production and staging environments’ root web.config files, all attributes of the <system.web><compilation> node are locked (including debug, explicit, defaultLanguage) as well as all its elements (buildProviders, expressionBuilders, and such), except for the <assemblies> element:
<compilation debug="false"
explicit="true" defaultLanguage="vb"
numRecompilesBeforeAppRestart="500"
lockAttributes="*" lockAllElementsExcept=
"assemblies" >
In the internal-facing preproduction environment, the <system.web><compilation> section is locked in the root web.config file, but we allow application publishers to specifically override the <assemblies> element and also to override debug="false" (to enable troubleshooting/debugging):
<compilation debug="false" explicit="true"
defaultLanguage="vb"
numRecompilesBeforeAppRestart="500"
lockAllAttributesExcept="debug"
lockAllElementsExcept="assemblies" >
Note the use here of lock attributes, which are new in ASP.NET 2.0. Again, detailed information on these attributes and examples of their use can be found at "General Attributes Inherited by Section Elements".
5. Remove Manually Set MaxConnection Values
Nearly all microsoft.com Web sites have ASP.NET applications that make calls to remote Web service clusters. The maximum number of concurrent remote Web service calls that can be made from a single Web server is determined by the maxConnection attribute of the <connectionManagement> element in the machine.config file. In ASP.NET 1.1, by default the maxConnection value was set to 2. This old default maxConnection value was much too low for sites like microsoft.com that have hundreds of different applications that make remote Web service calls. The result was that ASP.NET requests would queue while waiting for the remote Web service calls to complete. (You can view the number of queued ASP.NET requests via the perfmon counter ASP.NET\Requests Queued.) To enable more calls to execute concurrently to a remote Web service (and thereby improve the performance of applications on the site), we increased the maxConnection value to 40 for our quad processor Web servers. (The general recommended value for maxConnection is 12 times the number of CPUs, but tune this to suit your specific situation.)
In ASP.NET 2.0, however, you no longer need to configure maxConnection manually as it is now automatically scaled and set. This is a result of the new configuration section for the processModel tag in machine.config (for more information about the processModel element, see "processModel Element (ASP.NET Settings Schema))".
<system.web>
<processModel autoConfig="true" />
</system.web>
With autoConfig enabled in machine.config (this is the default setting), ASP.NET sets the value of the maxConnection parameter to 12n (where n is the number of CPUs). Enabling autoConfig also causes the following: the maxWorkerThreads parameter and the maxIoThreads parameter are set to 100, the minFreeThreads parameter is set to 88n, the minLocalRequestFreeThreads parameter is set to 76n, and the minWorkerThreads is set to 50.
Prior to making use of autoConfig in ASP.NET 2.0 to automatically scale and set values for maxConnection and the other attributes in the list, be sure to remove any manually set values for these parameters as these values would be used instead of the autoConfig values. This is something to keep in mind when migrating from ASP.NET 1.1 (where maxConnection needed to be explicitly set) to ASP.NET 2.0, where there are defaults.
Again, the autoConfig values for maxConnection and the other attributes listed previously are somewhat arbitrary and may not work for absolutely every instance, but I have found that these limits work well for nearly all microsoft.com applications.
If you decide that you need to tune the maxConnection value manually, be cautious when increasing its value as this can lead to an increase in CPU utilization. This increase is caused by the fact that more incoming requests can be processed by ASP.NET instead of having them wait for their turn to call the Web service. Of course, you should remember that the maxConnection attribute does not affect local Web service calls, only remote calls.
6. Beware of Unhandled Exceptions
When porting ASP.NET 1.1 Web sites or applications to ASP.NET 2.0, it is extremely useful to be aware of a major change in the default policy for unhandled exceptions. In the .NET Framework 1.1 and 1.0, unhandled exceptions on managed threads were ignored and, because the applications continued to run, these exceptions often remained hidden. Unless you attached a debugger to catch the exception, you would not have realized anything was wrong. Under ASP.NET 2.0, however, when an unhandled exception is thrown, the ASP.NET-based application will quit unexpectedly. This can seriously impact your site or application’s availability if there are a lot of unhandled exceptions that were previously covered up by the old default exception handling policy.
The best way to address this is to do thorough testing and eliminate the unhandled exceptions (which should really not be present in your application). However, for migrations of very large applications in which it may be difficult to determine where the exception is occurring or if you need to migrate a lot of legacy applications for which thorough, individual testing isn’t feasible, you have a couple of options. When migrating the microsoft.com Web site to ASP.NET 2.0, we changed the unhandled exception policy back to the default behavior that occurs in ASP.NET 1.1 and ASP.NET 1.0.
To enable this legacy default exception handling behavior, add the following code to the aspnet.config file:
<configuration>
<runtime>
<legacyUnhandledExceptionPolicy
enabled="true" />
</runtime>
</configuration>
The code is located in these two folders:
%WINDIR%\Microsoft.NET\Framework\v2.0.50727 (on x86 or SYSWOW64 systems) and %WINDIR%\Microsoft.NET\Framework64\v2.0.50727 (on x64 systems).
This change will essentially revert the .NET Framework to the old 1.1 and 1.0 behavior. Consider this a short-term fix because ultimately it is masking issues with your application that are really bugs. However, it is a very handy way to avoid availability issues due to unexpectedly terminating worker processes. For more information about this behavior change, see "Unhandled exceptions cause ASP.NET-based applications to unexpectedly quit in the .NET Framework 2.0".
7. Ensure Proper Proxy Server Configuration
A Web server administrator can specify the proxy server to use for HTTP requests to the Internet by configuring the <configuration><system.net><defaultProxy> element in the machine.config file.
In the microsoft.com production environment, we configure the <defaultProxy> value to use the system default proxy (as the firewall client is not installed) and we use <location allowOverride="false"> tags to prevent the <defaultProxy> element from being overridden by application devel- opers (who might accidentally publish a web.config file with an internal proxy server):
<configuration>
<location allowOverride="false">
<system.net>
<defaultProxy>
<proxy usesystemdefault="true" />
</defaultProxy>
</system.net>
</location>
</configuration>
In the microsoft.com internal preproduction and staging environments, we set the usesystemdefault attribute to false, bypassonlocal to true, and add a proxy bypasslist (see Figure 2). The bypasslist lists regular expressions that describe addresses that do not use the specified proxy. Again, this section is contained within <location allowOverride="false"> tags to prevent developers from specifying their own proxy server in their web.config files. (These proxy servers are often internal and calls to them break when pages are published to production.) Any attempts to specify a proxy server in preproduction or staging will result in an ASP.NET error at run time, which will force developers to remove this configuration prior to publishing to production.
Figure 2 Setting Bypasslist
<configuration>
<location allowOverride="false">
<system.net>
<defaultProxy>
<proxy
usesystemdefault="false"
proxyaddress = "https://proxy.server.foo.com:80"
bypassonlocal = "true" />
<bypasslist>
<add address="10\.*"/>
<add address="dns\.foo\.com" />
<add address="name1\.name2\.foo\.com" />
</bypasslist>
</defaultProxy>
</system.net>
</location>
</configuration>
8. Do Not Display Custom Errors to Everyone
As I mentioned, it is critical to not allow detailed ASP.NET error messages to be returned remotely by Web servers in the production environment.
In the microsoft.com production and staging environment’s root web.config files, the <configuration><system.web><customErrors> mode attribute is set to RemoteOnly so that custom errors are displayed to remote clients and ASP.NET errors are displayed to localhost (to allow troubleshooting by the Web server administrators). Note that the <customErrors> element is contained within a <location> tag with allowOverride="false" (see Figure 3). This is to prevent individual application owners from accidentally (or deliberately) setting mode="Off" and spilling detailed ASP.NET error messages into the Internet.
Figure 3 Preventing the Display of Error Messages
<configuration>
<location allowOverride="false">
<system.web>
<customErrors mode="RemoteOnly" defaultRedirect=
"/errorpages/generic_customerror.aspx">
<error statusCode="404" redirect="/errorpages/filenotfound_customerror.aspx" />
</customErrors>
</system.web>
</location>
<configuration>
Also remember, as I mentioned earlier, that the use of the <deployment retail="true"/> switch in the machine.config file turns off the ability to show detailed ASP.NET error messages both to remote clients and locally. This deployment retail switch should still be your primary method of turning off these error messages if you are running ASP.NET 2.0. (To get detailed information regarding ASP.NET exceptions, use the Application Event Log.)
In the microsoft.com internal-facing preproduction environment root web.config file, the <customErrors> mode attribute is set to off so that ASP.NET errors are always displayed to both localhost and remote clients. This is to enable debugging and troubleshooting. In addition, no custom error pages are configured:
<configuration>
<location allowOverride="false">
<system.web>
<customErrors mode="Off" />
</system.web>
</location>
<configuration>
9. Know When to Enable Tracing
ASP.NET traces are generated during the execution of an ASP.NET page, and capture interesting details about the Web request, page control tree, and execution of various stages of the page lifecycle and controls. As well, custom messages that can be written to the trace by the page developer may be displayed. The trace can be appended to the response output of the page being traced or examined as part of the list of traced requests in the application trace viewer. This feature is primarily intended for development-time debugging scenarios in internal preproduction environments and should not be used for production deployments.
In the microsoft.com production and staging environments root web.config files, the <configuration><system.web><trace> enabled attribute is set to "false" so that the ability to output trace information in a Web page is disabled. Note that the <trace> element is contained within a <location> tag with allowOverride="false". This is to prevent individual application owners from accidentally (or deliberately) setting enabled="true" and outputting detailed ASP.NET tracing information onto the Internet:
<configuration>
<location allowOverride="false">
<system.web>
<trace enabled="false" localOnly="true"
pageOutput="false" requestLimit="10" traceMode="SortByTime" />
</system.web>
</location>
<configuration>
<system.web><trace>
As mentioned earlier, use of the <deployment retail="true"/> switch in the machine.config file also turns off the ability to output ASP.NET trace output in a Web page. This switch should still be your primary method of turning off tracing output if you are running the .NET Framework 2.0.
To be completely certain that tracing cannot accidentally become enabled in external-facing production environments, at microsoft.com we remove the actual trace.axd handler from the root web.config file or comment it out, like so:
<!--
<add path="trace.axd" verb="*" type=
"System.Web.Handlers.TraceHandler"
validate="True" />
-->
10. Disable Session State Web Farms
Because all Web sites on microsoft.com are currently clustered using Network Load Balancing (NLB) with no affinity (to allow even distribution of requests across all servers in the cluster), there is no guarantee that the same server will handle all requests for a given application. As a result, the ASP.NET Session State module is disabled to prevent use of the Session property by application developers. The guideline we give to application developers who need to maintain state is to use View State (which maintains the state in a structure within the page code and, as a result, uses no server resources).
To disable Session State on a Web server, you should simply remove the following child node from the <httpModules> node:
<add name="Session" type=
"System.Web.SessionState.
SessionStateModule"/>
I’ve assumed here that you’ve already seen and even used ASP.NET configuration files (machine.config, root web.config, and individual application web.config files), but if you haven’t, the following ASP.NET Quickstart Tutorial will help: asp.net/QuickStart/aspnet/doc/management/fileformat.aspx.
The article mentions that the ASP.NET 2.0 configuration system now also includes a number of very helpful features that allow administrators to lock individual elements and attributes of configuration, using the lockItems and lockCollections attributes. These attributes and their use are documented in the following article "General Attributes Inherited by Section Elements".
Jeff Toews is a Systems Engineering Manager who has been a member of the Microsoft.com Operations Team, based in Redmond, WA, for six years. You can reach him with any technical questions or comments at mscomblg@microsoft.com.
© 2008 Microsoft Corporation and CMP Media, LLC. All rights reserved; reproduction in part or in whole without permission is prohibited.