Introduction

There is a file called aspnet.config in the .Net framework installation. This file is used to specify startup flags for both ASP.NET and CLR for those settings that are needed very early in the worker process lifetime, when the config system is not yet present.

In IIS 7.5 we added an attribute to the <applicationPools> configuration collection called CLRConfigFile, on the applicationHost.config file, which allows you to specify your own aspnet.config, for a particular application pool.

There is another attribute on the <applicationPools> called managedRuntimeLoader, which specify the library in charge of loading CLR. By default, this task is executed by ASP.NET native layer, in the webengine4.dll. If no loader is specified, or if an error occurs, the loading falls back to IIS itself.

IIS 7.5 issue.

There was a problem in IIS 7.5, in which the managedRuntimeLoader was never provided with the path to this CLRConfigFile, so it was forced to use the one in the framework directory. So if you wanted to use the CLRConfigFile, to setup different CLR flags for an application pool, you will need to set the attribute AND explicitly clear the managedRuntimeLoader attribute on the application pool definition on applicationHost.config:

  ...

<system.applicationHost>

  ...

        <applicationPools>

            <!-- Set the CLRConfigFile and clear managedRuntimeLoader -->

            <add name="MyAppPool" CLRConfigFile="c:\myconfigs\myapp1\aspnet.config" managedRuntimeLoader="" />

  ...

The second step shouldn’t be necessary and is confusing. The clearing of the attributes basically forces the fall back to IIS runtime loader, which uses the correct CLRConfigFile.

This has been fixed on IIS 8, where you only need to set the CLRConfigFile attribute.

To verify this is working, you can change one parameter on the aspnet.config such as disabling the server garbage collector. (Note: Disabling the server garbage collector is a good thing if you want to save memory, at the expense of throughput. Typical use of this is hosting scenarios, where web sites has low traffic and higher site density is desirable.)

<configuration>

    <runtime>

  ...

        <gcServer enabled="false"/>

Attaching a debugger to the worker process and using SOS command !eeversion will show the garbage collector mode being used:

C:\>cdb.exe -pn w3wp.exe

Microsoft (R) Windows Debugger Version 6.13.0009.1140 X86

Copyright (c) Microsoft Corporation. All rights reserved.

  …

0:044> .loadby sos clr

0:044> !eeversion

4.0.30319.239 retail

Workstation mode

SOS Version: 4.0.30319.239 retail build

Make sure your box has more than one logical core; otherwise workstation mode will always be used.

Implementing a loader

HRESULT __stdcall

LoadManagedRuntime(

        _In_ PCWSTR pwszRuntimeVersion,

        __deref_out IUnknown ** ppManagedRuntimeHost)

{

    return LoadRuntimeInternal(pwszRuntimeVersion, ppManagedRuntimeHost);

}

HRESULT __stdcall

LoadManagedRuntimeEx(

        _In_ PCWSTR pwszRuntimeVersion,

        _In_ PCWSTR pszClrConfigFile,

        __deref_out IUnknown ** ppManagedRuntimeHost)

{

    return LoadRuntimeInternal(pwszRuntimeVersion, ppManagedRuntimeHost, pszClrConfigFile);

}

HRESULT LoadRuntimeInternal(_In_ PCWSTR pwszRuntimeVersion, __deref_out IUnknown** ppHost /*= NULL*/, _In_ PCWSTR pwszAspNetHostConfig /*= NULL*/) {

    HMODULE hModule = NULL;

    HRESULT hr = S_OK;

    IUnknown *pHost = NULL;

    PFNCorBindToRuntimeHost pCorBindToRuntimeHost;

    TCHAR szPath[MAX_PATH + 1];

    TCHAR szAspConfig[MAX_PATH + 1];

    ExpandEnvironmentStrings(L"%windir%\\system32\\mscoree.dll", szPath, MAX_PATH);

    if(pwszAspNetHostConfig == NULL) {

        ExpandEnvironmentStrings(ASPNET_CONFIG, szAspConfig, MAX_PATH);

        pwszAspNetHostConfig = szAspConfig;

    }

    hModule = LoadLibrary(szPath);

    ON_NULL_EXIT(hModule);

    pCorBindToRuntimeHost = (PFNCorBindToRuntimeHost)GetProcAddress(hModule, "CorBindToRuntimeHost");

    ON_NULL_EXIT(pCorBindToRuntimeHost);

    hr = pCorBindToRuntimeHost(

                    L"v4.0.30319",

                    L"svr",

                    pwszAspNetHostConfig,

                    NULL, // reserved

                    STARTUP_CONCURRENT_GC|STARTUP_LOADER_OPTIMIZATION_MULTI_DOMAIN_HOST|STARTUP_LOADER_SAFEMODE|STARTUP_DISABLE_COMMITTHREADSTACK|STARTUP_HOARD_GC_VM,

                    CLSID_CLRRuntimeHost,

                    IID_IUnknown,

                    (VOID**)&pHost);

Cleanup:

    if (pHost != NULL && ppHost != NULL) {

        pHost->AddRef();

        *ppHost = pHost;

    }

    if(hModule != NULL) {

        FreeLibrary(hModule);

    }

    return hr;

}

  ...

<system.applicationHost>

  ...

        <applicationPools>

            <!-- Set the CLRConfigFile and managedRuntimeLoader -->

            <add name="MyAppPool" CLRConfigFile="c:\myconfigs\myapp1\aspnet.config" managedRuntimeLoader="c:\mynativedlls\myruntimeloader.dll" />

  ...

Note that debugging such a code on IIS will require the use of “Image File Execution Options”, since the code is executed very early on the worker process lifecycle. An easier way to test might be to load the library yourself and call the functions, the way IIS would (see the test project).

Conclusion

IIS 7.5 required for you to clean the managedRuntimeLoader attribute, if you wanted to change CLR parameters on your own CLRConfigFile.

Implementation of a managed runtime loader requires a native DLL which exports a LoadManagedRuntime() and/or LoadManagedRuntimeEx().

Thanks for reading.

managedRuntimeLoader.zip

Introduction.

A couple of new features are introduced in ASP.NET 4.5 to improve startup time of web apps. Both features use a flag on the web.config to enable them. The features don’t do any magic by themself. They just enable features available on the operating system underneath or in the CLR APIs.

Prefetch feature

The prefetch feature only works on Windows 8 Server or newer OS. It requires the Os to be configured and the enablePrefetchOptimization flag on the <system.web/compilation> section of web.config to be set. The flag only has effect at the app level. No subdirectory config or location tag makes sense here. The setting is set to false by default, so users must explicitly set it to true in order for it to work.

To enable the feature, run the Enable-MMAgent power shell command:

    powershell Enable-MMAgent -OperationAPI

The SuperFetch service might need to be started. Verify is running:

C:\>sc query sysmain

SERVICE_NAME: sysmain

        TYPE               : 20  WIN32_SHARE_PROCESS

        STATE              : 4  RUNNING

                                (STOPPABLE, NOT_PAUSABLE, ACCEPTS_SHUTDOWN)

        WIN32_EXIT_CODE    : 0  (0x0)

        SERVICE_EXIT_CODE  : 0  (0x0)

        CHECKPOINT         : 0x0

        WAIT_HINT          : 0x0

Prefetch Implementation.

The features simple uses the operation recorder API. See OperationStart and OperationEnd APIs on msdn. This APIs require an integer used as an Id of the operation. We use a hash code of the app domain app id, which is a string.

string domainId = HttpRuntime.AppDomainAppId;

int operationId = domainId.GetHashCode();

The OperationStart call is made the first time the app domain is created. The OperationEnd API is called at the end of the request in which we called OperationStart. A .pf file should be created on the prefetch directory:

C:\>dir %windir%\Prefetch

 Volume in drive C has no label.

 Volume Serial Number is CC5B-113C

 Directory of C:\Windows\Prefetch

02/16/2012  04:17 PM    <DIR>          .

02/16/2012  04:17 PM    <DIR>          ..

02/16/2012  04:17 PM           164,832 Op-W3WP.EXE-90E54C33-765F360E.pf

               1 File(s)        164,832 bytes

               2 Dir(s)  30,203,154,432 bytes free

Multi-Core Jitting feature.

The multi-core jitting feature is enabled by default, and work on all OS where ASP.NET 4.5 is available. You can disable it by setting the profileGuidedOptimizations enumerated value on the <system.web/compilation> section on the web.config, to None.

Multi-Core Jitting Implementation.

The feature uses the new class ProfileOptimization class of the System.Runtime namespace. The class is called the first time the app domain is created. First, we set the profile root to the codegen directory, which by default is something like:

"%windir%\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\myapp\weirdhash1\weirdhash2"

Remember that the temporary files folder value can be changed on tempDirectory attribute of the <system.web/compilation> section of the web.config.

After setting the profile root, we call StartProfile() with the filename "profileoptimization.prof". The CLR code underneath handles all the magic. If you examine the contents of the temporary ASP.NET file folder, you will see the file there:

C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\dotnetnuke\59a60e53\a2ac2624>dir *.prof

 Volume in drive C has no label.

 Volume Serial Number is CC5B-113C

 Directory of C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\dotnetnuke\59a60e53\a2ac2624

02/16/2012  04:38 PM             9,828 profileoptimization.prof

               1 File(s)          9,828 bytes

               0 Dir(s)  30,203,035,648 bytes free    

Conclusion.

The ASP.NET Prefetch feature uses the operation recording APIs to improve startup time of web applications. The feature is enabled when a Windows Server 8 machine is configured for the operation API and enabledPrefetchOptimization flag is set on the web.config.

The multi-core jitting feature is available in more operating systems and is on by default. It uses a new CLR feature for profile base optimization.

Thanks for reading.

During .Net 4.5 development we tried to improve the performance of ASP.NET in shared hosting scenarios, where many web sites might share the same machine. In such environment traffic is usually low. In this case startup time becomes very important. That is the time it takes a web site to receive a first request and respond to it, when the worker process was down, and the web site was already compiled, which is the most common case. Also assuming that other worker processes are running other web sites, so shared components will be already loaded.

Another important metric is the site density. The more sites we can fit in a box the lower the cost for the hosters. The site metric we used was the unshared working set. That is the total working set minus the shared working set.

Scenarios.

The first thing we needed to validate our work was a set of scenarios that could allow us to measure our progress, through this two metrics: Startup and Site Density. We decided to use community created and used web applications. We gathered a bunch of applications deployed by the Web Platform Installer (Web PI).

We focused on three applications: DotNetNuke, Blogengine and IBuySpy+*. The decision was based on our planned work, which included features that will deal with assemblies in the Bin folder. So we picked three apps that have different levels of Bin folder use. DotNetNuke has heavy use of assemblies in the Bin folder, Blogengine has some use and IBuySpy+ has none.

Startup

We added several features during this product cycle to improve startup time. These features are the results of the work of several teams at Microsoft, including CLR and Windows Server. ASP.NET team worked on enabling these features in our product. The feature list is below:

1. Prefetch.
2. Bin assemblies interning.
3. Multi-core JIT.

Memory.

The features implemented to aid the memory site were:

1. HighDensityWebHosting performance scenario.
2. Bin assemblies interning.

Numbers.

Baseline: Windows 2008 R2, .Net 4.0

Results: Windows 8 Server,  .Net 4.5

Hardware used:

  <Machine>

    <Hardware>

      <Processor Num="2">Intel(R) Xeon(R) CPU 5160 @ 3.00GHz</Processor>

      <Memory>4096</Memory>

      <HardDrives Num="2">

        <HardDrive>HP DF146ABAA9 SCSI Disk Device SCSI[136.73 GB]</HardDrive>

        <HardDrive>SEAGATE ST3300656SS SCSI Disk Device SCSI[279.39 GB]</HardDrive>

      </HardDrives>

    </Hardware>

  </Machine>

Other resources for the features implemented:

https://www.asp.net/vnext/whats-new#_Toc_perf

*IBuySpy+ is a variation of the IBuySpy (a web app we use extensively on the ASP.NET perf team), which added some of the new features of ASP.NET/.Net framework added during the v3.5 timeframe.

The scenario

The sample scenario we considered here is the hypothetical page in which we need to aggregate the results of five remote feeds, in a second tier. Requesting the feeds has a latency, which we set to 100ms in our test. Requesting this asynchronously have a clear advantage, since five request will take around 100ms. If we requested them synchronously, in sequence, it will take 500ms. The figure below shows the scenario:

The problem

Asynchronous request in ASP.Net 4.0 has some scalability issues when a huge load (beyond the hardware capabilities) is put on such scenario. The problem is due to the nature of allocation on asynchronous scenarios. In these conditions, allocation will happen when the asynchronous operation starts, and it will consumed when it completes. By that time, it’s very possible the objects have been moved to generation 1 or 2 by the garbage collector.

When this happens, increasing the load will show increase on request per second (rps) until a point. Once we pass that point, the time spend in the garbage collector will start to become a problem and the rps will start to dip, having a negative scaling effect. The figure below shows the effect of increasing the load in our particular scenario. The pick is achieved in the sample data with 770 clients.

CPU throttling

To try to fix this problem, we implemented a throttling mechanism in 4.5. Simply, when the CPU usage went beyond a point, requests were sent to the ASP.NET native queue. This queue already existed, for IIS integrated pipeline mode only, and is used also to limit concurrency based on the parameters maxConcurrentRequestsPerCPU and maxConcurrentThreadsPerCPU in aspnet.config (or through registry keys).

We added an additional setting in aspnet.config for the percentage of CPU usage we will monitor to start the throttling. The new setting is called percentCpuLimit. So whenever the CPU usage goes beyond that point, we will start sending some of the request to the queue, effectively reducing the concurrency and the amount of managed buffers that get allocated and collected by the GC. This improved considerably the scalability of this type of scenarios in our tests.

The default value of the setting is 99%. To disable the throttling, set percentCpuLimit to 0:

<configuration>

    <!-- ... -->

    <system.web>

        <applicationPool percentCpuLimit="0"/>

    </system.web>

</configuration>

Performance counters

In order to know if throttling is happening, the existing “ASP.NET\Requests Queued” counter can’t be used. The counter includes the requests waiting for a CLR thread pool thread and the requests in the ASP.NET queue. So we added the counter below:

Note the new counter will also include requests queued for any other concurrency limiting setting. That is maxConcurrentRequestsPerCPU and maxConcurrentThreadsPerCPU. The checks are made in that order, then percentCpuLimit is checked.

New defaults

There are a few settings that affect this type of scenarios. Some of them in the framework, some of them in IIS/http.sys. We review them and tried to change the default values of some of them, so they work better out of the box.

System.Net.ServicePointManager.DefaultConnectionLimit: This setting controls how many concurrent outbound connections you can start with System.Net classes, such as HttpWebRequest. The setting can also be set in machine.config setting <system.net/connectionManagement/add[@address='*'] maxconnection />, as long as <processModel autoConfig=false />. In .Net 4.0 we used to set this parameter to 12 times the number of logical CPUs, when autoConfig=true. We have changed that in 4.5 to Infinite (actually is int32.MaxValue).

requestQueueLimit: This setting can be set in <processModel requestQueueLimit /> on machine.config or in <system.web/applicationPool requestQueueLimit /> in aspnet.config. The setting used to work on 4.0 and below as a fix setting, meaning that once we reach that many total requests (5000 by default) we started to return 503 status code. We fixed this in ASP.NET 4.5 to return 503 when the number of requests on the ASP.NET queue is bigger than this value, rather than using the total number of requests.

Http.sys queueLength: <system.applicationHost/applicationPools/applicationPoolDefaults.queueLength /> parameter can be used in IIS to set this. The default in IIS 7.5 is 1000. This can sometimes not be sufficient on a managed application with heavy garbage collection. This is cause by some GC threads using higher priority threads, which can make the IIS threads processing the queue to starve for CPU. However, the setting affects non-paged memory. The default value was not changed. If your web site returns 503 status code, you can try increasing this value.

System.Data MaxPoolSize: The number of database connection on the pool can also affect the scalability of these types of scenarios. The value can be set inside the connection string. The value to choose depends primarily by the Sql server, rather than the client machine, so the default was not changed in 4.5.

Summary

To review, a new concurrency setting was added in 4.5 in aspnet.config named percentCpuLimit, to alleviate async scenarios scalability. A new performance counter named “ASP.NET\Requests In Native Queue” was also added, to know when the throttling is happening. And some default values were changed for some .Net settings. All this is only valid for IIS integrated pipeline mode.

There was a bug in ASP.Net 2.0 in which the “ASP.NET\Requests Queued” performance counter only reported values when a request was queued in the ASP.Net Queue. It didn’t count the requests that were waiting for the CLR thread pool. This was only in IIS7+ integrated mode and was different than the behavior in classic mode.

The problem is fixed now in ASP.Net 4.0, so if you were monitoring the counter before, you might start seeing values and worry about the performance of your application, when this behavior is expected. A patch for 2.0 was also released in KB979917. The problem was discovered by the SharePoint team who uses the counter to monitor the health of their sites. Also, updates released after this KB should contain the fix.

More Details.

Request for ASP.Net are always delivered first in the context of an IIS thread. At that point, ASP.Net queues a work item to the CLR thread pool and returns, so IIS can use this thread to process another request. Once the CLR thread pool decides to activate this work item, the request continues on the context of a CLR worker thread. It could take some time until the thread pool activates the requests, so these acts as a queue of request. Next, we monitor if we have too many concurrent request, and might decide to put the request in the ASP.Net queue

This behavior is very similar in both classic and integrated mode. The main difference is that in integrated mode there is a single queue for the whole app pool, while classic mode has a queue in each app domain. Also different is the settings used to configure those queues. None of these details are relevant to this discussion. Tmarq has blogged about this.

The point is that “Requests Queued” needs to be incremented just before queuing the work item to the CLR thread pool and decremented just after the work item started to run in the worker thread. If the request is then stored in the ASP.NET queue, the counter needs to be incremented once more, and decremented when it is finally removed from the queue and scheduled to be executed. This was done that way in classic mode, but in integrated mode we were only doing the second increment/decrement.

In conclusion:

The immediate effect of the bug was that the same application, under the same load, could show values on the counter in classic mode, but showed zero if moved to integrated mode. This is now fixed. On the other hand, customers that always used integrated mode might start to see values in this counter when moving to 4.0 (or applying the patch for 2.0) and be concerned, trying to find a performance issue that doesn’t exist. This is the main reason of this article. Hopefully, our customers will find it helpful.

Additional Resources

Other performance related articles: https://blogs.msdn.com/josere  

Many users seem to run into issues when using ASP.NET Performance counters. With that in mind, I decided to write this article to explain, in as much detail as possible, how the ASP.NET Performance counters were implemented. In the hope people will find this useful when dealing with such problems. This might be also useful for support engineers helping customers.

The basis.

The first thing to know is how counters were implemented in the Os at the time of ASP.NET performance counters initial coding. This is explained in great detail in the msdn web site. Don’t worry. You don’t need to read all that documentation to understand the basis of how this works.

Basically, what you need, in order to provide performance counters, is a native dll with at least three exports: Open, Close and Collect. You also need an .ini file with the counters names and descriptions, and an .h file with the indexes.

You also need a registry key under HKLM\SYSTEM\CurrentControlSet\services\<your_app>\Performance. This key should contain the “Library”, “Open”, “Close” and “Collect” values with your native dll and its exports.

Once this is ready, you can call lodctr.exe (or LoadPerfCounterTextStrings() from your code) passing the .ini file path as a parameter. This will register the performance counters. The call will populate some other registry values on the registry key mentioned before. The performance counters names are stored at HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Perflib\nnn, where nnn is the language id. You can use unlodctr.exe (or UnloadPerfCounterTextStrings()) to unregister. Errors during this process are usually stored in the event log.

Versioned versus Generic

ASP.NET Performance counters has generic categories (e.g. ASP.NET and ASP.NET Applications) and versioned categories (e.g. ASP.NET v2.50727 and ASP.NET Apps v2.50727). This allows side-by-side versions with different performance counters implementation. The higher version registered on the machine “owns” the generic objects, while all versions “owns” their respective versioned objects.

Note that if you have an ASP.NET application running in a lower version, then the generic counters will not report numbers for that application. You need to use the versioned categories in that case. For instance, if you have both 2.0 and 4.0 registered in the same machine, and a web application is running in an application pool using 2.0, then the ASP.NET and ASP.NET Application categories will not report data for that app. You need to use ASP.NET v2.50727 and ASP.NET Apps v2.50727 categories instead.

Global versus Instance-based.

ASP.NET has some counters that are global. That means that they are aggregated for all web apps running in that version of ASP.NET. These counters are reported thru the ASP.NET category (or ASP.NET vx.xxxxx).

Instance based counters can be obtained for a particular app. These counters are accessible with the ASP.NET Applications category (or ASP.NET Apps vx.xxxxx, for versioned). The instance name is the app domain id of the app. When hosted in IIS, this ID is made with the “metabase” path (e.g. _LM_W3SVC_1_ROOT_MYAPP). When running in Visual Studio, the ID is a hex integer, created with a hash code of a combination of virtual and physical path of the app.

ASP.NET performance counters files.

In the case of ASP.NET 2.0, the native library providing performance counters is called aspnet_perf.dll and it resides in the framework directory (e.g. C:\Windows\Microsoft.NET\Framework64\v2.0.50727). This library implements the exports for the generic performance counters (e.g. ASP.NET), the versioned performance counters (e.g. ASP.NET_2.0.50727) and the aspnet_state service performance counters. Older versions had the export methods directly on the aspnet_isapi.dll.

In 4.0 we added a new library called aspnet_counters.dll. This library resides in the System32 directory (e.g. c:\Windows\System32 and c:\Windows\SysWOW64). It replaced aspnet_perf.dll in the registration, but it’s actually a shim, which redirects all calls to aspnet_perf.dll. The later still is the one doing all the work.

The problem with the previous registration design was that a full path was required in the “Library” value in both the versioned and the generic keys. This created several problems. First, you needed separated registry keys for 64 bit and 32 bit. Also, in Windows 2003, the performance counters will sometimes stop working when using Windows Management Instrumentation (WMI). Additionally, 32 bit clients (e.g. mmc.exe /32 perfmon.msc) couldn’t access performance counters for the session state service.

Other files used by performance counters in the framework directory are the followings: The aspnet_perf2.ini file is used for generic counters, while aspnet_perf.ini is used to register the versioned counters. They share the same aspnet_perf.h. The state service files are aspnet_state_perf.ini and aspnet_state_perf.h.

ASP.NET performance counters registry keys.

ASP.NET has multiple registry keys under HKLM\SYSTEM\CurrentControlSet\services. The list below shows all the keys and their use:

·         ASP.NET: Registry key for the generic counters. When 2.0 is the highest version registered, this will contain the registration for 32 bit clients. In 4.0 this key is always platform agnostic, so both 64 and 32 clients can use this entry.

·         ASP.NET_64: Registry key for the generic counters on 64 bit Os. When 2.0 is the highest version registered, this will contain the registration for 64 bit clients. In 4.0 and above, this key will be deleted by the registration process.

·         ASP.NET_2.0.50727: Registry key for the 2.0 versioned counters. It contains the registration for 32 bit clients.

·         ASP.NET_64_2.0.50727: Registry key for the 2.0 versioned counters. It contains the registration for 64 bit clients.

·         ASP.NET_4.0.30319: Registry key for the 4.0 versioned counters. The key is platform agnostic. It will work with any client, regardless of its bitness.

·         aspnet_state: Contains registration for session state performance counters. In 64 bit Os, only 64 bit library will be present.

Each key also contains sub-keys. Their use is described below:

a)      Performance: Sub-key with “Library” and exports. Other values are set by the Os.

b)      Names: Pipe names. One per each running worker process. See “What happens at runtime?”. This sub-key appears only on the 32 bit versioned keys.

c)      Linkage: In 4.0 and above we store here the referenced version number of the ASP.NET registered, in a value called “Export”. This sub-key was not present or used in ASP.NET 2.0.

d)      Parameters: Not related to performance counters. Only on the session state service.

The registration process.

The registration of the performance counters is executed from within aspnet_regiis.exe tool. Registration happens with switches -i, -iru, -ir or -update (“undocumented”). During framework setup this tool is actually called as a custom action by the installation framework. The parameter used during setup is -iru. Note that starting with Vista, since the .Net framework is part of the Os, the performance counters can also be registered during package installation (e.g. pkgmgr /iu:IIS-WebServerRole;…;IIS-ASPNET). This is done by a tool called aspnetca.exe, in the IIS folder (e.g. %windir%\System32\inetsrv), which does a job similar to that of aspnet_regiis.exe.

To register the generic counters, aspnet_regiis.exe first calls UnloadPerfCounterTextStrings() with “ASP.NET” as parameter. This ensures we always have a clean registration. Additionally, in 4.0 and newer, we also unregister the “ASP.NET_64” driver and remove the associated registry key. Then aspnet_regiis.exe calls LoadPerfCounterTextStrings() passing the file path to aspnet_perf2.ini as parameter.

For versioned counters, we also unregister first (via UnloadPerfCounterTextStrings()) and then register with aspnet_perf.ini as parameter. This is done every time you call aspnet_regiis.exe with any registration switch.

The state service counters are a little bit different. They are only registered together with the service, so once this is registered, the performance counters are not unreg/reg again in subsequent calls to aspnet_regiis.exe. Also, under a 64 bit Os, the counters are not registered if you run the 32 bit version (wow64 process) of aspnet_regiis.exe.

When unregistering, aspnet_regiis.exe has to make sure the next available registered version of ASP.NET takes ownership of the generic counters. So after we call UnloadPerfCounterTextStrings() for the generic counters, we search the registry trying to find another version of ASP.NET in the HKLM\SOFTWARE\Microsoft\ASP.NET registry. If one is found, we call LoadPerfCounterTextStrings() with that version’s aspnet_perf2.ini. For instance, if both 2.0 and 4.0 are registered, then, calling aspnet_regiis.exe -u from the 4.0, would actually associate the 2.0 version with the generic counters. Versioned counters will simply be unregistered for that version.

The aspnet_regiis.exe uses log text files to store traces of the operations performed. The logs are stored in the %TEMP% folder and are named something like ASPNETSetup_00xxx.log. These files can be examined, along with the event log when looking for clues of what went wrong.

Running aspnet_regiis.exe -iru from the latest version is the safest and recommended workaround when trying to fix issues with ASP.NET performance counters.

What happens during runtime?

Internally, ASP.NET performance counters mechanism uses a client/server approach. In this case, the server is the worker process; say w3wp.exe, while the clients are other processes that consume performance counter data using Win32 API, such as PDH, registry functions (e.g. perfmon), or a higher level API such as System.Diagnostics or WMI.

During runtime, whenever a web application is run and the worker process starts, it loads webengine.dll (or webengine4.dll in .Net 4.0). This library, among other many things, will initialize the performance counters local storage in the process. It will also create a named pipe to which clients will be able to connect and read the counters data from. The name of the pipe is stored in the HKLM\SYSTEM\CurrentControlSet\services\ASP.NET_x.x.xxxxx\Names registry key, so clients can find it.

The processing of the named pipe is asynchronous. The library uses an IO completion thread from the CLR, which wakes up every time some client wants to get performance counter data. Other than that, the feature doesn’t consume much cycles, except when some part of ASP.NET runtime wants to set or increment a performance counter value. For instance, when a request comes, we need to increase the value of the "ASP.NET\Requests Current" counter. When the request is done, we decrement it.

When a client such as perfmon, wants to get performance counters data, using their API of choice, the Os will loop through all registered counters at HKLM\SYSTEM\CurrentControlSet\services, calling the Open export. This will eventually call the aspnet_perf.dll, which will initialize the client side of the connection. Among other things, the library will create two background threads.

One thread monitors the HKLM\SYSTEM\CurrentControlSet\services\ASP.NET_x.x.xxxxx\Names registry key, so it will be informed of any new worker process.

The second thread will enter a loop, waking up every 400 milliseconds (or sooner, if instructed by a call to the Collect export) and reading data thru the named pipes from all worker processes. The data is stored in a global area, and it will be returned to the clients through the Collect export. That call usually executes on the main thread of the client process. A locking mechanism is used to protect the shared memory.

If an error is detected during the reading of the pipes, the error is ignored and the data is cleared, just in case the worker process dies unexpectedly. So, counters could report zeroes when running on heavy load.

When transferring data thru the pipe, the client will first send its version number to the worker process, which will validate that it matches its own. If it does it will send the performance counter data back.

Any error during clients calls to Collect should be stored in the event log. There is not internal reporting for errors on the gathering threads.

Conclusion.

Hopefully, the information contained here will help somebody. As usual, I can’t really see what the possible uses of the information presented here are. In any case, I hope it will benefit someone.

Additional Resources

Other performance related articles: https://blogs.msdn.com/josere  

Performance counters in msdn: https://msdn.microsoft.com/en-us/library/aa373083(VS.85).aspx

OpenPerformanceData: https://msdn.microsoft.com/en-us/library/aa372200(VS.85).aspx

CollectPerformanceData: https://msdn.microsoft.com/en-us/library/aa371898(VS.85).aspx

ClosePerformanceData: https://msdn.microsoft.com/en-us/library/aa371895(VS.85).aspx  

lodctr: https://msdn.microsoft.com/en-us/library/ms894317.aspx

unlodctr: https://msdn.microsoft.com/en-us/library/ms929082.aspx

LoadPerfCounterTextStrings: https://msdn.microsoft.com/en-us/library/aa372187.aspx   

UnloadPerfCounterTextStrings: https://msdn.microsoft.com/en-us/library/aa373205(VS.85).aspx

It’s not a secret that the less server controls you have in an ASP.NET web page the better the performance. The memory allocation of the objects is fast but processing will vary depending on many factors. Additionally, the garbage collector needs to expend time releasing the memory. Furthermore, server controls are processed recursively several times during request, in the processing of events such as init or rendering. Also, the more controls you have the more memory you consume.

It’s very common to have a big number of instances of server controls. A simple grid page can contain hundreds of instances. And since they are created and destroyed in each request their number can be staggering. Imagine a web server under load serving thousands of requests per second.  

Furthermore, the internal implementation of the System.Web.UI.Control class (the base class for all server controls in the web forms page, including the page itself, user controls and many others) can have an impact on performance when used one way or another.

Occasionally used fields and rarely used fields

The class contains a big number of private fields used to support a large array of features. Some of these fields are used only occasionally, so to optimize the size of the controls, the ASP.NET team moved them to a different class. The control class contains a reference to this class and the first time the private field is needed, an instance of the occasional fields is created.

In addition to this, there is a group of fields that is used only rarely, so these fields were moved to yet another class and a reference was included in the occasional field class. With the instantiation happening pretty much the same way than the occasional fields.

The code below ilustrates this relationship. The control class will have the ocassional and rare fields encapsulated in private properties. Take a look at the reflector desassembly for the Control class for the actual implementation. Note that accessing a rare field will instantiate both the rare and the occasional field objects.

In that case, server controls using occasional or rare fields can easily double or triple the instance of object created. We have verified through performance runs that reducing the number of total instances improves the performance (and the memory consumption) of ASP.NET requests.  Reducing the number of controls that instantiate occasional and rare fields also improves performance and memory usage.

ASP.NET 4.0 optimization.

In fact, during .NET 4.0 development, we analyzed which fields were used more often than others in several scenarios and re-organized them accordingly. Moving the less used fields to the occasional and rare fields classes, or move them back to the control class itself when they were used very often.  This optimization improved the performance (throughput and working set usage) of the scenarios used.

In 2.0 the concept of ocassional and rare fields existed. So this was a concern back then too. However, with the years we've been adding new features that required new private fields in that class. So, the initial optimization was kind of stale. During 4.0 we added even more fields, and we noticied regressions in performance. So we decided it was time for a re-optimization. The regression disappeared and we even saw better performance in some scenarios.

Using Reflector on System.Web.dll on 2.0 and 4.0 versions will show the differences on implementation of the Control class.

Measuring your server controls usage.

Unfortunately, it’s hard from a customer stand point to control when the occasional fields and the rare fields are instantiated. Because this is triggered by the usage of specific features that is not evident that will use those fields.

The only thing we can do is to measure the number of controls and which ones are instantiating the occasional and rare fields. Seeing what type generates the rare and occasional fields might allow you to improve the performance of your pages, if this is vital in your scenario. In general, reducing the number of controls will improve your performance.  But it’s better to reduce those with occasional and/or rare fields.

Note that, like all performance improvement, you usually have a trade off to make. For instance, we’ve seen the user controls (.ascx files) instantiate the occasional fields. However, user controls has other advantages so you need to find a balance. A typical place to reduce controls is the ones created inside grid cells, since they are replicated in the table.

Using an http module to view your controls.

To view the fields’ usage in your server controls, we created an http module that uses reflection to hack into the controls hierarchy, starting with the page. Another notification is used to attach a table with a report so that can be seen when the page is requested, pretty much the same way trace.axd report is concatenated when trace=”true” is used on a page. We could have used global.asax instead but this solution seems cleaner to me.

You can use the module to calculate the controls usage in your own pages. You can deploy the mentioned module to your app and request your pages. You probably want to do that on a development environment. You don’t want your customers to see the report table on all your pages. Plus, an additional module in the pipeline has a performance cost, especially since we use reflection.

To use the module, you can copy the correct language version to your App_Code folder. I have provided a C# and a VB.NET versions in the attachment of this blog. Click here to get the modules source code.

You also need to edit your web.config to enable the module to run. The setting changed depends on the pipeline mode you are using.  For a integrated pipeline mode, you have to use the <system.webServer/modules> section:

<configuration>

    <!-- ... -->

    <system.webServer>

    <!-- ... -->

        <modules>

            <add name="ControlFieldsStatsModule" preCondition="managedHandler" type="ControlFieldsStatsModule, App_Code"/>

    <!-- ... -->

For Classic mode (includes Visual Studio mini web server, IIS6 etc) you use the <system.web/httpModules> section:

<configuration>

    <!-- ... -->

    <system.web>

    <!-- ... -->

        <httpModules>

            <add name="ControlFieldsStatsModule" type="ControlFieldsStatsModule, App_Code"/>

    <!-- ... -->

The report.

Requesting the page, will attach a table like below, with all control types and if they use occasional and/or rare fields.  The stats shown were obtained from the default page on a freshly created Dynamic Data project, connected to Northwind database:

Server Control Fields Stats:

Conclusion

This information could potentially be used to select what types of controls are more likely to improve the performance of the page if they are removed. The field’s usage was optimized in .Net 4.0, but it's impossible to do that for all scenarios.

Other performance related articles: https://blogs.msdn.com/josere

ControlFieldsStatsModule.zip

Historically, using ETW is not very friendly. Existent tools are hard-to-use and the data generated is raw and mostly unprocessed. You can today generate ASP.NET traces using logman.exe and then use tracerpt.exe to generate a CSV file. This data is hard to look at it.  Another option is to use FREB to generate traces. That log contains useful information, but sometimes is too much, especially if you just want to take a quick peek at a request or two. And ASP.NET events are only listed. The relation between the events is not used.

Currently, here are some issues associated with using ETW to measure ASP.NET performance:

1.       Not user friendly.

2.       Generates a lot of raw unprocessed data (for instance if generating a CSV file using logman.exe and tracerpt.exe).

3.       FREB generates a file per each request. When browsing the log folder, it’s impossible to know which file belongs to which request. There is no summary that shows all requests traced.

4.       Both methods just list the ASPNET events. They miss an important point. The information is not only in the fact that the event happened. An important piece of information is the duration between events. FREB shows duration for IIS event, but not for ASP.NET events. For instance, you can’t know how long your Page Load event took.

The tool that we are providing here will allow you to collect ETW traces for ASP.NET requests and look at them in a more friendly view. The tool doesn’t contain any executable. It just harnesses the functionality already in existence in Windows. In this case the data collector sets provided in the “Performance Monitor” (aka perfmon) in Vista or newer versions.

The tool is made out of three XML files. One file contains the data collector set definition (Perfecto.DataCollectorSet.xml). A second file (Reports.Microsoft.ASPNET.xml) contains the schema of the ASP.NET specific data included on ETW traces. A third xml file (Rules.Microsoft.ASPNET.xml) provides rules that will modify and rearrange the data so it’s easier to read and understand. Setup is very simple, just the XMLs and a setup.cmd script.

The tool can be used for purposes others than performance. For instance, you can see what modules are running in the pipeline. Or you can see what request are generating manage code to run.

We are considering adding this tool’s functionality to DebugDiag.

Setup steps:

1.       Download and unzip perfecto.zip.

2.       Run setup.cmd. This script will:

a)      Create a folder at “%programfiles%\perfecto” and copy there the three xml files listed before.

b)      Create the data collector set:

logman.exe import "Service\ASPNET Perfecto" -xml Perfecto.DataCollectorSet.xml

c)  Open “Performance Monitor” (perfmon).

Running steps:

1.       On perfmon, navigate to the "Performance\Data Collector Sets\User Defined\ASPNET Perfecto" node.

2.       Click the "Start the Data Collector Set" button on the tool bar.

3.       Wait for/or make requests to the server (more than 10 seconds).

4.       Click the "Stop the Data Collector Set" button on the tool bar.

5.       Click the "View latest report" button on the tool bar or navigate to the last report at "Performance\Reports\User Defined\ASPNET Perfecto".

Perfecto Requirements

1.       Only works on Vista or newer Os.

2.       Requires IIS tracing: (run dism /online /enable-feature /featurename:IIS-HttpTracing).

3.       Only ~140K of disk space is needed for the Xml files at "%programfiles%\perfecto".

Using the data collector set.

1.       How to View the Performance Counters Graph: The report has three different views in perfmon data collector sets. You can right-click on the report, select "View", and "Performance Monitor" to see the graphic view of the performance counters collected.

2.       How to Send Data by Email: Right-click on the report, select "View", and "Folder". Select the files you want to send, right click and select "Send To\Mail Recipient". You need to have email client properly configured.

3.       How to Start/Stop Collection From Batch File:

logman.exe start -n "Service\ASPNET Perfecto"

logman.exe stop -n "Service\ASPNET Perfecto"

Note: The View commands are also available as toolbar buttons.

Sometimes you can see an error like below:

Error Code: 0xc0000bf8
Error Message: At least one of the input binary log files contain fewer than two data samples.

This usually happens when you collected data too fast. The performance counters are set by default to collect every 10 seconds. So a fast start/stop sequence may end without enough counter data being collected. Always allow more than 10 seconds between a start and stop commands. Or otherwise delete the performance counters collector or change the sample interval.

Default sizes of the log files

One problem users usually have with ETW is that they can forget to turn off collection, so the ETL files can grow very big. To avoid this, we made the default in the data controller set to have a circular file: so older events are overwritten when the file grows beyond the default 5MB size. There is also a limit of 1GB for all files. This includes .etl, .blg and the reports XMLs.

These settings can be changed on the perfmon UI if you find they are not good for you:

1.       To change maximum size of the log file, right click on the “ASPNET Perfecto” data collector set and chose properties. Select the "Stop Condition" tab.

2.       To disable circular file, double click the AspnetTrace collector to open the properties and select the "File" tab.

3.       To change the size of all log files, open the data manager in “Reports\User Defined\ASPNET Perfecto”. Right click for properties and change the “Maximum root path size:”.

ETW Performance Effect on Production Servers.

Enabling ETW events for the ASP.NET provider will make a hello world page around 38% slower, without impacting noticeable the memory usage. The effect on a web app such as IBuySpy is lower: around 11%.                 

Sample report:

Below is a table that shows and excerpt of how the report looks like:

Other performance related articles: https://blogs.msdn.com/josere

perfecto.zip