If you’re using IIS Express without Visual Studio installed (for example, WebMatrix), then you’ll first need to install ASP.NET 4.5 features:

dism /online /enable-feature /featureName:NetFx4Extended-ASPNET45 /all

UPDATE: The above dism command is specific to Win8.  Basically, you need to make sure ASP.NET features are enabled on the machine in order for performance counters to get registered.  This happens automatically when you install VS.  For Win7 client, you can enable ASP.NET 3.5 features with a command like the following (note that /all isn’t available for automatically enabling parent features):

dism /online /enable-feature /featurename:IIS-WebServerRole /featurename:WAS-WindowsActivationService /featurename:WAS-ProcessModel /featurename:WAS-NetFxEnvironment /featurename:WAS-ConfigurationAPI /featurename:IIS-ApplicationDevelopment /featurename:IIS-ASPNET /featurename:IIS-DefaultDocument /featurename:IIS-NetFxExtensibility /featurename:IIS-ISAPIExtensions /featurename:IIS-ISAPIFilter /featurename:IIS-RequestFiltering /featurename:IIS-Metabase /featurename:IIS-WMICompatibility /featurename:IIS-LegacyScripts /featurename:IIS-IIS6ManagementCompatibility /featurename:IIS-WebServerManagementTools /featurename:IIS-HttpTracing

Re-running on the original hardware after setting 'Number of RSS Queues' to 4 for each NIC:

• 4 Intel Xeon Quad-Core processors (16 cores total)
• 4 Broadcom BCM5708C NetXtreme II GigE NICs

ORIGINAL POST:

In the previous post, I stated that when adding multiple NICs to a web server you should ensure that the network load is balanced across those NICs.  In order to maximize CPU on your multi-processor server you will still need to ensure that those NICs support Receive Side Scaling (RSS), a technology that distributes network processing across multiple CPUs.  In this post I will demonstrate how to verify that RSS is working.

Recall our webserver specs:

• 4 Intel Xeon Quad-Core processors (16 cores total)
• 4 Broadcom BCM5708C NetXtreme II GigE NICs

After load-balancing our server and clients, we now have the following wcat node configuration:

After collecting data on the server NICs, enabling processor counters for all processors and running my wcat scenario, I was able to see what happened to incoming network traffic.  In the the table below, notice that only 8 of the 16 CPUs processed the Deferred Procedure Calls (DPCs).  Meanwhile, the other 8 CPUs spent more time Idle.  Something in my server configuration was preventing RSS from utilizing all 16 cores.

To compare, we borrowed some hardware from the IIS performance lab which had previously gone through this RSS exercise.

The new webserver has a total of 20 processors and 10 NICs:

• 20 Intel Xeon CPU E7 – 4860 processors
• 6 Intel Gigabit ET Dual Port Server Adapter NICs
• 4 Intel 82576NS Gigabit Ethernet Controller NICS

The following is the new load-balanced wcat node configuration:

And the following was the compilation of NIC and counter data for a wcat run.  RSS was configured properly on this node – notice how the DPC processing was evenly distributed across all processors (taking into account that RSS CPUs 8-9 and e-f had increased load).

To compile this data, I did the following:

1. Noted which client(s) pointed to which IP(s), based on my load balancing
2. Found the # of requests sent in the per-client statistics section of the wcat log
3. Used IPConfig to find the physical address for each NIC
4. Used NICStats.exe (available from the Windows Driver Kit) to map the NIC physical addresses to the RSS CPUs*
5. Collected Processor performance counters for all CPU instances:
• Processor(N)\% Processor Time
• Processor(N)\% Idle Time
• Processor(N)\% Interrupt Time
• Processor(N)\% DPC Time

Conclusion:

When running a web server with multiple NICs, be sure to verify that RSS is working properly.  If you have questions, check with the NIC vendor and do your own testing once the NICs are installed.

Script requirements:

• Debugging Tools for Windows
• TinyGet, from the IIS Resource Kit Tools
• PsKill from PsTools

Debug Script:

reg add "HKLM\Software\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\w3wp.exe" /v Debugger /t REG_SZ /d "cdb.exe -c \".server tcp:port=9999\"" /f
iisreset /restart
start /b tinyget5 -srv:localhost -uri:/notfound.aspx -status:404
sleep 3
windbg -remote tcp:port=9999,server=localhost
pskill cdb
pskill w3wp
reg delete "HKLM\Software\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\w3wp.exe" /f

UPDATE:

If you are setting managed breakpoints, you will want to break when the managed assembly loads using "sxe ld:Assembly".  For instance, "sxe ld:clr" or "sxe ld:System.Web.dll".

If you want to debug compilation, you should delete the "Temporary ASP.NET Files" under the framework installation folder for your website after the iisreset.

For reference, the web server has the following specs:

• 4 Intel Xeon Quad-Core processors (16 cores total)
• 4 Broadcom BCM5708C NetXtreme II GigE NICs

Each NIC on the web server has a separate IP address:

> nslookup webserver

Server:  dnsserver
Address:  192.168.0.1

Name:    webserver

Addresses:  192.168.0.2
          192.168.0.3
          192.168.0.4
          192.168.0.5

If you remember from my “Hello, World” post, we use wcat to apply load from 3 remote, physical clients. Each client uses the web server’s hostname when issuing requests, which should ideally use a round robin rotation across the web server IPs.

Before I proceed, let me describe what happens to incoming network traffic as I understand it:

• tcpip.sys issues network interrupt when it receives data
• OS dispatches an ISR (interrupt service routine) to handle the interrupt
• tcpip.sys can’t issue another interrupt while the ISR is running
• ISR schedules a lower priority DPC (deferred procedure call) to handle the rest of the work
• ISR quickly returns so that tcpip.sys can issue more interrupts

If the NICs are getting equal load and RSS (receive-side scaling) is enabled on the web server, then the network interrupts and their corresponding DPCs should be distributed across the multiple cores.  We collected ETW profiles and found that while our interrupts were handled by all cores, DPCs were only handled by 8 of the 16 cores with cpu0 doing most of the work:

This led us to wonder if the NICs really were getting equal load from a round robin DNS lookup.  The quickest and easiest way to ensure a balanced load was to manually edit the client etc\hosts files and force each client to use a separate NIC.  Since there were 4 NICs we needed a 4th client, which also means an increase in load.

The following results are for Server R2 with .NET 4. Baseline is not load balanced, Result is.

Notice the improvements in %CPU and throughput?  Using Result as our new load-balanced baseline, we re-ran HelloWorld on the same build where we saw the original regression.  After maxing our CPU and increasing our load, throughput is now less than 5% off from the baseline.

Conclusion: When adding multiple NICs, carefully tune your web server and infrastructure in order to maximize its performance.  Profile to verify that your server is behaving as expected.  I recognize that it is not this easy to load balance production servers, but there should be tools available to help you do this.  I don’t work on production servers, so don’t ask me which tools – but I would love to hear about them from you!

Update: To collect the ETW profiles above, I enabled INTERRUPT and DPC on the xperf command-line:

xperf –on PROC_THREAD+LOADER+INTERRUPT+DPC –f <etwFile>

Introduction

There is a file called aspnet.config in the .Net framework installation. This file is used to specify startup flags for both ASP.NET and CLR for those settings that are needed very early in the worker process lifetime, when the config system is not yet present.

In IIS 7.5 we added an attribute to the <applicationPools> configuration collection called CLRConfigFile, on the applicationHost.config file, which allows you to specify your own aspnet.config, for a particular application pool.

There is another attribute on the <applicationPools> called managedRuntimeLoader, which specify the library in charge of loading CLR. By default, this task is executed by ASP.NET native layer, in the webengine4.dll. If no loader is specified, or if an error occurs, the loading falls back to IIS itself.

IIS 7.5 issue.

There was a problem in IIS 7.5, in which the managedRuntimeLoader was never provided with the path to this CLRConfigFile, so it was forced to use the one in the framework directory. So if you wanted to use the CLRConfigFile, to setup different CLR flags for an application pool, you will need to set the attribute AND explicitly clear the managedRuntimeLoader attribute on the application pool definition on applicationHost.config:

  ...

<system.applicationHost>

  ...

        <applicationPools>

            <!-- Set the CLRConfigFile and clear managedRuntimeLoader -->

            <add name="MyAppPool" CLRConfigFile="c:\myconfigs\myapp1\aspnet.config" managedRuntimeLoader="" />

  ...

The second step shouldn’t be necessary and is confusing. The clearing of the attributes basically forces the fall back to IIS runtime loader, which uses the correct CLRConfigFile.

This has been fixed on IIS 8, where you only need to set the CLRConfigFile attribute.

To verify this is working, you can change one parameter on the aspnet.config such as disabling the server garbage collector. (Note: Disabling the server garbage collector is a good thing if you want to save memory, at the expense of throughput. Typical use of this is hosting scenarios, where web sites has low traffic and higher site density is desirable.)

<configuration>

    <runtime>

  ...

        <gcServer enabled="false"/>

Attaching a debugger to the worker process and using SOS command !eeversion will show the garbage collector mode being used:

C:\>cdb.exe -pn w3wp.exe

Microsoft (R) Windows Debugger Version 6.13.0009.1140 X86

Copyright (c) Microsoft Corporation. All rights reserved.

  …

0:044> .loadby sos clr

0:044> !eeversion

4.0.30319.239 retail

Workstation mode

SOS Version: 4.0.30319.239 retail build

Make sure your box has more than one logical core; otherwise workstation mode will always be used.

Implementing a loader

HRESULT __stdcall

LoadManagedRuntime(

        _In_ PCWSTR pwszRuntimeVersion,

        __deref_out IUnknown ** ppManagedRuntimeHost)

{

    return LoadRuntimeInternal(pwszRuntimeVersion, ppManagedRuntimeHost);

}

HRESULT __stdcall

LoadManagedRuntimeEx(

        _In_ PCWSTR pwszRuntimeVersion,

        _In_ PCWSTR pszClrConfigFile,

        __deref_out IUnknown ** ppManagedRuntimeHost)

{

    return LoadRuntimeInternal(pwszRuntimeVersion, ppManagedRuntimeHost, pszClrConfigFile);

}

HRESULT LoadRuntimeInternal(_In_ PCWSTR pwszRuntimeVersion, __deref_out IUnknown** ppHost /*= NULL*/, _In_ PCWSTR pwszAspNetHostConfig /*= NULL*/) {

    HMODULE hModule = NULL;

    HRESULT hr = S_OK;

    IUnknown *pHost = NULL;

    PFNCorBindToRuntimeHost pCorBindToRuntimeHost;

    TCHAR szPath[MAX_PATH + 1];

    TCHAR szAspConfig[MAX_PATH + 1];

    ExpandEnvironmentStrings(L"%windir%\\system32\\mscoree.dll", szPath, MAX_PATH);

    if(pwszAspNetHostConfig == NULL) {

        ExpandEnvironmentStrings(ASPNET_CONFIG, szAspConfig, MAX_PATH);

        pwszAspNetHostConfig = szAspConfig;

    }

    hModule = LoadLibrary(szPath);

    ON_NULL_EXIT(hModule);

    pCorBindToRuntimeHost = (PFNCorBindToRuntimeHost)GetProcAddress(hModule, "CorBindToRuntimeHost");

    ON_NULL_EXIT(pCorBindToRuntimeHost);

    hr = pCorBindToRuntimeHost(

                    L"v4.0.30319",

                    L"svr",

                    pwszAspNetHostConfig,

                    NULL, // reserved

                    STARTUP_CONCURRENT_GC|STARTUP_LOADER_OPTIMIZATION_MULTI_DOMAIN_HOST|STARTUP_LOADER_SAFEMODE|STARTUP_DISABLE_COMMITTHREADSTACK|STARTUP_HOARD_GC_VM,

                    CLSID_CLRRuntimeHost,

                    IID_IUnknown,

                    (VOID**)&pHost);

Cleanup:

    if (pHost != NULL && ppHost != NULL) {

        pHost->AddRef();

        *ppHost = pHost;

    }

    if(hModule != NULL) {

        FreeLibrary(hModule);

    }

    return hr;

}

  ...

<system.applicationHost>

  ...

        <applicationPools>

            <!-- Set the CLRConfigFile and managedRuntimeLoader -->

            <add name="MyAppPool" CLRConfigFile="c:\myconfigs\myapp1\aspnet.config" managedRuntimeLoader="c:\mynativedlls\myruntimeloader.dll" />

  ...

Note that debugging such a code on IIS will require the use of “Image File Execution Options”, since the code is executed very early on the worker process lifecycle. An easier way to test might be to load the library yourself and call the functions, the way IIS would (see the test project).

Conclusion

IIS 7.5 required for you to clean the managedRuntimeLoader attribute, if you wanted to change CLR parameters on your own CLRConfigFile.

Implementation of a managed runtime loader requires a native DLL which exports a LoadManagedRuntime() and/or LoadManagedRuntimeEx().

Thanks for reading.

Originally posted at https://blogs.msdn.com/b/josere/

Introduction.

A couple of new features are introduced in ASP.NET 4.5 to improve startup time of web apps. Both features use a flag on the web.config to enable them. The features don’t do any magic by themself. They just enable features available on the operating system underneath or in the CLR APIs.

Prefetch feature

The prefetch feature only works on Windows 8 Server or newer OS. It requires the Os to be configured and the enablePrefetchOptimization flag on the <system.web/compilation> section of web.config to be set. The flag only has effect at the app level. No subdirectory config or location tag makes sense here. The setting is set to false by default, so users must explicitly set it to true in order for it to work.

To enable the feature, run the Enable-MMAgent power shell command:

    powershell Enable-MMAgent -OperationAPI

The SuperFetch service might need to be started. Verify is running:

C:\>sc query sysmain

SERVICE_NAME: sysmain

        TYPE               : 20  WIN32_SHARE_PROCESS

        STATE              : 4  RUNNING

                                (STOPPABLE, NOT_PAUSABLE, ACCEPTS_SHUTDOWN)

        WIN32_EXIT_CODE    : 0  (0x0)

        SERVICE_EXIT_CODE  : 0  (0x0)

        CHECKPOINT         : 0x0

        WAIT_HINT          : 0x0

Prefetch Implementation.

The features simple uses the operation recorder API. See OperationStart and OperationEnd APIs on msdn. This APIs require an integer used as an Id of the operation. We use a hash code of the app domain app id, which is a string.

string domainId = HttpRuntime.AppDomainAppId;

int operationId = domainId.GetHashCode();

The OperationStart call is made the first time the app domain is created. The OperationEnd API is called at the end of the request in which we called OperationStart. A .pf file should be created on the prefetch directory:

C:\>dir %windir%\Prefetch

 Volume in drive C has no label.

 Volume Serial Number is CC5B-113C

 Directory of C:\Windows\Prefetch

02/16/2012  04:17 PM    <DIR>          .

02/16/2012  04:17 PM    <DIR>          ..

02/16/2012  04:17 PM           164,832 Op-W3WP.EXE-90E54C33-765F360E.pf

               1 File(s)        164,832 bytes

               2 Dir(s)  30,203,154,432 bytes free

Multi-Core Jitting feature.

The multi-core jitting feature is enabled by default, and work on all OS where ASP.NET 4.5 is available. You can disable it by setting the profileGuidedOptimizations enumerated value on the <system.web/compilation> section on the web.config, to None.

Multi-Core Jitting Implementation.

The feature uses the new class ProfileOptimization class of the System.Runtime namespace. The class is called the first time the app domain is created. First, we set the profile root to the codegen directory, which by default is something like:

"%windir%\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\myapp\weirdhash1\weirdhash2"

Remember that the temporary files folder value can be changed on tempDirectory attribute of the <system.web/compilation> section of the web.config.

After setting the profile root, we call StartProfile() with the filename "profileoptimization.prof". The CLR code underneath handles all the magic. If you examine the contents of the temporary ASP.NET file folder, you will see the file there:

C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\dotnetnuke\59a60e53\a2ac2624>dir *.prof

 Volume in drive C has no label.

 Volume Serial Number is CC5B-113C

 Directory of C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\dotnetnuke\59a60e53\a2ac2624

02/16/2012  04:38 PM             9,828 profileoptimization.prof

               1 File(s)          9,828 bytes

               0 Dir(s)  30,203,035,648 bytes free    

Conclusion.

The ASP.NET Prefetch feature uses the operation recording APIs to improve startup time of web applications. The feature is enabled when a Windows Server 8 machine is configured for the operation API and enabledPrefetchOptimization flag is set on the web.config.

The multi-core jitting feature is available in more operating systems and is on by default. It uses a new CLR feature for profile base optimization.

Thanks for reading.

Originally posted at https://blogs.msdn.com/b/josere/

Performance Monitor

Windows ships with the Performance Monitor tool, or PerfMon for short. This GUI allows you to view live performance counter data on either the local machine or on remote systems. Simply run `perfmon` and navigate to Performance->Monitoring Tools->Performance Monitor. Right click to add or remove counters. Press ‘Ctrl+G’ to switch the graph type.  Report View is handy when monitoring multiple counters.

Perfecto and LogMan

Perfecto is a Data Collector Set that plugs into PerfMon, and was created by Jose Reyes. This is a quick, simple way to collect ASP.NET, IIS, CLR and system counters which give a good overview of ASP.NET performance.

LogMan is a command line tool for managing PerfMon Data Collector Sets.

Try the following:

1. Download Perfecto, unzip and run the setup script
2. Run `logman start –n “Service\ASPNET Perfecto”`
3. Make a few web requests to localhost
4. Run `logman stop –n “Service\ASPNET Perfecto”`
5. View the report at "%SystemDrive%\PerfLogs\Admin\ASPNET Perfecto\000001\report.html”

Don’t be afraid to customize Perfecto to your needs: add performance counters, change the report directory, etc.  Right click ‘ASPNET Perfecto’ to edit the properties in PerfMon.  Right click again and Save Template… to create an updated version of Perfecto.DataCollectorSet.xml.

TypePerf and TList

For basic counter collections, you can use the TypePerf command-line tool. It can be combined with the TList command in order to collect counters which are specific to a given process. For example, you can collect the Processor Time for all processes by running:

`for /t “tokens=2 delims=. “ %a in (‘tlist’) do typeperf –sc 1 “Process(%a)\% Processor Time”

WCat

WCat also supports the collection of performance counters.  This is our primary way of collecting performance counters since most of our scenarios use WCat.  Take a look at our settings.ubr file to see which counters we track:

settings
{
    counters
    {
        interval = 10;

        counter = ".NET CLR Exceptions(w3wp)\\# of Exceps Thrown / sec";
        counter = ".NET CLR Jit(w3wp)\\% Time in Jit";
        counter = ".NET CLR Jit(w3wp)\\IL Bytes Jitted / sec";
        counter = ".NET CLR Loading(w3wp)\\% Time Loading";
        counter = ".NET CLR Loading(w3wp)\\Current appdomains";
        counter = ".NET CLR Loading(w3wp)\\Current Assemblies";
        counter = ".NET CLR LocksAndThreads(w3wp)\\# of current logical Threads";
        counter = ".NET CLR LocksAndThreads(w3wp)\\# of current physical Threads";
        counter = ".NET CLR LocksAndThreads(w3wp)\\# of current recognized threads";
        counter = ".NET CLR LocksAndThreads(w3wp)\\# of total recognized threads";
        counter = ".NET CLR LocksAndThreads(w3wp)\\Queue Length / sec";
        counter = ".NET CLR LocksAndThreads(w3wp)\\Total # of Contentions";
        counter = ".NET CLR LocksAndThreads(w3wp)\\Contention Rate / sec";
        counter = ".NET CLR LocksAndThreads(w3wp)\\Current Queue Length";
        counter = ".NET CLR Memory(w3wp)\\% Time in GC";
        counter = ".NET CLR Memory(w3wp)\\# Bytes in all Heaps";
        counter = ".NET CLR Memory(w3wp)\\# Gen 0 Collections";
        counter = ".NET CLR Memory(w3wp)\\# Gen 1 Collections";
        counter = ".NET CLR Memory(w3wp)\\# Gen 2 Collections";
        counter = ".NET CLR Memory(w3wp)\\# Induced GC";
        counter = ".NET CLR Memory(w3wp)\\Allocated Bytes/sec";
        counter = ".NET CLR Memory(w3wp)\\Finalization Survivors";
        counter = ".NET CLR Memory(w3wp)\\Gen 0 heap size";
        counter = ".NET CLR Memory(w3wp)\\Gen 1 heap size";
        counter = ".NET CLR Memory(w3wp)\\Gen 2 heap size";
        counter = ".NET CLR Memory(w3wp)\\Large Object Heap size";
        counter = ".NET CLR Memory(w3wp)\\Large Object Heap size";
        counter = ".NET CLR Memory(w3wp)\\# of Pinned Objects";
        counter = ".NET CLR Security(w3wp)\\% Time in RT checks";
        counter = ".NET CLR Security(w3wp)\\Stack Walk Depth";
        counter = ".NET CLR Security(w3wp)\\Total Runtime Checks";
        counter = "ASP.NET Applications(__Total__)\\Cache Total Entries";
        counter = "ASP.NET Applications(__Total__)\\Cache Total Hit Ratio";
        counter = "ASP.NET Applications(__Total__)\\Cache Total Turnover Rate";
        counter = "ASP.NET Applications(__Total__)\\Output Cache Entries";
        counter = "ASP.NET Applications(__Total__)\\Output Cache Hits";
        counter = "ASP.NET Applications(__Total__)\\Output Cache Hit Ratio";
        counter = "ASP.NET Applications(__Total__)\\Output Cache Turnover Rate";
        counter = "ASP.NET Applications(__Total__)\\Compilations Total";
        counter = "ASP.NET Applications(__Total__)\\Errors Total/Sec";
        counter = "ASP.NET Applications(__Total__)\\Pipeline Instance Count";
        counter = "ASP.NET Applications(__Total__)\\Requests Executing";
        counter = "ASP.NET Applications(__Total__)\\Requests in Application Queue";
        counter = "ASP.NET Applications(__Total__)\\Requests/Sec";
        counter = "ASP.NET\\Application Restarts";
        counter = "ASP.NET\\Request Wait Time";
        counter = "ASP.NET\\Requests Current";
        counter = "ASP.NET\\Requests Queued";
        counter = "ASP.NET\\Requests Rejected";
        counter = "Memory\\Available MBytes";
        counter = "Memory\\Cache Faults/sec";
        counter = "Memory\\Demand Zero Faults/sec";
        counter = "Memory\\Page Faults/sec";
        counter = "Memory\\Pages/sec";
        counter = "Memory\\Transition Faults/sec";
        counter = "Memory\\Pool Nonpaged Bytes";
        counter = "Memory\\Pool Paged Bytes";
        counter = "PhysicalDisk(_Total)\\% Disk Time";
        counter = "PhysicalDisk(_Total)\\% Disk Read Time";
        counter = "PhysicalDisk(_Total)\\% Disk Write Time";
        counter = "Process(w3wp)\\% Processor Time";
        counter = "Process(w3wp)\\Handle Count";
        counter = "Process(w3wp)\\ID Process";
        counter = "Process(w3wp)\\Private Bytes";
        counter = "Process(w3wp)\\Thread Count";
        counter = "Process(w3wp)\\Virtual Bytes";
        counter = "Process(w3wp)\\Working Set";
        counter = "Processor(_Total)\\% Interrupt Time";
        counter = "Processor(_Total)\\% Privileged Time";
        counter = "Processor(_Total)\\% Processor Time";
        counter = "Processor(_Total)\\% User Time";
        counter = "System\\Context Switches/sec";
        counter = "System\\System Calls/sec";
        counter = "Web Service(_Total)\\Get Requests/sec";
        counter = "Web Service(_Total)\\Post Requests/sec";
        counter = "Web Service(_Total)\\Connection Attempts/sec";
        counter = "Web Service(_Total)\\Current Connections";
        counter = "Web Service(_Total)\\ISAPI Extension Requests/sec";
        counter = "Web Service Cache\\URI Cache Hits %";
        counter = "Web Service Cache\\Kernel: URI Cache Hits %";
        counter = "Web Service Cache\\File Cache Hits %";
    }
}

PAL

Performance Analysis of Logs (PAL) is a tool by Microsoft Premier Field Engineer, Clint Huffman. I haven’t had a chance to preview this tool yet, so I don’t want to speculate. It does seem to have a powerful feature set and following, so it’s definitely worth a look if you need a more comprehensive view.

Step 1: State the Objective

We want to ensure that the basic ASP.NET pipeline does not regress from one release to the next.  We use server throughput (requests per second) as our performance metric.  We want throughput to be on-par or better than the previous release (aka, baseline).

Step 2: Create the Scenario

We have a basic IIS website with the following ASPX page:

<html>
    <%=”Hello, World!” %>
</html>

Step 3: Run the Scenario

Most of our scenarios use the wcat load tool to test the web server.  You can download wcat here.

In order to detect regressions we need to have reproducible results.  Here are some of the ways that we reduce variability:

• Eliminate the variables.  Since I am measuring ASP.NET performance, .NET should be my one variable.  Other factors such as hardware, OS, IIS, wcat version and the scenario itself should remain constant.
• Use a private network.  Eliminate unnecessary network traffic to the web server which could affect results.
• Minimize server processes.  Again, we want to avoid competition for server resources.  We usually disable Windows Firewall and run the wcat controller and clients from separate machines.
• Measure warm.  Since we are testing throughput and not startup, we can reduce the variability by adding a warm-up period.  This will reduce JIT compilation and population of caches which could cause more variance.
• Max the CPU.  Server load and system resources factor into the equation as well.  We try to keep this near-constant by applying a full load during testing.  Our goal is to achieve at least 90% CPU usage during our runs.

We quantify our test variance (aka, noise) by running multiple iterations.  We ignore the first iteration which we’ve found to have greater variance, and calculate the average and standard deviation for the remaining three.  Standard deviation is our noise indicator.

Step 4: Analyze the Results

After doing our baseline and test runs, we should have results like the following:

These are the columns:

• Baseline, Result – Requests per second average for the last 3 iterations of the run
• Diff – Percentage of how far off the Result is from the Baseline
• Baseline StdDev, Result StdDev – Percentage of variance between the run iterations
• Pass / Fail – Whether the Diff is above our threshold.

Based on our experience we’ve set our threshold at 5%.  We define failures as runs that are more than 5% regressed.  Runs that show >5% improvement should also be investigated in order to understand and validate the cause.

We consider a run noisy if the standard deviation exceeds a threshold, which we also set at 5%.  If a failed run is noisy, we throw out one more iteration to see if the Diff and StdDev improve.  If they do we may ignore the failure and wait for the next run.  If the runs continue to be noisy, then the scenario should be investigated in order to further reduce the variance.

Step 5: Investigate Regressions

As the saying goes, “Measure Early, Measure Often”.  The less changes there are between your runs, the easier it will be to track down the cause of your regressions.  This is why the ASP.NET performance lab runs daily with builds from multiple branches.  Often times I’m able to quickly identify a regression simply by looking at the source control history.

Another way to quickly diagnose regressions is to enable performance counters or other non-invasive tracing which could help identify the cause.  We always save the wcat logs with our performance counters and other useful information such as throughput, working set, percent cpu usage, and HTTP responses.  The HTTP responses can rule out test failures, while the other diagnostics (combined with source control) can help narrow down the cause.

Finally, if a quick diagnosis is not possible, find a good profiler.  Some of the tools we use are the Visual Studio (F1) profiler, CLR Profiler, and XPerf.  I hope to demonstrate some of these in future posts.

https://blogs.msdn.com/b/web_performance_blog/

I hope that our blog becomes a valuable resource for you and I welcome your feedback.  Happy reading!

-Christy Henriksson, ASP.NET Performance Champ