If you’re using IIS Express without Visual Studio installed (for example, WebMatrix), then you’ll first need to install ASP.NET 4.5 features:

dism /online /enable-feature /featureName:NetFx4Extended-ASPNET45 /all

UPDATE: The above dism command is specific to Win8.  Basically, you need to make sure ASP.NET features are enabled on the machine in order for performance counters to get registered.  This happens automatically when you install VS.  For Win7 client, you can enable ASP.NET 3.5 features with a command like the following (note that /all isn’t available for automatically enabling parent features):

dism /online /enable-feature /featurename:IIS-WebServerRole /featurename:WAS-WindowsActivationService /featurename:WAS-ProcessModel /featurename:WAS-NetFxEnvironment /featurename:WAS-ConfigurationAPI /featurename:IIS-ApplicationDevelopment /featurename:IIS-ASPNET /featurename:IIS-DefaultDocument /featurename:IIS-NetFxExtensibility /featurename:IIS-ISAPIExtensions /featurename:IIS-ISAPIFilter /featurename:IIS-RequestFiltering /featurename:IIS-Metabase /featurename:IIS-WMICompatibility /featurename:IIS-LegacyScripts /featurename:IIS-IIS6ManagementCompatibility /featurename:IIS-WebServerManagementTools /featurename:IIS-HttpTracing

Re-running on the original hardware after setting 'Number of RSS Queues' to 4 for each NIC:

• 4 Intel Xeon Quad-Core processors (16 cores total)
• 4 Broadcom BCM5708C NetXtreme II GigE NICs

ORIGINAL POST:

In the previous post, I stated that when adding multiple NICs to a web server you should ensure that the network load is balanced across those NICs.  In order to maximize CPU on your multi-processor server you will still need to ensure that those NICs support Receive Side Scaling (RSS), a technology that distributes network processing across multiple CPUs.  In this post I will demonstrate how to verify that RSS is working.

Recall our webserver specs:

• 4 Intel Xeon Quad-Core processors (16 cores total)
• 4 Broadcom BCM5708C NetXtreme II GigE NICs

After load-balancing our server and clients, we now have the following wcat node configuration:

After collecting data on the server NICs, enabling processor counters for all processors and running my wcat scenario, I was able to see what happened to incoming network traffic.  In the the table below, notice that only 8 of the 16 CPUs processed the Deferred Procedure Calls (DPCs).  Meanwhile, the other 8 CPUs spent more time Idle.  Something in my server configuration was preventing RSS from utilizing all 16 cores.

To compare, we borrowed some hardware from the IIS performance lab which had previously gone through this RSS exercise.

The new webserver has a total of 20 processors and 10 NICs:

• 20 Intel Xeon CPU E7 – 4860 processors
• 6 Intel Gigabit ET Dual Port Server Adapter NICs
• 4 Intel 82576NS Gigabit Ethernet Controller NICS

The following is the new load-balanced wcat node configuration:

And the following was the compilation of NIC and counter data for a wcat run.  RSS was configured properly on this node – notice how the DPC processing was evenly distributed across all processors (taking into account that RSS CPUs 8-9 and e-f had increased load).

To compile this data, I did the following:

1. Noted which client(s) pointed to which IP(s), based on my load balancing
2. Found the # of requests sent in the per-client statistics section of the wcat log
3. Used IPConfig to find the physical address for each NIC
4. Used NICStats.exe (available from the Windows Driver Kit) to map the NIC physical addresses to the RSS CPUs*
5. Collected Processor performance counters for all CPU instances:
• Processor(N)\% Processor Time
• Processor(N)\% Idle Time
• Processor(N)\% Interrupt Time
• Processor(N)\% DPC Time

Conclusion:

When running a web server with multiple NICs, be sure to verify that RSS is working properly.  If you have questions, check with the NIC vendor and do your own testing once the NICs are installed.

Script requirements:

• Debugging Tools for Windows
• TinyGet, from the IIS Resource Kit Tools
• PsKill from PsTools

Debug Script:

reg add "HKLM\Software\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\w3wp.exe" /v Debugger /t REG_SZ /d "cdb.exe -c \".server tcp:port=9999\"" /f
iisreset /restart
start /b tinyget5 -srv:localhost -uri:/notfound.aspx -status:404
sleep 3
windbg -remote tcp:port=9999,server=localhost
pskill cdb
pskill w3wp
reg delete "HKLM\Software\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\w3wp.exe" /f

For reference, the web server has the following specs:

• 4 Intel Xeon Quad-Core processors (16 cores total)
• 4 Broadcom BCM5708C NetXtreme II GigE NICs

Each NIC on the web server has a separate IP address:

> nslookup webserver

Server:  dnsserver
Address:  192.168.0.1

Name:    webserver

Addresses:  192.168.0.2
          192.168.0.3
          192.168.0.4
          192.168.0.5

If you remember from my “Hello, World” post, we use wcat to apply load from 3 remote, physical clients. Each client uses the web server’s hostname when issuing requests, which should ideally use a round robin rotation across the web server IPs.

Before I proceed, let me describe what happens to incoming network traffic as I understand it:

• tcpip.sys issues network interrupt when it receives data
• OS dispatches an ISR (interrupt service routine) to handle the interrupt
• tcpip.sys can’t issue another interrupt while the ISR is running
• ISR schedules a lower priority DPC (deferred procedure call) to handle the rest of the work
• ISR quickly returns so that tcpip.sys can issue more interrupts

If the NICs are getting equal load and RSS (receive-side scaling) is enabled on the web server, then the network interrupts and their corresponding DPCs should be distributed across the multiple cores.  We collected ETW profiles and found that while our interrupts were handled by all cores, DPCs were only handled by 8 of the 16 cores with cpu0 doing most of the work:

This led us to wonder if the NICs really were getting equal load from a round robin DNS lookup.  The quickest and easiest way to ensure a balanced load was to manually edit the client etc\hosts files and force each client to use a separate NIC.  Since there were 4 NICs we needed a 4th client, which also means an increase in load.

The following results are for Server R2 with .NET 4. Baseline is not load balanced, Result is.

Notice the improvements in %CPU and throughput?  Using Result as our new load-balanced baseline, we re-ran HelloWorld on the same build where we saw the original regression.  After maxing our CPU and increasing our load, throughput is now less than 5% off from the baseline.

Conclusion: When adding multiple NICs, carefully tune your web server and infrastructure in order to maximize its performance.  Profile to verify that your server is behaving as expected.  I recognize that it is not this easy to load balance production servers, but there should be tools available to help you do this.  I don’t work on production servers, so don’t ask me which tools – but I would love to hear about them from you!

Update: To collect the ETW profiles above, I enabled INTERRUPT and DPC on the xperf command-line:

xperf –on PROC_THREAD+LOADER+INTERRUPT+DPC –f <etwFile>

Performance Monitor

Windows ships with the Performance Monitor tool, or PerfMon for short. This GUI allows you to view live performance counter data on either the local machine or on remote systems. Simply run `perfmon` and navigate to Performance->Monitoring Tools->Performance Monitor. Right click to add or remove counters. Press ‘Ctrl+G’ to switch the graph type.  Report View is handy when monitoring multiple counters.

Perfecto and LogMan

Perfecto is a Data Collector Set that plugs into PerfMon, and was created by Jose Reyes. This is a quick, simple way to collect ASP.NET, IIS, CLR and system counters which give a good overview of ASP.NET performance.

LogMan is a command line tool for managing PerfMon Data Collector Sets.

Try the following:

1. Download Perfecto, unzip and run the setup script
2. Run `logman start –n “Service\ASPNET Perfecto”`
3. Make a few web requests to localhost
4. Run `logman stop –n “Service\ASPNET Perfecto”`
5. View the report at "%SystemDrive%\PerfLogs\Admin\ASPNET Perfecto\000001\report.html”

Don’t be afraid to customize Perfecto to your needs: add performance counters, change the report directory, etc.  Right click ‘ASPNET Perfecto’ to edit the properties in PerfMon.  Right click again and Save Template… to create an updated version of Perfecto.DataCollectorSet.xml.

TypePerf and TList

For basic counter collections, you can use the TypePerf command-line tool. It can be combined with the TList command in order to collect counters which are specific to a given process. For example, you can collect the Processor Time for all processes by running:

`for /t “tokens=2 delims=. “ %a in (‘tlist’) do typeperf –sc 1 “Process(%a)\% Processor Time”

WCat

WCat also supports the collection of performance counters.  This is our primary way of collecting performance counters since most of our scenarios use WCat.  Take a look at our settings.ubr file to see which counters we track:

settings
{
    counters
    {
        interval = 10;

        counter = ".NET CLR Exceptions(w3wp)\\# of Exceps Thrown / sec";
        counter = ".NET CLR Jit(w3wp)\\% Time in Jit";
        counter = ".NET CLR Jit(w3wp)\\IL Bytes Jitted / sec";
        counter = ".NET CLR Loading(w3wp)\\% Time Loading";
        counter = ".NET CLR Loading(w3wp)\\Current appdomains";
        counter = ".NET CLR Loading(w3wp)\\Current Assemblies";
        counter = ".NET CLR LocksAndThreads(w3wp)\\# of current logical Threads";
        counter = ".NET CLR LocksAndThreads(w3wp)\\# of current physical Threads";
        counter = ".NET CLR LocksAndThreads(w3wp)\\# of current recognized threads";
        counter = ".NET CLR LocksAndThreads(w3wp)\\# of total recognized threads";
        counter = ".NET CLR LocksAndThreads(w3wp)\\Queue Length / sec";
        counter = ".NET CLR LocksAndThreads(w3wp)\\Total # of Contentions";
        counter = ".NET CLR LocksAndThreads(w3wp)\\Contention Rate / sec";
        counter = ".NET CLR LocksAndThreads(w3wp)\\Current Queue Length";
        counter = ".NET CLR Memory(w3wp)\\% Time in GC";
        counter = ".NET CLR Memory(w3wp)\\# Bytes in all Heaps";
        counter = ".NET CLR Memory(w3wp)\\# Gen 0 Collections";
        counter = ".NET CLR Memory(w3wp)\\# Gen 1 Collections";
        counter = ".NET CLR Memory(w3wp)\\# Gen 2 Collections";
        counter = ".NET CLR Memory(w3wp)\\# Induced GC";
        counter = ".NET CLR Memory(w3wp)\\Allocated Bytes/sec";
        counter = ".NET CLR Memory(w3wp)\\Finalization Survivors";
        counter = ".NET CLR Memory(w3wp)\\Gen 0 heap size";
        counter = ".NET CLR Memory(w3wp)\\Gen 1 heap size";
        counter = ".NET CLR Memory(w3wp)\\Gen 2 heap size";
        counter = ".NET CLR Memory(w3wp)\\Large Object Heap size";
        counter = ".NET CLR Memory(w3wp)\\Large Object Heap size";
        counter = ".NET CLR Memory(w3wp)\\# of Pinned Objects";
        counter = ".NET CLR Security(w3wp)\\% Time in RT checks";
        counter = ".NET CLR Security(w3wp)\\Stack Walk Depth";
        counter = ".NET CLR Security(w3wp)\\Total Runtime Checks";
        counter = "ASP.NET Applications(__Total__)\\Cache Total Entries";
        counter = "ASP.NET Applications(__Total__)\\Cache Total Hit Ratio";
        counter = "ASP.NET Applications(__Total__)\\Cache Total Turnover Rate";
        counter = "ASP.NET Applications(__Total__)\\Output Cache Entries";
        counter = "ASP.NET Applications(__Total__)\\Output Cache Hits";
        counter = "ASP.NET Applications(__Total__)\\Output Cache Hit Ratio";
        counter = "ASP.NET Applications(__Total__)\\Output Cache Turnover Rate";
        counter = "ASP.NET Applications(__Total__)\\Compilations Total";
        counter = "ASP.NET Applications(__Total__)\\Errors Total/Sec";
        counter = "ASP.NET Applications(__Total__)\\Pipeline Instance Count";
        counter = "ASP.NET Applications(__Total__)\\Requests Executing";
        counter = "ASP.NET Applications(__Total__)\\Requests in Application Queue";
        counter = "ASP.NET Applications(__Total__)\\Requests/Sec";
        counter = "ASP.NET\\Application Restarts";
        counter = "ASP.NET\\Request Wait Time";
        counter = "ASP.NET\\Requests Current";
        counter = "ASP.NET\\Requests Queued";
        counter = "ASP.NET\\Requests Rejected";
        counter = "Memory\\Available MBytes";
        counter = "Memory\\Cache Faults/sec";
        counter = "Memory\\Demand Zero Faults/sec";
        counter = "Memory\\Page Faults/sec";
        counter = "Memory\\Pages/sec";
        counter = "Memory\\Transition Faults/sec";
        counter = "Memory\\Pool Nonpaged Bytes";
        counter = "Memory\\Pool Paged Bytes";
        counter = "PhysicalDisk(_Total)\\% Disk Time";
        counter = "PhysicalDisk(_Total)\\% Disk Read Time";
        counter = "PhysicalDisk(_Total)\\% Disk Write Time";
        counter = "Process(w3wp)\\% Processor Time";
        counter = "Process(w3wp)\\Handle Count";
        counter = "Process(w3wp)\\ID Process";
        counter = "Process(w3wp)\\Private Bytes";
        counter = "Process(w3wp)\\Thread Count";
        counter = "Process(w3wp)\\Virtual Bytes";
        counter = "Process(w3wp)\\Working Set";
        counter = "Processor(_Total)\\% Interrupt Time";
        counter = "Processor(_Total)\\% Privileged Time";
        counter = "Processor(_Total)\\% Processor Time";
        counter = "Processor(_Total)\\% User Time";
        counter = "System\\Context Switches/sec";
        counter = "System\\System Calls/sec";
        counter = "Web Service(_Total)\\Get Requests/sec";
        counter = "Web Service(_Total)\\Post Requests/sec";
        counter = "Web Service(_Total)\\Connection Attempts/sec";
        counter = "Web Service(_Total)\\Current Connections";
        counter = "Web Service(_Total)\\ISAPI Extension Requests/sec";
        counter = "Web Service Cache\\URI Cache Hits %";
        counter = "Web Service Cache\\Kernel: URI Cache Hits %";
        counter = "Web Service Cache\\File Cache Hits %";
    }
}

PAL

Performance Analysis of Logs (PAL) is a tool by Microsoft Premier Field Engineer, Clint Huffman. I haven’t had a chance to preview this tool yet, so I don’t want to speculate. It does seem to have a powerful feature set and following, so it’s definitely worth a look if you need a more comprehensive view.

Step 1: State the Objective

We want to ensure that the basic ASP.NET pipeline does not regress from one release to the next.  We use server throughput (requests per second) as our performance metric.  We want throughput to be on-par or better than the previous release (aka, baseline).

Step 2: Create the Scenario

We have a basic IIS website with the following ASPX page:

<html>
    <%=”Hello, World!” %>
</html>

Step 3: Run the Scenario

Most of our scenarios use the wcat load tool to test the web server.  You can download wcat here.

In order to detect regressions we need to have reproducible results.  Here are some of the ways that we reduce variability:

• Eliminate the variables.  Since I am measuring ASP.NET performance, .NET should be my one variable.  Other factors such as hardware, OS, IIS, wcat version and the scenario itself should remain constant.
• Use a private network.  Eliminate unnecessary network traffic to the web server which could affect results.
• Minimize server processes.  Again, we want to avoid competition for server resources.  We usually disable Windows Firewall and run the wcat controller and clients from separate machines.
• Measure warm.  Since we are testing throughput and not startup, we can reduce the variability by adding a warm-up period.  This will reduce JIT compilation and population of caches which could cause more variance.
• Max the CPU.  Server load and system resources factor into the equation as well.  We try to keep this near-constant by applying a full load during testing.  Our goal is to achieve at least 90% CPU usage during our runs.

We quantify our test variance (aka, noise) by running multiple iterations.  We ignore the first iteration which we’ve found to have greater variance, and calculate the average and standard deviation for the remaining three.  Standard deviation is our noise indicator.

Step 4: Analyze the Results

After doing our baseline and test runs, we should have results like the following:

These are the columns:

• Baseline, Result – Requests per second average for the last 3 iterations of the run
• Diff – Percentage of how far off the Result is from the Baseline
• Baseline StdDev, Result StdDev – Percentage of variance between the run iterations
• Pass / Fail – Whether the Diff is above our threshold.

Based on our experience we’ve set our threshold at 5%.  We define failures as runs that are more than 5% regressed.  Runs that show >5% improvement should also be investigated in order to understand and validate the cause.

We consider a run noisy if the standard deviation exceeds a threshold, which we also set at 5%.  If a failed run is noisy, we throw out one more iteration to see if the Diff and StdDev improve.  If they do we may ignore the failure and wait for the next run.  If the runs continue to be noisy, then the scenario should be investigated in order to further reduce the variance.

Step 5: Investigate Regressions

As the saying goes, “Measure Early, Measure Often”.  The less changes there are between your runs, the easier it will be to track down the cause of your regressions.  This is why the ASP.NET performance lab runs daily with builds from multiple branches.  Often times I’m able to quickly identify a regression simply by looking at the source control history.

Another way to quickly diagnose regressions is to enable performance counters or other non-invasive tracing which could help identify the cause.  We always save the wcat logs with our performance counters and other useful information such as throughput, working set, percent cpu usage, and HTTP responses.  The HTTP responses can rule out test failures, while the other diagnostics (combined with source control) can help narrow down the cause.

Finally, if a quick diagnosis is not possible, find a good profiler.  Some of the tools we use are the Visual Studio (F1) profiler, CLR Profiler, and XPerf.  I hope to demonstrate some of these in future posts.

https://blogs.msdn.com/b/web_performance_blog/

I hope that our blog becomes a valuable resource for you and I welcome your feedback.  Happy reading!

-Christy Henriksson, ASP.NET Performance Champ

A few updates were made to the TypeDescriptor classes to support multi-targeting:

1. TypeDescriptor.InterfaceType

With the different VS project types came slightly different multi-targeting solutions.  Windows and Web Forms attached the TargetFrameworkProvider to control instances on the design surface.  Workflow projects took a more global solution: attaching the provider to all types.  However, this broke down when trying to attach the provider to interfaces since they don’t have a common parent.

The new InterfaceType property allowed us to attach a TypeDescriptionProvider to all interfaces.  This is similar to ComObjectType which allows you to attach to all COM types.

2. Performance improvements to TypeDescriptor.Refresh and TypeDescriptor.AddProvider

Attaching providers was slow… VERY slow.  Never before had we attached so many TypeDescriptionProviders at design time – it was a true stress test.

When attaching a TypeDescriptionProvider to a type, any derived type or instance of that type also gets the custom provider.  This results in a complex chaining of providers, each of which may have its own cache.  When providers are added or removed, we must ensure that the cache for providers higher in the chain are refreshed.  In .NET4 we optimized TypeDescriptor to avoid some cache refreshes that weren’t necessary.

3. TypeDescriptionProvider.GetRuntimeType

Previously, TypeDescriptionProvider.GetReflectionType always returned types that were compatible with runtime types.  Sometimes they were runtime types; and sometimes they were just imposters which redirected CLR APIs to the real runtime type while modifying certain reflection data.  I’m glossing over things, but you get the idea.

In .NET4, the TargetFrameworkProvider introduced new reflection types which were no longer compatible with runtime types.  Now that there were multiple target frameworks, types no longer fit into a single global “universe”.  Now reflection types came from a 4.0 universe, or a 3.5 universe, etc.  However, since the designer is still just running .NET4 we needed to add this new API to map types back into to the runtime.

4. TypeDescriptionProvider.IsSupportedType

The designer is always running the most recent framework, so we know that the target version is always equal or less than the runtime.  Assuming that there are no breaking changes, it means that there is always a runtime type that corresponds to the target reflection type.  However, when targeting a lower version there is not always a reflection type for a given runtime type (if the type was added after the target version).

Ideally TypeDescriptionProvider.GetReflectionType would just return null in this case.  However this allowance was never built into pre-existing code and documentation.  To avoid breaking this legacy contract we added TypeDescriptionProvider.IsSupportedType.  This method returns false if a runtime type does not exist for a given reflection type.

With non-dynamic objects, you can order on LINQ’s Expression.Property to reflect against an arbitrary property at runtime:

static void Main(string[] args) {
    var names = new[] {
        new { Name = "Joe" },
        new { Name = "Bob" },
        new { Name = "Tom" },
        new { Name = "Jack" },
    };
    Print(names);
    Print(Sort(names, "Name"));
}

static void Print(IEnumerable<dynamic> source) {
    foreach (var i in source.Select(s => s.Name)) Console.WriteLine(i);
    Console.WriteLine();
}

public static T[] Sort<T>(T[] source, string property) {
    var param = Expression.Parameter(typeof(T));
    var getter = Expression.Property(param, property);
    var lambda = Expression.Lambda<Func<T, dynamic>>(getter, param).Compile();
    return source.OrderBy(lambda).ToArray();
}

Of course, this doesn’t work for dynamic objects.  You need to sort on DynamicObject.TryGetMember with a get member binder that can invoke the property.  My first inclination was to write a helper method for getting the value with a binder, and using Expression.Call to invoke it:

class DynamicPerson : DynamicObject {
    private string _name;
    public DynamicPerson(string name) {
        _name = name;
    }
    public override bool TryGetMember(GetMemberBinder binder, out object result) {
        if (binder.Name == "Name") {
            result = _name;
            return true;
        }
        return base.TryGetMember(binder, out result);
    }
}

class MyBinder : GetMemberBinder {
    public MyBinder(string property) : base(property, ignoreCase:true) { }
    public override DynamicMetaObject FallbackGetMember(DynamicMetaObject target, DynamicMetaObject errorSuggestion) {
        throw new NotImplementedException();
    }
}


static void Main(string[] args) {
    var names = new dynamic[] {
        new DynamicPerson("Joe"),
        new DynamicPerson("Bob"),
        new DynamicPerson("Tom"),
        new DynamicPerson("Jack")
    };
    Print(names);
    Print(Sort(names, "Name"));
}

static DynamicObject[] Sort(IEnumerable source, string property) {
    var param = Expression.Parameter(typeof(DynamicObject), "o");
    var param2 = Expression.Constant(property);
    var getter = Expression.Call(typeof(Example2), "GetDynamicMember", new Type[0], param, param2);
    var lambda = Expression.Lambda<Func<DynamicObject, dynamic>>(getter, param).Compile();
    return source.Cast<DynamicObject>().OrderBy(lambda).ToArray();
}

static object GetDynamicMember(DynamicObject o, string property) {
    object result = null;
    if (o.TryGetMember(new MyBinder(property), out result)) {
        return result;
    }
    return null;
}

This looks ugly and also doesn’t work in partial trust since the lambda executed by LINQ is using your internals.  Fortunately there’s a simple fix: Expression.Dynamic.  Using this, LINQ can bind to the dynamic property using the runtime binder.

using Microsoft.CSharp.RuntimeBinder;

static DynamicObject[] Sort(IEnumerable source, string property) {
    var binder = Binder.GetMember(CSharpBinderFlags.None, property, typeof(Example3), new CSharpArgumentInfo[] {
        CSharpArgumentInfo.Create(CSharpArgumentInfoFlags.None, null) });
    var param = Expression.Parameter(typeof(DynamicObject), "o");
    var getter = Expression.Dynamic(binder, typeof(object), param);
    var lambda = Expression.Lambda<Func<DynamicObject, dynamic>>(getter, param).Compile();
    return source.Cast<DynamicObject>().OrderBy(lambda).ToArray();
}

UPDATE: There are some considerations you should take into account before extending DynamicObject