We also had the opportunity at the //BUILD/ conference, held in Anaheim, California in September to share with the developer community and ecosystem partners more of those performance features and optimizations.  Example areas included Virtualization Scalability, low latency networking performance, Power optimizations, and Web Scalability.  I am sharing below some of those sessions, so please have a look and enjoy!

Windows Server 8 Keynote

Windows Server 8 Overview

Windows Server Performance Improvements and optimizations

Reduce costs using new power efficiencies in Windows Server 8

Cheers,
Ahmed Talat
Windows Server Performance Team

Tom Basham

Virtualization Performance PM, Windows Fundamentals Team

Every VM in Hyper-V has a default NUMA node preference. Hyper-V uses this NUMA node preference when assigning physical memory to the VM and when scheduling the VM’s virtual processors. A VM runs best when the virtual processors and the memory that backs the VM are both on the same NUMA node, since such “remote” memory access is significantly slower than “local” access.

By default, the Hypervisor will assign the NUMA node preference every time the VM is run, and when choosing the NUMA node, it will find the best fit based on the resources the VM needs – preference will be given to a NUMA node that can support the VM’s memory on locally available node memory.

Obviously, the Hypervisor can’t predict the RAM or CPU usage needs of VMs that haven’t been created yet. It may have already distributed the existing VMs across the NUMA nodes, fragmenting the remaining NUMA resources such none of the nodes can’t fully satisfy the next VM to be created. Under these circumstances, you may want to place specific VMs on specific NUMA nodes to avoid the use of remote NUMA memory or alleviate CPU contention.

Consider the situation with the following 4 VMs:

·         Test-1, configured with 2 virtual processors and 3 GB of memory

·         Test-2, configured with 4 virtual processors and 4 GB of memory

·         Test-3, configured with 2 virtual processors and 3 GB of memory

·         Test- 4, configured with 1 virtual processors and 1 GB of memory

The partial screen shot below for Performance Monitor (PerfMon) show how the Hypervisor has nicely balanced the 4 virtual machines across 2 different NUMA nodes (see “Preferred NUMA Node Index”).

Notice how Hyper-V has balanced on the VMs across the two NUMA nodes, two VMs on node 0, and two VMs on node 1. In this case the memory requirements of each VM can be met by the local node memory (“Remote Physical Pages” is zero for all the VMs).

Now consider the case where I wanted test-2, my biggest, memory and CPU hungry VM to have its own dedicated NUMA node. How can this be achieved?

The trick is to set the NUMA node preference for the VM. In this way when the VM is started, Hyper-V will use the NUMA node preference for the VM.

Tony Voellm previously wrote a blog posting about setting and getting the NUMA node preferences for a VM (https://blogs.msdn.com/tvoellm/archive/2008/09/28/Looking-for-that-last-once-of-performance_3F00_-Then-try-affinitizing-your-VM-to-a-NUMA-node-.aspx). 

By using Tony’s script, I can explicitly set the preference appropriately to force test-2 to always be on first NUMA node and all the other machines (test-1, test-3, and test-4) on the second NUMA node (e.g. “numa.ps1 /set test-2 0”, “numa.ps1 /set test-1 1”, “numa.ps1 /set test-3 1”, “numa.ps1 /set test-4 1”).

Setting the preference take affect when the VM is started, so if you set the preference on an already running VM, the VM will need to be restarted for it to take effect. And the preference moves with the VM, so if the VM is moved to another host, either by Quick Migration, or Live Migration, the preference will take effect on the new host.

After restarting the VMs (so that my preference to took effect), you can see in the partial screen shot from PerfMon below that test-2 is only node 0, and the other VMs are on node 1.

Now whenever the test-2 VM is started, it will also be assigned to NUMA node 0, regardless of what is already running on node 0.

Finally a word of caution: be careful when choosing to set the NUMA node preference of a VM; if all the VMs are assigned to the same NUMA node, the VMs may behave poorly if they cannot get enough CPU time, or if there isn’t enough local memory on the NUMA node for the new VM, the VM won’t even start.

Tim Litton & Tom Basham

WIndows Performance Team 

New builds of Microsoft Windows are produced almost every day for internal development and testing. In order to detect performance regressions as soon as possible, those builds have to be evaluated and compared to their predecessors as well as previous public releases. A range of performance tests are used for these comparisons, including one called “FSCT” (which stands for File Server Capacity Tool).  FSCT was developed by the Windows Server Performance team as a tool capable of simulating multiple concurrent users accessing a file server using CIFS/SMB/SMB2. Its architecture allows for usage of “workloads”. Each workload consists of a set of basic scenarios (e.g. upload of a file using Explorer, upload of a file using xcopy, Microsoft Word opening a file, etc.), information on how often those scenarios should be performed by users, and what files on the file server under test should be used.  FSCT measures a file server’s capacity for a given workload.  This includes the highest throughput the server can sustain and the maximum number of active users the server can support.  FSCT also reports on server resource utilization, which can help identify performance bottlenecks such as network or storage bandwidth or CPU utilization.

The HomeFolders workload

The publically available version of “FSCT” comes with a single workload called HomeFolders. It was created by working with Microsoft IT administrators to capture Event Tracing for Windows (ETW) traces from real, heavily accessed file servers and their clients. The data included server side traces, client API traces and network traces, to capture the requests type distribution, parameters distribution, file set characteristics, and connection characteristics. Finally, the workload was created and tested to achieve approximately the same usage patterns found in the traces.

The workload development effort included creating the scenarios, creating the file sets and defining the number of runs per user per hour of each scenario. It is important to mention that “FSCT” and the HomeFolders workload were developed independently. The “FSCT” architecture allows for the creation of custom workloads and scenarios, as well as tweaking the existing workloads. However, as mentioned previously, only the HomeFolders is included in the initial release of “FSCT”.

You can find more information on FSCT at https://blogs.technet.com/josebda/archive/2009/09/16/file-server-capacity-tool-fsct-1-0-available-for-download.aspx

We wanted to inform you that an incredibly useful white paper was recently published from our friends in the High Speed Networking team titled "Networking Deployment Guide: Deploying High-Speed Networking Features." The document covers concepts, deployment instructions, and diagnostic monitoring tools for high speed networking features available in Windows Server 2008 and 2008 R2 such as TCP Chimney, Receive Side Scaling (RSS), netDMA, and virtual machine queue (VMQ).

If you've ever been curious about the difference between the Automatic and Enabled settings for TCP Chimney, or wanted to know which registry keys to set to enable VMQ interrupt coalesing (tip: coalescing may reduce the overhead of VMQ interrupt handling!), this document (direct link) is where to get the information.

Happy Configuring!

Matthew Robben

Program Manager, Windows Server Performance Team

If you’re the type of person who frequents our server performance blog, you’ve probably seen (or watched) this screen more than a few times:

This is, of course, the Performance tab in Windows Task Manager. While confusion over the meaning of the Physical Memory counters is a regular question we field on the perf team, today I’m going to explain how CPU utilization (referred to here as CPU Usage) may not mean what you would expect!

[Note: In the screenshot above, CPU utilization is shown as a percentage in the top left. The two graphs on the top right show a short history of CPU usage for two cores. Each core gets its own graph in Task Manager.]

CPU utilization is a key performance metric. It can be used to track CPU performance regressions or improvements, and is a useful datapoint for performance problem investigations. It is also fairly ubiquitous; it is reported in numerous places in the Windows family of operating systems, including Task Manager (taskmgr.exe), Resource Monitor (resmon.exe), and Performance Monitor (perfmon.exe).

The concept of CPU utilization used to be simple.  Assume you have a single core processor fixed at a frequency of 2.0 GHz. CPU utilization in this scenario is the percentage of time the processor spends doing work (as opposed to being idle). If this 2.0 GHz processor does 1 billion cycles worth of work in a second, it is 50% utilized for that second. Fairly straightforward.

Current processor technology is much more complex. A single processor package may contain multiple cores with dynamically changing frequencies, hardware multithreading, and shared caches. These technological advances can change the behavior of CPU utilization reporting mechanisms and increase the difficulty of performance analysis for developers, testers, and administrators. The goal of this post is to explain the subtleties of CPU utilization on modern hardware, and to give readers an understanding of which CPU utilization measurements can and cannot be compared during performance analysis.

CPU Utilization’s Uses

For those who are unaware, CPU utilization is typically used to track CPU performance regressions or improvements when running a specific piece of code. Say a company is working on a beta of their product called “Foo.” In the first test run of Foo a few weeks ago, they recorded an average CPU utilization of 25% while Foo was executing. However, in the latest build the average CPU utilization during the test run is measured at 75%. Sounds like something’s gone awry.

CPU utilization can also be used to investigate performance problems. We expect this type of scenario to become common as more developers use the Windows Performance Toolkit to assist in debugging applications. Say that Foo gets released for beta. One customer says that when Foo is running, their system becomes noticeably less responsive. That may be a tough bug to root cause. However, if the customer submits an XPerf trace, CPU utilization (and many other nifty metrics) can be viewed per process. If Foo.exe typically uses 25% CPU on their lab test machines, but the customer trace shows Foo.exe is using 99% of the CPU on their system, this could be indicative of a performance bug.

Finally, CPU utilization has important implications on other system performance characteristics, namely power consumption. Some may think the magnitude of CPU utilization is only important if you’re bottlenecked on CPU at 100%, but that’s not at all the case. Each additional % of CPU Utilization consumes a bit more juice from the outlet, which costs money. If you’re paying the electricity bill for the datacenter, you certainly care about that!

Before I go further, I want to call out a specific caveat for the more architecturally-aware folks. Earlier, I used the phrase “cycles worth of work”. I will avoid defining the exact meaning of “work” for a non-idle processor. That discussion can quickly become contentious. Metrics like Instructions Retired and Cycles per Instruction can be very architecture and instruction dependent and are not the focus of this discussion. Also, “work” may or may not include a plethora of activity, including floating point and integer computation, register moves, loads, stores, delays waiting for memory accesses and IO’s, etc.  It is virtually impossible for every piece of functionality on a processor to be utilized during any given cycle, which leads to arguments about how much functionality must participate during “work” cycles.

Now, a few definitions:

Processor Package: The physical unit that gets attached to the system motherboard, containing one or more processor cores. In this blog post “processor” and “processor package” are synonymous.

Processor Core: An individual processing unit that is capable of executing instructions and doing computational work. In this blog post, the terms “CPU” and “core” are intended to mean the same thing. A “Quad-Core” processor implies four cores, or CPU’s, per processor package.

Physical Core: Another name for an instance of a processor core.

Logical Core: A special subdivision of a physical core in systems supporting Symmetric Multi-Threading (SMT). A logical core shares some of its execution path with one or more other logical cores . For example, a processor that supports Intel’s Hyper-Threading technology will have two logical cores per physical core. A “quad-core, Hyper-Threaded” processor will have 8 logical cores and 4 physical cores.

Non Uniform Memory Access (NUMA) – a type of system topology with multiple memory controllers, each responsible for a discrete bank of physical memory. Requests to each memory bank on the system may take different amounts of time, depending on where the request originated and which memory controller services the request.

NUMA node: A topological grouping of a memory controller, associated CPU’s, and associated bank of physical memory on a NUMA system.

Hardware thread: A thread of code executing on a logical core.

Affinitization: The process of manually restricting a process or individual threads in a process to run on a specific core, package, or NUMA node.

Virtual Processor: An abstract representation of a physical CPU presented to a guest virtual machine.

Comparisons & Pitfalls

CPU utilization data is almost always useful. It is a piece of information that tells you something about system performance. The real problem comes when you try to put one piece of data in context by comparing it to another piece of data from a different system or a different test run. Not all CPU utilization measurements are comparable - even two measurements taken on the same make and model of processor. There are a few sources of potential error for folks using utilization for performance analysis; hardware features and configuration, OS features and configuration, and measurement tools can all affect validity of the comparison.

1.       Be wary of comparing CPU utilization from different processor makes, families, or models.

This seems obvious, but I mentioned a case study above where the Foo performance team got a performance trace back from a customer, and CPU utilization was very different from what was measured in the lab. The conclusion that 99% CPU utilization = a bug is not valid if processors are at all different, because you’re not comparing apples to apples. It can be a useful gut-check, but treat it as such.

Key takeaway #1: Processor of type A @ 100% utilization IS NOT EQUAL TO Processor of type B @ 100% utilization

2.       Resource sharing between physical cores may affect CPU utilization (for better or worse)

Single-core processors, especially on servers, are uncommon; multi-core chips are now the norm. This complicates a utilization metric for a few reasons. Most significantly, resource sharing between processor cores (logical and physical) in a package makes “utilization” a very hard-to-define concept. L3 caches are almost always shared amongst cores; L2 and L1 might also be shared. When resource sharing occurs, the net effect on performance is workload dependent. Applications that benefit from larger caches could suffer if cache space is shared between cores, but if your workload requires synchronization, it may be beneficial for all threads to be executing with shared cache. Cache misses and other cache effects on performance are not explicitly called out in the performance counter set.  So the reported utilization includes time spent waiting for cache or memory accesses, and this time can grow or shrink based on the amount and kind of resource sharing.  

Key takeaway #2: 2 HW threads on the same package @ 100% utilization IS NOT EQUAL TO 2 HW threads on different packages @ 100% utilization (for better or worse)

3.       Resource sharing between logical cores may affect CPU utilization (for better or worse)

Resource sharing also occurs in execution pipelines when SMT technologies like Intel’s Hyper-threading are present. Logical cores are not the same as physical cores - execution units may be shared between multiple logical cores. Windows considers each logical core a CPU, but seeing the term “Processor 1” in Windows does not imply that the corresponding silicon is a fully functioning, individual CPU.

Consider 2 logical cores sharing some silicon on their execution path. If one of the logical cores is idle, and the other is running at full bore, we have 100% CPU utilization for one logical core. Now consider when both logical cores are active and running full bore. Can we really achieve double the “work” of the previous example? The answer is heavily dependent on the workload characteristics and the interaction of the workload with the resources being shared. SMT is a feature that improves performance in many scenarios, but it makes evaluating performance metrics more…interesting. 

Key takeaway #3: 2 HW threads on the same logical core @ 100% utilization IS NOT EQUAL TO 2 HW threads on different logical cores @ 100% utilization (for better or worse)

4.       NUMA latencies may affect CPU utilization (for better or worse)

An increasing percentage of systems have a NUMA topology. NUMA and resource sharing together imply that system topology can have dramatic effects on overall application performance. Similar to the previous two pitfalls, NUMA effects on performance are workload dependent.

If you want to see which cores belong to which NUMA nodes, right click on a process in the “Processes” tab of Task Manager and click “set affinity…”. You should get a window similar to the one below, which shows the CPU-to-node mapping if a server is NUMA-based. Another way to get this information is to execute the “!NUMA” command in the Windows Debugger (windbg.exe).

 Key takeaway #4: 2 HW threads on the same NUMA node @ 100% utilization IS NOT EQUAL TO 2 HW threads on different NUMA nodes @ 100% utilization (for better or worse)

5.       Processor power management (PPM) may cause CPU utilization to appear artificially high

Power management features introduce more complexity to CPU utilization percentages. Processor power management (PPM) matches the CPU performance to demand by scaling the frequency and voltage of CPU’s. During low-intensity computational tasks like word processing, a core that nominally runs at 2.4 GHz rarely requires all 2.4 billion potential cycles per second. When fewer cycles are needed, the frequency can be scaled back, sometimes significantly (as low as 28% of maximum). This is very prevalent in the market - PPM is present on nearly every commodity processor shipped today (with the exception of some “low-power” processor SKUs), and Windows ships with PPM enabled by default in Vista, Windows 7, and Server 2008 / R2.

In environments where CPU frequency is dynamically changing (reminder: this is more likely than not), be very careful when interpreting the CPU utilization counter reported by Performance Monitor or any other current Windows monitoring tool. Utilization values are calculated based on the instantaneous (or possibly mean) operating frequency, not the maximum rated frequency.

Example: In a situation where your CPU is lightly utilized, Windows might reduce the operating frequency down to 50% or 28% of its maximum. When CPU utilization is calculated, Windows is using this reference point as the “maximum” utilization. If a CPU nominally rated at 2.0 GHz is running at 500 MHz, and all 500 million cycles available are used, the CPU utilization would be shown as 100%. Extending the example, a CPU that is 50% utilized at 28% of its maximum frequency is using approximately 14% of the maximum possible cycles during the time interval measured, but CPU utilization would appear in the performance counter as 50%, not 14%.

You can see instantaneous frequencies of CPUs in the “Performance Monitor” tool. See the “Processor Performance” object and select the “% of Maximum Frequency” counter.

[Side note related to Perfmon and power management: the “Processor Frequency” and “% of Maximum Frequency” counters are instantaneous samples, not averaged samples. Over a sample interval of one second, the frequency can change dozens of times. But the only frequency you’ll see is the instantaneous sample taken each second. Again, ETW or other more granular measurement tools should be used to obtain statistically better data for calculating utilization.]

Key takeaway #5: 2 HW threads @ 100% utilization and 50% of rated frequency IS NOT EQUAL TO 2 HW threads @ 100% utilization and 100% of rated frequency

Key takeaway #6: 4 HW threads @ 100% utilization and 50% of rated frequency IS NOT EQUAL TO 2 HW threads @ 100% utilization and 100% of rated frequency

6.       Special Perfmon counters should be used to obtain CPU utilization in virtualized environments

Virtualization introduces more complexity, because allocation of work to cores is done by the hypervisor rather than the guest OS. If you want to view CPU utilization information via Performance Monitor, specific hypervisor-aware performance counters should be used. In the root partition of a Windows Server running Hyper-V, the “Hypervisor Root Virtual Processor % Total Runtime” counter can be used to track CPU utilization for the Virtual Processors to which a VM is assigned. For deeper analysis of Hyper-V Performance Counters and Processor Utilization in virtualized scenarios, see blog posts here and here.

Key takeaway #7:  In a virtualized environment, unique Perfmon counters exposed by the hypervisor to the root partition should be used to get accurate CPU utilization information.

7.       “% Processor Time” Perfmon counter data may not be statistically significant for short test runs

For someone performing performance testing and analysis, the ability to log CPU utilization data over time is critical. A data collector set can be configured via logman.exe to log the “% Processor Time”counter in the “Processor Information” object for this purpose. Unfortunately, counters logged in this fashion have a relatively coarse granularity in terms of time intervals; the minimum interval is one second.  Relatively long sample sizes need to be taken to ensure statistical significance in the utilization data. If you need higher precision, then out-of-band Windows tools like XPerf in the Windows Performance Toolkit can measure and track CPU utilization with a finer time granularity using the Event Tracing for Windows (ETW) infrastructure.

Key takeaway #8: Perfmon is a good starting point for measuring utilization but it has several limitations that can make it less than optimal. Consider alternatives like XPerf in the Windows Performance Toolkit.

Best Practices for Performance Testing and Analysis Involving CPU Utilization

If you want to minimize the chances that hardware and OS features or measurement tools skew your utilization measurements, consider the following few steps:

1.       If you’re beginning to hunt down a performance problem or are attempting to optimize code for performance, start with the simplest configuration possible and add complexity back into the mix later.  

a.       Use the “High Performance” power policy in Windows or disable power management features in the BIOS to avoid processor frequency changes that can interfere with performance analysis.

b.      Turn off SMT, overclocking, and other processor technologies that can affect the interpretation of CPU utilization metrics.  

c.       Affinitize application threads to a core.  This will enhance repeatability and reduce run-to-run variations. Affinitization masks can be specified programmatically, from the command line, or can be set using the GUI in Task Manager..

d.      Do NOT continue to test or run in production using this configuration indefinitely. You should strive to test in out-of-box or planned production configuration, with all appropriate performance and power features enabled, whenever possible.

Key Takeaway #9:  When analyzing performance issues or features, start with as simple a system configuration as possible, but be sure to analyze the typical customer configuration at some point as well.

2.       Understand the system topology and where your application is running on the system in terms of cores, packages, and nodes when your application is not explicitly affinitized. Performance issues can suddenly appear in complex hardware topologies; ETW and XPerf in the Windows Performance Toolkit can help you to monitor this information.

a.       Rebooting will generally change where unaffinitized work is allocated to CPUs on a machine. This can make topology-related performance issues reproduce intermittently, increasing the difficulty to root cause and debug problems. Reboot and rerun tests several times, or explicitly affinitize to specific cores and nodes to help flush out any issues related to system topology. This does not mean that the final implementation is required to use thread affinity, or that affinity should be used to work around potential issues; it just improves repeatability and clarity when testing and debugging.

3.       Use the right performance sampling tools for the job. If your sample sets will cover a long period of time, Perfmon counters may be acceptable. ETW generally samples system state more frequently and is correspondingly more precise than Perfmon, making it effective even with shorter duration samples. Of course, there is a tradeoff - depending on the number of ETW “hooks” enabled, you may end up gathering significantly more data and your trace files may be large.

Finally, keep in mind that these problems are not isolated to the Windows operating system family. The increase in processor features and complexity over the past decade has made performance analysis, testing, and optimization a challenge on all platforms, regardless of OS or processor manufacturer.

And If you are comparing CPU utilization between two different test runs or systems, use the guidance in this post to double check that the comparison makes sense. Making valid comparisons means you’ll spend more of your time chasing valid performance issues.

Summary of Key Takeaways

Key takeaway #1: Processor of type A @ 100% utilization IS NOT EQUAL TO Processor of type B @ 100% utilization

Key takeaway #2: 2 HW threads on the same package @ 100% utilization IS NOT EQUAL TO 2 HW threads on different packages @ 100% utilization (for better or worse)

Key takeaway #3: 2 HW threads on the same logical core @ 100% utilization IS NOT EQUAL TO 2 HW threads on different logical cores @ 100% utilization (for better or worse)

Key takeaway #4: 2 HW threads on the same NUMA node @ 100% utilization IS NOT EQUAL TO 2 HW threads on different NUMA nodes @ 100% utilization (for better or worse)

Key takeaway #5: 2 HW threads @ 100% utilization and 50% of rated frequency IS NOT EQUAL TO 2 HW threads @ 100% utilization and 100% of rated frequency

Key takeaway #6: 4 HW threads @ 100% utilization and 50% of rated frequency IS NOT EQUAL TO 2 HW threads @ 100% utilization and 100% of rated frequency

Key takeaway #7: In a virtualized environment, unique Perfmon counters exposed by the hypervisor to the root partition should be used to get accurate CPU utilization information.

Key takeaway #8: Perfmon is a good starting point for measuring utilization but it has several limitations that can make it less than optimal. Consider alternatives like XPerf in the Windows Performance Toolkit.

Key Takeaway #9: When analyzing performance issues or features, start with as simple a system configuration as possible, but be sure to analyze the typical customer configuration at some point as well.

Feel free to reply with questions or additional (or alternative) perspectives, and good luck!

Matthew Robben

Program Manager

Windows Server Performance Team

And don't forget the Windows Server 2008 Turning Guide is still availiable and has recently been refreshed for SP2.

Windows Server Performance Team

Energy efficient hardware and software reduces operational costs and directly impacts an organization’s bottom line. We’re in the midst of developing Windows Server 2008 R2, and one of our goals for the product is to build a server operating system that is more power efficient than all of our previous releases. Furthermore, to help IT administrators better understand server power management and optimize their current Windows Server 2008 installations, we’re releasing a comprehensive white paper called “Power In, Dollars Out: Reducing the Flows in the Data Center” today. The white paper gives detailed explanations of many factors affecting server power efficiency, and contains a list of best practices for optimization.

One of the stated best practices is to properly configure Windows Server 2008’s power management features. According to the Green Grid, just turning on PPM features in the operating system can reduce power consumption by 20%. In Windows Server, this can be done simply by choosing the Balanced or Power Saver power policies (found in the Power Options applet in the Control Panel). Of course, PPM is a complicated technology, with many more toggles than a simple on/off switch. We’ve done quite a bit of work on the Windows Server processor power management (PPM) algorithms and parameters during R2 development. One of the results of this work was the development of a set of parameters that can boost power efficiency by up to 10% on standard benchmark workloads.

Good news - you don’t need to wait until R2 to deploy these new parameters on your servers. This blog post will describe PPM technology, explain the parameters involved, and show benchmark test results for the parameter changes on a commodity server. It will also give you a handy command-line walkthrough of the powercfg.exe commands necessary to implement these changes in your environment.

First, some context. Power management requires cooperation from the hardware and the operating system to work efficiently. For example, hardware might support low power states, but the operating system schedules computational work and is in the best position to decide when low power states can be leveraged. The Advanced Configuration and Power Interface (ACPI) defines an interface between the operating system and server hardware to be used for power management purposes.

ACPI Processor Performance States

The processor has traditionally consumed the most power in a server, which makes it a great candidate for power-efficiency optimizations. To add detail and flexibility for processor power management, ACPI defines a few sets of states for processors. Performance states, or P-states, are one such state that can be leveraged to increase power efficiency.

P-States

Processors can transition between multiple performance states, or P-states. P-states define incremental levels of processor performance, from P0 (most performant) to Pn (least performant). The ACPI specification does not specify a maximum number of P-states, so Pn is used to refer to the highest numbered, lowest performant P-state that a processor supports.

Each successively higher numbered P-state consumes less power than the previous P-state. Processors can dynamically switch between these states during operation to provide only as much computational capacity as is necessary, which saves power during periods of low usage.

Figure 1 below shows a hypothetical set of six P-states that would be available to a processor. Note that the maximum P-state (P0) has the highest frequency, while successively higher numbered P-states reduce in frequency. In this case, the minimum P-state is P5, so the terms Pn and P5 would be interchangeable.

Figure 1. Illustration of P-state number and corresponding frequency

Tuning P-State Parameters for Increased Power Efficiency

Windows Server contains a number of configurable P-state parameters. These can be used to finely tune the power/performance balance of Windows Server PPM. The defaults for these parameters are tuned to deliver excellent power efficiency for most systems and workloads out of the box. However, these are “safe” defaults. They balance performance and power efficiency. Default settings are shown in Table 1. Note that “P-state increase” in this context refers to a transition to a lower numbered, more performant P-state, whereas “P-state decrease” refers to a transition to a higher numbered, less performant P-state. Looking back to Figure 1, an increase would mean moving upward in the chart while a decrease would mean moving downward.

Table 1. Default P-State Parameter Settings in Windows Server 2008

¹The utilization percentage referenced here is not the same as the CPU usage counter in the Task Manager tool. Without going into more details, this setting is best optimized through empirical experimentation.

²A “state domain” is a dependency between different processor cores or packages on a server.  Often, processor designs require that if one core is at a particular performance or idle state, the other cores or packages in the domain must also be at the same state. The hardware notifies the operating system of this dependency by establishing a domain through the ACPI interface.

During Windows Server 2008 R2 development, our team determined a set of parameters that can boost energy efficiency with a very minor performance cost. Notice in Table 1 that the decrease time default is larger than the increase time default. This setting favors P-state increases over decreases. The default increase and decrease percentage settings of 30 and 50 percent, the default domain accounting policy, and the increase and decrease policy defaults favor P-state increases as well.

To tune the machine for more aggressive power savings, we suggest reducing the decrease time to 100 ms to match the increase time, changing the increase and decrease policies to favor P-state decrease, and switching the domain accounting policy to 0 (off). We left the increase and decrease percentages as their defaults to ensure that the system PPM parameters were not completely biased toward power savings and to reduce negative performance consequences. Table 2 summarizes these changes.

Important:  Modifying any of these parameters changes the behavior of performance state handling from the out-of-box experience. Before you deploy to production servers, validate the effects of any changes in a test environment.

Table 2. Default and New PPM Parameter Values

These parameters can only be set using the powercfg.exe command-line tool, which is installed by default to the Windows\System32 folder on Windows Server 2008. The commands to change the P-state settings by using powercfg.exe are given at the end of this post.

Energy Efficiency Analysis of P-State Settings

To test the efficiency of these new power settings (henceforth called “Aggressive” settings), we performed a set of benchmark runs on a four-socket quad-core server. Table 3 gives the system configuration.

Table 3. Four-Socket Quad-Core Server Configuration

We ran the SPECPower benchmark with both the default settings and the Aggressive power saving settings. Figure 2 and Figure 3 show the power usage and power efficiency across different workload levels. The Aggressive settings exhibit significant power efficiency over the default settings at a majority of the load levels. The maximum power saving is achieved at 60-percent workload level on this configuration with approximately 10-percent improvement in power efficiency when it is compared to the default setting. There is a negligible reduction in overall throughput at utilization levels above 97%.

Figure 2.  System power across varying SPECPower load levels

Figure 3.  System power efficiency across varying SPECPower load levels

These settings were tested on commodity servers with the SPECPower workload. Your particular hardware and workload might deliver different results. Please test any parameter changes before deploying in your production environment.

Changing P-State Parameters with Powercfg.exe

If you decide you want to deploy the new P-state parameter settings in your environment, you’ll first need to verify that your Windows Server 2008 installation is configured to use the Balanced power policy. Verify this by going to Power Options in the Control Panel.

Done? Next, you need to start a command prompt with administrator privileges. Get the binary dataset that represents the current Balanced AC power settings for P-states with the following command line (corrected from earlier versions of this post, thanks to Asmus for the heads up!):

>powercfg /getpossiblevalue sub_processor procperf 2

You should see the following:

Type: BINARY

Value: 640864000000A0860100E09304001E00000032000000

This value represents an encoded dataset of power policy parameters. The parameter values for this dataset can be shown with the decode command:

>powercfg /ppmperf /decode 640864000000A0860100E09304001E00000032000000

Verify that your power parameter values match the defaults shown below and in Table 1. If your parameter settings do not match these values, your Windows Server parameters may have already been reconfigured for optimal power efficiency in your environment.

Busy Adjust Threshold: 100

Time Check: 100

Increase Time: 100000

Decrease Time: 300000

Increase Percent: 30

Decrease Percent: 50

Domain Accounting Policy: 0

Increase Policy: 0

Decrease Policy: 1

Next, you need to change the parameter values to match the “Aggressive” settings described in this post. To do so, use the following command:

>powercfg /ppmperf /encode base 640864000000A0860100E09304001E00000032000000 /decreasetime 100000 /domainaccountingpolicy 1 /increasepolicy 1 /decreasepolicy 0

After executing this command, powercfg will print out a binary dataset representing the new values, like the one shown below.

640364000000A0860100A08601001E00000032000000

You need to apply the new dataset by using the setpossiblevalue command:

>powercfg /setpossiblevalue /sub_processor /procperf 2 binary 640364000000A0860100A08601001E00000032000000

Finally, use the setactive command to enable the new parameter set. No reboot is necessary for these parameters to take effect.

>powercfg /setactive scheme_balanced

If you want to restore the default setttings, use the setpossiblevalue command with the default dataset value (shown below), and follow it with a setactive command:

>powercfg /setpossiblevalue /sub_processor /procperf 2 binary 640864000000A0860100E09304001E00000032000000

>powercfg /setactive scheme_balanced

That’s it! You’ve taken your first step to increasing energy efficiency in your datacenter. As our white paper explains, there’s even more you can do. It’s a highly recommended read for cost-sensitive administrators.

Thanks for reading!

Matthew Robben

Program Manager

Windows Server Performance Team

Ahmed Talat

Performance Manager

Windows Server Performance Team

With the recent release of Hyper-V, customers are starting to ask us how to configure Hyper-V to get the best performance.  It’s generally recognized that there is overheard running a virtualized environment, but the question that really needs to be answered is how much?

With this in mind, I thought I’d share some of our recent testing of Hyper-V and how disk workloads perform when using Fixed or Dynamic VHDs.  The goal here is to provide some data that backs up the tuning guidance that can be here: https://www.microsoft.com/whdc/system/sysperf/Perf_tun_srv.mspx.

The following graph shows the relative performances for a number of different scenarios (with Dynamic VHD being the baseline).

Fixed VHD always performs better than a Dynamic VHD in most scenarios by roughly 10% to 15% with the exception of 4k writes, where Fixed VHD performs significantly better.

We ran 16 virtual machines when performing these tests (see “How We Tested” below) with the goal of evaluating how well Hyper-V performed in the server consolidation scenario.  Being able to consolidate a number of physical machines onto a single machine and have the virtual machines able to handle the load is a very important design goal for Hyper-V.

The exact result that a customer is going to see will depend on quite a few variables (e.g. how large and often the reads and writes are, how many outstanding I/O there can be at one time), so performing real-world testing is the best way to assess what impact virtualization will have.

Recently, QLogic  published a benchmark for I/O throughput for storage devices going through Windows Server 2008 Hyper-V (https://www.qlogic.com/promos/products/hyper-v.aspx) that closely matches the native performance, thus demonstrating Hyper-V’s ability to bring the advantages of virtualization to large-scale datacenters.

How We Tested

Hardware: DP DL580 G5, 16 x 2.4 GHz (Intel E7340), 16 GB RAM

Disk: HP P800, 25 spindles

Virtual Machine Setup: 16 Virtual machines, each running Windows Server 2008 Enterprise Edition (64 bit), 1 CPU, 796 MB RAM

Testing Software: We used IO Meter (https://www.iometer.org) to generate the workload for the I/O system, with a maximum number of 8 outstanding I/Os per virtual machine to a 100MB file.