Microsoft.com Operations

The Mystery of the Missing Process Performance Counter in Perfmon

MSCOM — Thu, 18 Dec 2008 16:57:00 GMT

One typical day I receive a request from a performance tester who is trying to collect some baseline production data about www.microsoft.com , the request seemed simple collect a couple of performance counter values during peak hours and post the data back to him . I started to collecting data from my machines. One of the counters that he was interested in was Process à Handle Count à w3wp. I could not find the process counter the counters what I had was as follows.

I quickly remembered about resource kit utility in win2k that was used to resolve this type of issue exctrlst.exe . Found a copy of it and copied it over to my 2k8 server not sure it would really work on 2k8 I decided to test my luck. I had a feeling that it was just a registry change so at the same time I started regmon on the server to capture registry changes. Launched the tool and selected the perfproc counter and check marked performance counters enabled.

Stopped regmon and filleted operation by regsetvalue Viola got it!!

Closed Perfmon and opened it again. Verified the perfmon to make sure the process counter showed up.

Now I know what it sets in registry so I did want to do this remotely. Quickly put through a for loop for the following command

Reg add HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\PerfProc\Performance /v "Disable Performance Counters" /t REG_DWORD /d 0

Eg) for /f %i in (server.txt) do Reg add \\%i\HKLM\SYSTEM\CurrentControlSet\Services\PerfProc\Performance /v "Disable Performance Counters" /t REG_DWORD /d 0

One more issue resolved

SE Tips and Tricks -- Save aTS Session Run Remote... Find Hyper V information

MSCOM — Fri, 11 Jul 2008 10:58:00 GMT

Editors Note: One the System Engineers that run the servers that host www.Microsoft.com told me that he wanted to start posting some of the methods and tips and tricks that they use when managing one of the most heavily visitied web site in the world. This post kicks off this series.

1) How to remotely find out if a machine has Hyper V is installed?

When you install the hyperv Role on windows 2008 it updates the BCD store (replacement of boot.ini in 2008 & Vista) of the machine with an entry "hypervisorlaunchtype" and sets it to auto. So you can remotely query the BCD store of a machine to find out if Hyper V is installed or not.

Here is a quick way from command prompt you can use the following command if you have psexec in the system path . You can use WINRS too, but psexec seems to be easy.

psexec \\machinename bcdedit | find "hyper"

e.g.) C:\>psexec \\hypervhost2k8 bcdedit | find "hyper"

PsExec v1.94 - Execute processes remotely
Copyright (C) 2001-2008 Mark Russinovich
Sysinternals - www.sysinternals.com

hypervisorlaunchtype Auto
bcdedit exited on hypervhost2k8 with error code 0.

There are other quick ways too by checking existence of vmbus.sys, hvboot.sys and using WMI to query BCD.

2) How to remotely find out which is the Physical host of a VM?

In HyperV language physical host is a machine that hosts the root partition and guest will be called a

child partition installed on top of HyperV. You can remotely query the registry value HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Virtual Machine\Guest\Parameters\hostname of the child partition to find out the machine name that hosts the root partition.

Here is command that can be run remotely

reg query "\\machinename\HKLM\SOFTWARE\Microsoft\Virtual Machine\Guest\Parameters" /V hostname

e.g.) C:\>reg query "\\hypervguest01\HKLM\SOFTWARE\Microsoft\Virtual Machine\Guest\Parameters" /v physicalHostName

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Virtual Machine\Guest\Parameters

PhysicalHostName REG_SZ hypervhost2k8

Microsoft.com Operations Performance Analysis of IIS 7.0/Windows Server 2008

MSCOM — Mon, 09 Jun 2008 14:12:00 GMT

Now that Windows Server 2008 has been released earlier this year many enterprises are assessing the value proposition of moving to the Windows Server 2008 and IIS 7.0. As an IT Pro or decision maker in this process you want real world working proof to make your decision clearer. Sure, the new file based configuration system and modular architecture of IIS 7.0 make administrating your web servers a whole lot more flexible. But how does it perform against its predecessor IIS 6.0 / Windows Server 2003?

Microsoft.com Operations has been an early and active adopters of IIS 7.0 / Windows Server 2008. We actually started running a single server on the www.microsoft.com site back in June 2005. We set out to answer this question. Is IIS 7.0 / Windows Server 2008 more performant then IIS 6.0 / Windows Server 2003? We performed a side by side performance analysis on the www.microsoft.com site which was taking live internet traffic during peak traffic hours.

During this performance assessment we found that the CPU usage was higher on Microsoft Windows Server 2008 RTM servers, which is driven by the higher number of RPS the servers were handling. As mentioned below the hardware load balancers are sending more traffic to the Windows Server 2008 RTM servers (31% more RPS) as they are completing the outstanding web requests faster than the W2K3 SP2 server. A good metric listed below is the “efficiency” or “cost” of the number of requests per CPU cycle (RPS / CPU Utilization %). Using this metric, IIS 7.0/W2K8 RTM is over 10% more efficient then IIS 6.0/WS03 SP2 in handling live web platform traffic for the www.microsoft.com site.

Listed below the graphical data results is the methodology used in this performance analysis.

Performance Metrics	W2K3 SP2	WS2K8 RTM	Change
RPS / CPU Utilization (%) = Requests per CPU Cycle	4.36	4.84	10.9%

	W2K3 SP2	W2K8 RTM	Change
Performance Indicators	A	B	(A-B)/A
CPU Utilization (%)	44.8%	52.8%	-17.9%
Web Service -Total Methods Requests/Sec “RPS”	194	255	31.4%
Web Service – Current Connections	280	294	5.0%
Load Balancing – Current Client Connections	116	116	0
.NET CLR Memory – % Time in GC	1.1%	2.5%

Server Efficiency (RPS/ CPU %) – Efficiency of serving live web platform traffic

W2K3 SP2 4.36 “requests per CPU cycle”

W2K8 RTM 4.84 ~ 10.9% increased efficiency

CPU Utilization (%)

W2K3 SP2 44.8%

W2K8 RTM 52.8% ~ 17.9% degradation (This is impacted by the increased RPS the W2K8 servers are handling)

Web Service – Total Methods Requests/Sec (RPS)

W2K3 SP2 194

W2K8 RTM 255 ~ 31.4% more traffic is being sent to the W2K8 RTM servers

Web Service – Current Connections

W2K3 SP2 280

W2K8 RTM 294 ~ 5% increase

Load Balancing – Current Client Connections

W2K3 SP2 116

W2K8 RTM 116 Equal – as the hardware load balancer maintains the same amount of outstanding open client connections.

.NET CLR Memory – % Time in GC

W2K3 SP2 1.1%

W2K8 RTM 2.5% No significant degradation in “Time in GC”

Performance Overview and methodology:

Site: www.microsoft.com

During this assessment the live www.microsoft.com site was comprised on 80 servers. There were four load balanced clusters of twenty servers each, located in multiple datacenters. This performance analysis was developed by using a single twenty server cluster on the www.microsoft.com site. The cluster is comprised of 19 Windows Server 2008 RTM servers (W2K8 RTM) and a single Windows Server 2003 SP2 (W2K3 SP2) server all taking live internet facing traffic. This Lone Windows Server 2003 has become quite famous. In fact, he actually has his own blog located at https://blogs.technet.com/windowsserver/pages/about-lone-server.aspx

Hardware:

Model: HP DL585 G1 (4 dual-core CPUs)

RAM: 32GB

OS:

Windows Server 2008 RTM (Build: 6.0.6001.18000) Enterprise version x64

Load Balancing:

We are using a hardware load balancing solution. The load balancing algorithm we are using is based on “Least Current Client Connections” to each load balanced member server of the cluster (not round robin, or other any other load balancing algorithms). The hardware load balancer will maintain the same number of current client connections to each member of the cluster. So if a W2K8 server is completing web requests faster than a W2K3 server, the load balancer will send more traffic to the server W2K8 RTM server.

Methodology:

We gathered performance monitor logs for 72 hours during peak traffic (Monday - Wednesday) for a cluster of www.microsoft.com web servers. In all the charts above the W2K8 RTM data is based on an average of all the W2K8 RTM servers in a single cluster. While the W2K3 SP2 data is based on the single (Lone) server that was in rotation.

New IIS 7.0 White Paper…Internet Information Services 7.0 in Windows Server 2008 Provides an Improved Web and Application Platform

MSCOM — Fri, 25 Apr 2008 12:20:00 GMT

This paper discusses the management, security, performance and extensibility improvements when Windows Server 2008 is deployed to host and manage applications and services that run on the server and/or over the Web. Check it out on this TechNet link!

Modern .NET Development and The Joy of Simple LINQ to SQL

MSCOM — Wed, 16 Apr 2008 11:51:00 GMT

Introduction – Modern .NET Development

It took me a while to realize it, but Microsoft .NET application development has significantly evolved in the past year and a half. This is not your dad’s .NET. The following releases have added many considerable advances in .NET application development technologies:

Microsoft Product	Release Date	Major Features
.NET Framework version 3.0	November 11th, 2006	WPF, WCF
Silverlight version 1.0	September 5th, 2007	XAML rendering for web applications, JavaScript API
.NET Framework version 3.5	November 19th, 2007	Language integrated query (LINQ)
Visual Studio 2008	November 19th, 2007	IDE for all things .NET
Silverlight version 2.0	Beta 1 on March 5, 2008, RTM late summer 2008	Managed CLR, C#, .NET Framework, greater parity with WPF XAML
Expression Blend	v1 May 2007, v2 beta March 2008	XAML user experience (UX) IDE

If you fancy yourself an adept .NET developer, you had better be ramped up on these new technologies.

Today, if you are considering user interface/experience development, you should consider leveraging the rich UX elements and platform independence offered by Silverlight, and the power of authoring with the Expression Studio. If you are designing some middleware, you should consider developing software services using Windows Communication Foundation (WCF) technology. For coding backend data access it is now wise to evaluate LINQ to Entities or LINQ to SQL techniques. And before you write any C# foreach loop on an array or other collection, you might be able to write it better using LINQ to Objects techniques instead.

The bottom line here is that .NET developers have many new tools available to leverage for developing just about any application. Using these new techniques should eventually accelerate the delivery of applications that look better and are better-built. I say “eventually” because these new tricks will take some ramp up time to learn and master. And first, the developer needs to be aware of them and know how to judiciously leverage them.

However, we are talking about a pretty big load of new stuff to digest here. So let us discuss just one example of one these new tricks: LINQ to SQL. The primary purpose of rest of this article is to illustrate one simple example application of LINQ to SQL.

Simple LINQ to SQL

A couple of months ago, I was tasked with developing an internal application that would periodically transfer some data from a SQL database into a monitoring system. The SQL database was a Hewlett Packard Systems Insight Manager installation. The monitoring system is a Microsoft Systems Center Operations Manager (SCOM) 2007 installation, accessed via the System Center Operations Manager 2007 SDK, and the Microsoft.EnterpriseManagement Namespace.

LINQ to SQL is well-suited for this situation as it helps avoid changing either the source or destination installations, and with a minimal “footprint” of moving parts. With a single stand-alone console application, I was able to get this job done, and with a not-too-kludgey implementation. I developed a single, modest-length class file to do the whole thing, with no hard-coded transact SQL code!

I won’t drag you through my entire solution. I pared down the code to illustrate just the LINQ to SQL mechanics, as run against that Hewlett Packard Systems Insight Manager installation:

using System;

using System.Data.Linq;

using System.Data.Linq.Mapping;

using System.Data.Linq;

namespace LinqLite

{

class Program

{

static void Main(string[] args)

{

DataContext db = new DataContext("YOUR HPSIM SQL SERVER CONNECTION STRING GOES HERE");

Table<CIM_Chassis> TableCIM_Chassis = db.GetTable<CIM_Chassis>();

Table<Devices> TableDevices = db.GetTable<Devices>();

Table<Notices> TableNotices = db.GetTable<Notices>();

var LinqQuery = from c in TableCIM_Chassis

join d in TableDevices on c.NodeID equals d.DeviceKey

join n in TableNotices on d.DeviceKey equals n.DeviceKey

where n.NoticeId > 35000 //TODO: make this a LastIdProcessed variable

orderby n.NoticeId

select new { n.NoticeId, n.NoticeSeverity, n.Generated, n.Comments, d.DeviceKey, d.ProductName, d.Name };

foreach (var qdata in LinqQuery)

Console.WriteLine(string.Format("{0}\t{1}\t{2}\t{3}\t{4}\t{5}\t{6}"

, qdata.NoticeId

, qdata.NoticeSeverity

, qdata.Generated

, qdata.Comments

, qdata.DeviceKey

, qdata.ProductName

, qdata.Name

));

Console.ReadLine();

}

[Table]

class CIM_Chassis

{

[Column] public long NodeID = 0;

}

[Table]

class Devices

{

[Column] public int DeviceKey = 0;

[Column] public string ProductName = string.Empty;

[Column] public string Name = string.Empty;

}

[Table]

class Notices

{

[Column] public int NoticeId = 0;

[Column] public int NoticeSeverity = 0;

[Column] public long Generated = 0;

[Column] public string Comments = string.Empty;

[Column] public int DeviceKey = 0;

}

I call this solution simple LINQ to SQL primarily because it is all done in one code file, and does not involve using the fancy Visual Studio 2008 Object Relational Designer (O/R Designer). When you use the O/R Designer, you get a handy GUI for assembling some classes that map to the SQL data. You can easily end up with several auto-generated class files with many code stubs, just in case you might need them. You use these classes in your LINQ queries to access the data. But if you are doing simple read-only selections, you can hand-code the classes, make them lean, and place them in-line in the code, like I did:

[Table]

class CIM_Chassis

{

[Column] public long NodeID = 0;

}

[Table]

class Devices

{

[Column] public int DeviceKey = 0;

[Column] public string ProductName = string.Empty;

[Column] public string Name = string.Empty;

}

[Table]

class Notices

{

[Column] public int NoticeId = 0;

[Column] public int NoticeSeverity = 0;

[Column] public long Generated = 0;

[Column] public string Comments = string.Empty;

[Column] public int DeviceKey = 0;

}

These class definitions only list the fields that I am interested in, and not the entire tables. Another simple part of my example is the brief “using” list:

using System;

using System.Data.Linq;

using System.Data.Linq.Mapping;

using System.Data.Linq;

Hopefully my core LINQ selection code is not too hard to decipher – seven fields selected from two joined tables, with a third table joined in to filter the data to only notices for “chassis” devices. First, a DataContext is created to set up a connection the SQL Server database. Next, references are made to the table classes using DataContext.GetTable(TEntity) Generic Method:

Table<CIM_Chassis> TableCIM_Chassis = db.GetTable<CIM_Chassis>();

Table<Devices> TableDevices = db.GetTable<Devices>();

Table<Notices> TableNotices = db.GetTable<Notices>();

Then the LINQ query is defined, complete with joins, a where clause, and orderby:

var LinqQuery = from c in TableCIM_Chassis

join d in TableDevices on c.NodeID equals d.DeviceKey

join n in TableNotices on d.DeviceKey equals n.DeviceKey

where n.NoticeId > 35000 //TODO: make this a LastIdProcessed variable

orderby n.NoticeId

select new { n.NoticeId, n.NoticeSeverity, n.Generated, n.Comments, d.DeviceKey, d.ProductName, d.Name };

I put the “TODO” comment on the where clause to indicate that a LastIdProcessed variable could be persisted and recalled to use for filtering the notices data. Thus, you might fetch only new notices that have occurred after the last time this process was executed.

Finally, a foreach construct is employed to “harvest” and display the selected data:

foreach (var qdata in LinqQuery)

Console.WriteLine(string.Format("{0}\t{1}\t{2}\t{3}\t{4}\t{5}\t{6}"

, qdata.NoticeId

, qdata.NoticeSeverity

, qdata.Generated

, qdata.Comments

, qdata.DeviceKey

, qdata.ProductName

, qdata.Name

));

The Console.ReadLine() at the very end merely serves as a way to pause program execution, so the displayed data can be viewed before the console window disappears.

MSDN’s “LINQ to SQL: .NET Language-Integrated Query for Relational Data” article does well to explain more about how this code works.

Conclusion

I hope this article has helped demystify LINQ for you, and has maybe inspired you to consider using this powerful .NET feature. I have a lot yet to learn about LINQ and the rest of those modern .NET technologies previously listed in that first table. But, I feel it is ripe time to dig into these things to advance the craft.

Microsoft.com Engineering Operations TechCenter NEW on TechNet.Microsoft.com

MSCOM — Fri, 11 Apr 2008 15:44:00 GMT

The Microsoft.com Operations Team is pleased to announce the launch of our new Microsoft.com Engineering Operations TechCenter. This site ids designed with the ITPro/Systems Engineer in mind. We get to work on some cutting edge technologies and our goal is to share our experiences, best practices, and findings with the ITPro/Systems Engineering community. Take a look at the key articles that are in this inaugural launch:

Introduction to the Microsoft.com Engineering Operations Team

Introduces the Microsoft.com Engineering Operations team, which is made up of systems engineers who support some of the largest and most heavily visited sites on the Internet, and describes the architectural infrastructure of some of the properties we support, including www.microsoft.com, Microsoft® Update, MSDN® and TechNet.

Migrating a Web Server from IIS 6.0 to IIS 7.0 by Using the Microsoft Web Deploy Tool

Get an inside look at how to install the new Microsoft Web Deploy Utility (MS Deploy) Tool as well as migrating your Web server or Web site from a computer that is running Information Services (IIS) version 6.0 on Microsoft® Windows Server™ 2003 to a computer that is running Internet Information Services (IIS) version 7.0 on Windows Server 2008.

How Microsoft.com Engineering Operations Delegates Configuration in IIS 7.0

Reduce administration overhead with IIS7.0 delegated configuration. Find out how the Microsoft.com Engineering Operations team uses delegated configuration in Internet Information Services (IIS) 7.0. Topics include performance and availability considerations, security, and delegating administrative control to Web sites.

Customizing IIS 7.0 by Installing and Configuring Roles, Role Services, Features, and Modules

Modular architecture in IIS7.0 enables administrators to reduce your web server footprint, by only installing features you need. Get a look under the hood and see how Microsoft.com Engineering Operations team installs role services, features and the associated Internet Information Services (IIS) version 7.0 modules that are required to run www.microsoft.com Web servers.

The Microsoft.com Engineering Operations TechCenter. will have also provide links to all of the existing content that we had produced over the last couple of years.

In the works are content about:

1. How MSCOM Ops Configure Application Pools in IIS7.0

2. How MSCOM Ops Adopts New Technology

Also in the planning stages are information about how we are using Hyper-V, as well as findings on how the new TCP/IP stack effects performance on SQL replication in high latency situations (i.e., replication from West Coast to East Coast).

We are very excited to get this project launched, it is the first TechCenter that is focused on the ITPro/System Engineer.

Microsoft.com Engineering Operations Forum Is Now LIVE!!

MSCOM — Wed, 12 Mar 2008 14:03:00 GMT

We are pleased to announce that the Microsoft.com Engineering Operations Forum is now live on TechNet. Come join us in discussions that are focused on Engineering Operations. Ask questions, give us your insights, help us build the community of folks that are engaged in real-world systems engineering. Log in with your Windows Live ID and be a part of the discussions.

Systems Engineering Architecture Consultation…”Help Us to Help You!”

MSCOM — Fri, 11 Jan 2008 16:06:00 GMT

MSCOM Operations get lots of requests from both internal and external customers on how we operate www.microsoft.com, Microsoft Update, and the Microsoft Download Center (just to name a few). Those customers are asking about a wide variety of topics that we may be able to help them with. Topics like our best practices we use in rolling out new technologies like Windows 2008 and IIS7.0 to our production web environment ,or how we use Peer-to Peer replication in our SQL topologies. Sometimes they just want to chew the fat with fellow System Engineers about the common problems that we all face as SEs.

These customer interactions are one of the best parts of working in MSCOM Operations. We truly learn as much in these customer engagements as (hopefully) our customers learn from us. To help us target these discussions we have created the following Infrastructure Architecture Straw Man. We provide this to customers that have pending engagements with us to try and get a sense of what their environments look like. This really helps us put the right Subject Matter Expert (SME) from our team in front of the customer with a good idea of what direction the discussion is likely to go.

Hopefully this will be of some use to you as well.

Infrastructure Architecture Questions/Topics Straw Man

This is intended to provide a list of topics to address with customers to assist in providing infrastructure architectural guidance.

First question(s) to ask is what are the business problems that need to be solved, the “must haves”. These then need to be prioritized. The infrastructure architecture can be very different depending on the result of this prioritization.

Next try to get a data flow diagram. Where are the calls coming from, what are the expected results from those calls and what components will need to be touched for each call. This will also help to flesh out the infrastructure.

Then try and get a high level diagram of the number of hosting locations/data centers; server clusters etc. which may or may not be known. Also try to determine upfront the target audience (public internet, corporate user’s intranet, partners extranet) and the approximate number of end users.

Finally get a list of requirements the customer thinks they need, then ensure that they understand the ramifications of those requirements. Example: Customer: “We need 5 nines availability.” Architect: “Great, please understand that equates to 2.59 sec of downtime per month.”

Below is a non prioritized list of requirement topics.

1. Availability

a. Defined as providing the required functional benefit to the users, not simply that “a server is up” metric.

Availability %	Downtime per year	Downtime per month*		Downtime per week
98%	7.30 days		14.4 hours	3.36 hours
99%	3.65 days		7.20 hours	1.68 hours
99.5%	1.83 days		3.60 hours	50.4 min
99.9%	8.76 hours		43.2 min	10.1 min
99.99%	52.6 min		4.32 min	1.01 min
99.999%	5.26 min		25.9 sec	6.05 sec
99.9999%	31.5 sec		2.59 sec	0.605 sec

b. Each one comes with an associated cost.

2. Performance

a. Web Site

i. Size of the page

ii. Number of calls to render the page

iii. Type of calls (http, https etc.)

b. Web Services

i. Will this be used to call SQL

c. Back End (SQL)

d. Scalability

i. 100’s or 1000’s or 1,000,000 Req/sec

e. Performance testing – what is the acceptable

3. Monitoring

a. What need to be monitored

i. Servers

ii. Applications

iii. Web services

iv. Connectivity

v. Events collected by monitoring should be “actionable”

4. Manageability

a. Directly related to Ops costs (people are the most expensive component)

b. Simple cookie-cutter vs. complex one-offs

5. Scale

a. Firewall/DMZ

b. Bandwidth

c. Server specific

i. Global Load Balancing

ii. Local Load Balancing

6. Content Distribution

a. Content delivery networks (CDN)

b. Caching strategies

c. Nature of content

i. Static

ii. Dynamic

iii. Make up

1. Jpegs

2. gifs

3. Flash

4. Silverlight

7. Performance

a. Geo Location (high latency links)

b. Server resource utilization under various loads (always good to load stress a server until it starts to fail to develop maximum capacity levels)

a. Running “cool” . i.e. CPU 40%, memory (depends on server role)

i. Determine load thresholds

b. Running “normal” i.e. CPU sustained 60-75%, memory (depends on server role)

i. Determine load thresholds

c. Running “Hot” i.e. CPU sustained 90%, memory (depends on server role)

i. Determine load thresholds

8. Security

a. Application security

b. Infrastructure security (DDoS, Intrusion detection, etc.)

i. Firewalls

ii. DMZ

iii. Router ACLs

9. Traffic Analysis

a. Volume of traffic

i. Requests/sec

b. Geo location of traffic

c. Nature of traffic

i. ASP.net, etc.

10. Geo-location of traffic

b. Type of client requests

i. For static content

ii. For Dymanic contest

11. Network (Frontend)

a. Latency concerns

b. Geo-redundancy needs

12. Network Backend

a. Latency concerns

b. Geo-redundancy needs

13. Virtualization

c. Clients/Application Servers

d. Storage

e. Applications

14. Identity Management/Authentication

a. Windows auth

b. Certificate based

c. Protecting Personal Identifiable Information (PII)

d. Cookies

IIS7.0 Lessons Learned In MSCOM Production Environment - On Demand Webcast

MSCOM — Tue, 04 Dec 2007 15:48:00 GMT

MSCOM has been running IIS7.0 in production since June 12, 2007. On November 27 we did a webcast entitled:

How Microsoft IT Is Providing an Improved Web and Applications Platform with Internet Information Services 7.0 in Windows Server 2008

In case you missed this live event we would like to invite you to use the link above to view this On Demand. Learn about some of the new benefits and features in Microsoft Internet Information Services (IIS) version 7.0 in the Windows Server 2008 operating system as experienced during the deployment of IIS7 as the Web and application platform for Microsoft.com. IIS7 in Windows Server 2008 provides management, security, performance, and extensibility improvements when Windows Server 2008 is deployed to host and manage applications and services that run on the server and/or over the Web.

Analyzing HTTP Traffic On Your IIS 7.0 Cluster

MSCOM — Thu, 29 Nov 2007 11:11:00 GMT

If you have ever run an enterprise web site you have probably received a panicked call from a customer or from your Tier 1 monitoring team that your site was responding slowly or throwing 500 errors and that you needed to resolve this issue ASAP. Maybe you are rolling out a new application or software across your site and want to perform side by side analysis of servers to ensure their health.

Over the past year we have rolled out multiple pre-release versions of Windows Server 2008 / IIS 7.0 onto the www.microsoft.com site. During this process we have had to monitor the overall health of the site. Which included being able to quickly identify content/application that were not behaving well on a server, cluster of servers or across the entire site. When running an enterprise of IIS servers it is invaluable to understand the traffic patterns of your site and to have the tools to help identify problem areas. There are many tools in the wild to monitor the health of your enterprise, but I will show you how to use Logparser to harvest your IIS logs and pinpoint a site problem. The goal of this blog is to help you identify issues or anomalous traffic patterns on your IIS web site.

For the purposes of this blog I will use an eight server cluster for our analysis. Sometimes tracking down issues on your site can be like trying to find a needle in a haystack. I like to start by looking at the overall health of the site or for the purposes of my examples, a cluster of servers. Next, I like to drill down into more detail on the server that looks unhealthy. In my examples I will drill down by URI and followed next by the associated window of time that the server/URI was unhealthy.

Here is where the fun begins! The first step is to perform a query by HTTP status codes for the cluster. Using Logparser I created a file called statuscodes_by_servers_in_cluster.sql and fed in the input IIS log files from all the IIS servers in the cluster which are called u_ex07111409.log. This result in an easy to read table showing you the summarized results by HTTP status code for each of the servers in your cluster.

Summarized HTTP status codes by server for a complete a complete cluster

logparser -rtp:-1 file:statuscodes_by_servers_in_cluster.sql?logfilename=u_ex07111409.log

sc-status Srv1 Srv2 Srv3 Srv4 Srv5 Srv6 Srv7 Srv8

--------- ------ ------ ------ ------ ------ ------ ------ ------

200 576313 585078 573956 574218 566388 599807 572982 589815

206 242 238 247 262 236 216 192 246

301 6148 6388 6378 6154 6140 6600 6293 6398

302 166817 168700 166471 165909 163386 172658 166764 170363

304 125493 126640 125908 126204 124139 130669 125855 129026

400 4 3 3 2 2 7 7 3

403 130 106 146 170 148 154 148 95

404 9023 9149 9835 8833 8811 9489 9250 9369

405 0 1 0 0 0 0 0 0

406 2 10 5 11 4 7 8 4

500 1361 342 363 337 332 323 340 351

501 18 22 21 25 31 27 34 30

I created the following Logparser file (statuscodes_by_servers_in_cluster.sql) to create the site by side server table of HTTP status code values.

statuscodes_by_servers_in_cluster.sql

SELECT

sc-status,

SUM(_Srv1) AS Srv1,

SUM(_Srv2) AS Srv2,

SUM(_Srv3) AS Srv3,

SUM(_Srv4) AS Srv4,

SUM(_Srv5) AS Srv5,

SUM(_Srv6) AS Srv6,

SUM(_Srv7) AS Srv7,

SUM(_Srv8) AS Srv8

USING

CASE s-computername WHEN 'ServerName1' THEN 1 ELSE 0 END AS _Srv1,

CASE s-computername WHEN 'ServerName2' THEN 1 ELSE 0 END AS _Srv2,

CASE s-computername WHEN 'ServerName3' THEN 1 ELSE 0 END AS _Srv3,

CASE s-computername WHEN 'ServerName4' THEN 1 ELSE 0 END AS _Srv4,

CASE s-computername WHEN 'ServerName5' THEN 1 ELSE 0 END AS _Srv5,

CASE s-computername WHEN 'ServerName6' THEN 1 ELSE 0 END AS _Srv6,

CASE s-computername WHEN 'ServerName7' THEN 1 ELSE 0 END AS _Srv7,

CASE s-computername WHEN 'ServerName8' THEN 1 ELSE 0 END AS _Srv8

FROM

\\ServerName1\logshare$\%logfilename%,

\\ServerName2\logshare$\%logfilename%,

\\ServerName3\logshare$\%logfilename%,

\\ServerName4\logshare$\%logfilename%,

\\ServerName5\logshare$\%logfilename%,

\\ServerName6\logshare$\%logfilename%,

\\ServerName7\logshare$\%logfilename%,

\\ServerName8\logshare$\%logfilename%

GROUP BY

sc-status

ORDER BY

sc-status

You will notice that the HTTP 500 status codes are rather high on one of our servers (SRV1). Now that we have identified which server looks unhealthy we can drill down and identify the problem application or URI which is causing the high level of 500’s server errors.

The next Logparser query I like to use is a spill by URI for the specific HTTP 500 status code on the cluster. Using Logparser I created a file called cs-uri-stem_by_servers_in_cluster.sql and fed in the same input IIS log files from all the IIS servers in the cluster called u_ex07111409.log.

URI spill for ‘500’ HTTP status code for a complete cluster:

logparser -rtp:-1 file:cs-uri-stem_by_servers_in_cluster.sql?logfilename=u_ex07111409.log

sc-status Srv1 Srv2 Srv3 Srv4 Srv5 Srv6 Srv7 Srv8 cs-uri-stem

--------- ---- ---- ---- ---- ---- ---- ---- ---- ----------------------------------------------

500 0 0 0 0 0 0 0 1 /brasil/pr/2002/ms_edu_minas.stm

500 0 0 0 1 0 0 0 0 /brasil/pr/2002/ms_games_br.stm

500 0 0 0 0 0 1 0 0 /brasil/technet/topicos/codered.stm

500 1 0 0 0 0 0 0 0 /brasil/windows2000/requisitos.stm

500 0 0 0 0 2 0 0 0 /egypt/

500 0 0 0 0 1 0 0 1 /esp

500 0 0 0 0 0 0 0 35 /fwquery/

500 28 9 10 35 21 20 23 31 /gbadapp/errorpages/error.aspx

500 267 160 184 260 270 270 274 275 /ibadapp/errorpages/error.aspx

500 1 0 0 0 0 0 0 0 /middleeast/press/2001/may/gold_cert.stm

500 0 0 0 1 0 0 0 0 /middleeast/press/2001/may/officexpa.stm

500 1060 7 17 38 35 32 38 0 /obadapp/errorpages/error.aspx

500 0 0 0 0 0 0 1 0 /projectserver

500 4 2 1 2 1 0 4 5 /shared/1/navigation.asmx/DisplayDlNavHtml

500 0 0 0 0 2 0 0 1 /shared/2/navigation.asmx/DisplayDlNavHtml

500 0 0 0 0 0 0 0 2 /xml/overview.asp

I created the following Logparser file (cs-uri-stem_by_servers_in_cluster.sql) to create the URI spill by server for the HTTP 500 status code.

cs-uri-stem_by_servers_in_cluster.sql

SELECT

sc-status,

SUM(_Srv1) AS Srv1,

SUM(_Srv2) AS Srv2,

SUM(_Srv3) AS Srv3,

SUM(_Srv4) AS Srv4,

SUM(_Srv5) AS Srv5,

SUM(_Srv6) AS Srv6,

SUM(_Srv7) AS Srv7,

SUM(_Srv8) AS Srv8,

cs-uri-stem