Clint Huffman's Windows Performance Blog

My blog has moved

Clint Huffman — Tue, 23 Jun 2009 16:45:10 GMT

It took awhile, but I was able get a new blog created using my email name “ClintH” versus “clint_huffman”, so I will no longer be posting blog entries to *this* blog. All of my blog entries will now be at: https://blogs.technet.com/clinth

I have two large blog entries there already which talk about my recent System Page Table Entries (PTE) adventures.

Please update your RSS feeds to https://blogs.technet.com/clinth

Thank you!

Do not use Task Manager in Hyper-V for Processor Measurements

Clint Huffman — Mon, 11 May 2009 23:21:17 GMT

I’m doing another blog post about the processor utilization in a Hyper-V environment because it seems like my last blog post on this subject was misinterpreted.

When Ewan Fairweather and I ran a few BizTalk servers as guest computers to 100% CPU. We brought up Task Manager and it showed very little, if any CPU usage. We thought to ourselves that either Hyper-V has the most efficient use of CPU utilization or there is something wrong. It turns out that the host computer (root partition) is considered just another virtual computer running on the hypervisor. This means that the CPU usage we see in Task Manager is only the CPU usage of the host computer only. Therefore, if you want to know how much CPU utilization your physical processors are really getting, then use the “\Hyper-V Hypervisor Logical Processor(*)\% Total Run Time”. The processor resource is the only resource affected in this way – meaning you can measure memory and network using the normal counters.

Read Tony Voellm’s blog (he is the Hyper-V Performance Lead) at:
https://blogs.msdn.com/tvoellm/

I was involved in testing performance of Hyper-V in a Microsoft BizTalk Server environment with Ewan Fairweather and we wrote this article:
https://msdn.microsoft.com/en-us/library/cc768535.aspx

Always use the Hyper-V Hypervisor performance counters first!

Windows Prefers Wired Connections

Clint Huffman — Sun, 19 Apr 2009 14:11:36 GMT

Awhile ago someone asked me about how to make their laptop prefer their wired internet connection versus their wireless when both are connected to the internet. The short answer is that Windows (Vista, 7, 2008, and I’m pretty sure XP and 2003 does as well) does this by default. The key here is the network interface metric. When you have more than one default gateway defined (indicated by a network destination of 0.0.0.0), then the internet bound packets go out the interface with the lowest metric. In the case of a tie (same metric), then the internet bound packets will go out the interface listed first.

Below is the routing table of my laptop when it is docked at my home network where I am connected to the internet through my wired docking station and by my wireless access point. My wired IP address is 192.168.1.12 and my wireless IP address is 192.168.1.13. Windows is aware of this condition and automatically prefers my wired connection by giving it a lower metric of 20 versus my wireless connection metric of 25. In this case, this is my Windows 7 (Beta 1) laptop.

You can, of course, permanently alter your metrics by editing your TCP/IP settings on your network adapter’s advanced settings. By default, it is set to Automatic metric allowing Windows to decide which metrics are best. Here is a screenshot.

In conclusion, Windows will automatically prefer your wired connections versus your wireless connections when both are connected to the internet. This is great for us road warriors.

Scott Landry adds:

You should know that Vista made a change to how we handle existing sockets – after plugging in, connections will not be switched over, you must re-establish the connection in order to make use of a wired connection. For example, if you’re downloading something from a website and realize that it would go faster by plugging in, you’d have to cancel and start over after plugging in. This is a change from XP and 2003. Here is a good reference:

The Cable Guy Strong and Weak Host Models https://technet.microsoft.com/en-us/magazine/2007.09.cableguy.aspx

Determining the Maximum Sizes of Kernel Memory Pools

Clint Huffman — Thu, 26 Mar 2009 22:23:34 GMT

While onsite with customers, I have found several more cases where the Windows Server 2003 32-bit is running dangerously low on kernel memory. Here lately it has been a lack of pool paged memory. In any case, you can use my earlier blog posts to estimate the kernel memory size. With that said, when you run into this issue, you need to really know what the real maximum sizes are. In this blog post, I will show you two relatively easy ways to get this information.

The memory pool usage in pool paged memory and pool non-paged memory should not exceed 80% of their respective maximums. The pool usage values are exposed the performance counters, “\Memory\Pool Paged Bytes” and “\Memory\Pool NonPaged Bytes”, but their maximum sizes can only be found by debugging the kernel. I know that debugging the kernel sounds very intimidating, but I assure you it is easy, but *dangerous* if you mess it up, so try this in a test environment first.

Getting Maximum Sizes for Pool Paged Bytes and Pool Non-Paged Bytes:

Install the Microsoft Debugging Tools for Windows. This is a free download from https://download.microsoft.com. Installation of these tools does not require a reboot.
Use Microsoft WinDBG or Process Explorer (both Microsoft tools):

WinDBG method:

Open WinDBG.

Ensure the symbol path is set: Click File, “Symbol File Path…”. Ensure the following path is in place without the double quotes: “SRV*C:\symbols*https://msdl.microsoft.com/download/symbols”. Click OK.
Click File, “Kernel Debug…”. Select the Local tab, then click OK.
Select “No” to any Workspace dialog boxes.
At the “lkd>” prompt, type “!vm”
“PagedPool Maximum” is the maximum size for the Pool Paged memory.
“NonPagedPool Max” is the maximum size for the Pool Non-Paged memory.

Process Explorer method: This is preferred if you need to get the maximum pool sizes often.

Download Process Explorer from:
https://technet.microsoft.com/en-us/sysinternals/bb896653.aspx or run directly from https://live.sysinternals.com.
No installation is required.
Run procexp.exe.
Ensure the symbol path is set: Click Options, “Configure Symbols…”
Set the Symbols path to (without the double quotes):
“SRV*C:\symbols*https://msdl.microsoft.com/download/symbols”
Set the Dbghelp.dll path to the installation directory of the Debugging Tools for Windows. Default location is: “C:\Program Files\Debugging Tools for Windows\dbghelp.dll” Click OK.
Click View, “System Information…”
“Paged Limit” is the maximum size for the Pool Paged Memory.
“Nonpaged Limit” is the maximum size for the Pool Non-Paged memory.
If you need to get the maximum pool sizes again, then just follow steps 2, 6 and 7.

Helpful PowerShell One-Liners

Clint Huffman — Tue, 24 Mar 2009 01:16:27 GMT

While doing a BizTalk Health check this week, the customer and I thought it would be cool to try to do as much as possible in an interactive PowerShell session to collect the data we needed. We created a lot of PowerShell one-liners and I thought I’d share it. These one-liners use WMI, so nearly all of them can be ran from a remote workstation, so this means that PowerShell is not needed on the remote BizTalk servers unless otherwise noted.

Be careful of word wrap in this email. All of these commands are *one-liners* – meaning they all go on one line ;-).

# List of BizTalk Servers. This is needed, so that the rest of the commands will execute on each of them per command.

$BizTalkServers = "BizTalkServer1","BizTalkServer2"

# Ping servers (latency check)

$BizTalkServers | foreach-object {"================";$_;"================";ping $_}

# Pathping (packet loss check)

$BizTalkServers | foreach-object {"================";$_;"================";pathping $_}

# Get Total Physical Memory

$BizTalkServers | ForEach-Object {"================";$_;"================";Get-wmiobject -Query "SELECT TotalPhysicalMemory FROM Win32_ComputerSystem" -ComputerName $_} | Format-List TotalPhysicalMemory

# Installed hotfixes (quick)

$BizTalkServers | ForEach-Object {"================";$_;"================";Get-wmiobject -Query "SELECT HotFixID,ServicePackInEffect FROM Win32_QuickFixEngineering" -ComputerName $_} | Format-list HotfixID, ServicePackInEffect

# NumberOfProcessors

$BizTalkServers | ForEach-Object {"================";$_;"================";Get-wmiobject -Query "SELECT NumberOfProcessors FROM Win32_ComputerSystem" -ComputerName $_} | Format-List NumberOfProcessors

# System Information

$BizTalkServers | ForEach-Object {"================";$_;"================";Get-wmiobject -Query "SELECT Caption, CSDVersion, MaxProcessMemorySize, PAEEnabled, ServicePackMajorVersion FROM Win32_OperatingSystem" -ComputerName $_} | Format-List Caption, ServicePackMajorVersion, CSDVersion, MaxProcessMemorySize, PAEEnabled

# Clock Synchronization Check

$BizTalkServers | ForEach-Object {"================";$_;"================";Get-wmiobject -Query "SELECT * FROM Win32_LocalTime" -ComputerName $_} | format-list Year, Month, Day, Hour, Minute, Second

# BizTalk Host Instance information (WMIC) (Must be ran locally) (WMIC is native to the operating system)

wmic /NAMESPACE:"\\root\MicrosoftBizTalkServer" PATH MSBTS_HostInstance GET HostName, HostType, Logon, NTGroupName, RunningServer

# BizTalk Host Instance information (PowerShell) (Must be ran locally) (PowerShell must be installed)

$BizTalkServers | ForEach-Object {"================";$_;"================";Get-wmiobject -Namespace "root\MicrosoftBizTalkServer" -Query "SELECT * FROM MSBTS_Host" -ComputerName $_} | Format-List HostName, HostType, Logon, NTGroupName, RunningServer

Microsoft BizTalk Server Explained in Simple Terms

Clint Huffman — Wed, 18 Mar 2009 20:52:29 GMT

At TechEd 2008, I manned the Microsoft BizTalk Server booth. Nearly every question I got was, “I’ve heard of BizTalk, but what does it do?”. If you are like me 3 years ago which is sounds like many people are, I wish someone would have just came out and given me a simple description of BizTalk. Now that I know BizTalk, here is the *simple* description/FAQ of what it is and why it is a great product.

What is BizTalk?

BizTalk is a message conversion system. You give it MessageA and it converts it to MessageB. For example, if MessageA is an EDI (Electronic Data Interchange) message, then BizTalk is able to easily read the message and convert it into just about any other message type such as an XML document (MessageB). Will it convert a Word document to a PDF document? It can, but it was really designed to allow businesses to easily communicate with each other using practically any message format and nearly any network protocol.

I can just code this in .NET why do I need BizTalk?

Sure, you can fire up Visual Studio and write this on your own, but did you think about guaranteed delivery, disaster recovery, tracking, troubleshooting, security, or authentication? What if the network protocol to receive and send these messages needs to change? How long would it take to change your custom application compared to a simple configuration change (no recompile necessary) in BizTalk. When you keep creating monolithic applications to handle your business to business (B2B) transactions, it becomes increasingly difficult to manage it and you find yourself trying to write an infrastructure to handle it. BizTalk provides all of this infrastructure for you. It has guaranteed delivery, standardized tracking, a business rules engine, redundancy, and much more.

I am not doing B2B. Is BizTalk still helpful?

Yes. Many people use BizTalk as an integration platform – meaning all of those old systems that you have that don’t communicate with each other can be bridged together with BizTalk. The marketing term for this is Solution Oriented Architecture (SOA). Also, many companies love BizTalk so much that all of their business logic internally in their company flows through it with the confidence of BizTalk’s ability to make nearly any business process a structured transaction.

BizTalk is huge, do I really need all of that infrastructure?

BizTalk is not for everyone. Again, you can certainly write your own .NET application to do very basic message conversion, but once your company matures and needs enterprise level messaging, reliability, and flexibility, then that is when you get BizTalk.

What is BizTalk really good at?

BizTalk is great when dealing with frequent changes in how you do business with other businesses. For example, if you deal with a lot of businesses that have different message requirements, frequent network protocol changes, then BizTalk is for you. Also, many customers really like how easy it is to map the schemas of messages. For example, FName in MessageA can be easily mapped to FirstName in MessageB regardless of how the messages are structured.

You keep talking about “messaging”. Does this mean that BizTalk does email?

The messaging we are talking about would be something like reading in a purchase order from CompanyA and sending the order to your fulfillment and shipping departments. BizTalk can also send email if you want it to, but it’s really designed to get businesses to talk to each other hence why it is called “Biz-Talk”.

Additional Information

One of my colleagues at Microsoft requested that I include information on some of the other features that BizTalk Server provides:

Multiple adapters provide support for most industry standard transports (FTP, HTTP, File, SOAP, WCF, SQL, etc.)
Multiple accelerators are available that support for industry standard document formats (HIPAA, EDI, SWIFT)
RFID Platform
High availability
Fault tolerance
Scalability
End to end message tracking (including message body tracking if needed.
Messaging subsystem accommodates transactional messaging, i.e. BizTalk can send to/receive from trading partners within the context of a distributed transaction as long as the adapter supports transactions.
XLANGs workflow engine
Business Activity Monitoring
Business Rule Engine (BRE)
Management functionality via Management console
Integration with MOM
Integration with Enterprise Single Sign-On

These features are available in BizTalk Server, which allows you to quickly integrate your business processes with other internal business processes and with external trading partners. Several of these features are described in further detail at https://www.microsoft.com/biztalk/en/us/capabilities.aspx.

Thanks to Trace Young (BizTalk Technical Writer) for this bit of information.

Bling.ps1 (a functional proof of concept for PAL v2.0)

Clint Huffman — Wed, 31 Dec 2008 18:15:24 GMT

Work on PAL v2.0 (PowerShell version) is coming along nicely. While working on it, I created a functional proof of concept PowerShell script called Bling.ps1 which will read a performance monitor log (BLG or CSV) and will create graphical charts for all of the counters in it. PAL v1.x only creates charts and stats for performance counters named in the threshold files, so this is a new feature. Oh, and BLING stands for BLg INto imaGes. :-)

Bling.ps1 v1.0 (Proof of Concept for PAL 2.0)
https://www.codeplex.com/PAL/Release/ProjectReleases.aspx?ReleaseId=21260

No installation is required, but dependent, free, products are required.

Required Products (free):
- PowerShell v1.0 or greater.
- Microsoft .NET Framework 3.5 Service Pack 1
- Microsoft Chart Controls for Microsoft .NET Framework 3.5

Syntax:
Bling.ps1 /LOG:[AbsolutePathToPerfmonLog[;AbsolutePathToPerfmonLog] /OUTPUTDIR:[AbsolutePathToOutputDirectory]

Note: Use full, absolute paths in each of the arguments.

Basic Example:
.\Bling.ps1 /LOG:C:\Users\clinth\Documents\SamplePerfmonLog.blg /OUTPUTDIR:C:\Users\clinth\Documents\Output

See the Readme.txt file for more details.

Note: Be sure to set PowerShell to unrestricted access by typing the following command in an elevated PowerShell command prompt:
Set-ExecutionPolicy Unrestricted

Special Thanks to Greg Varveris for his help with understanding the Microsoft Chart Controls for Microsoft .NET Framework 3.5.

All my posts are provided "AS IS" with no warranties, and confer no rights. For PFE Job Opportunities at Microsoft, please visit our website at: https://members.microsoft.com/careers/search/default.aspx - search for keyword “PFE”
“PFE: The best place to be at Microsoft”

Another RunAs Radio Show

Clint Huffman — Wed, 31 Dec 2008 18:14:02 GMT

A few weeks ago, I did another show on the internet radio talk show called Run As Radio at https://www.ranasradio.com. The session I did with Richard and Greg this time was on Windows Management Instrumentation (WMI). It should be up within the next few weeks. :-)

Also, I’m proposing to do another show on Windows Performance Tools. The various tools I use to diagnose root causes of performance issues in Windows. In the meantime, I’ll try to blog about them as I go.

I survived an internet radio talk show

Clint Huffman — Thu, 18 Dec 2008 00:20:23 GMT

About a month ago, I was on Run As Radio (www.runasradio.com) doing a session on the PAL tool. This was a lot of fun. Here is the link.

Clint Huffman Does Performance Analysis of Logs!https://www.runasradio.com/default.aspx?showNum=82

Also, a friend of mine, Shane Creamer, did an earlier session. It worked well because his session talks about core performance counters and the Vitals Signs workshop and mine talks about how to use the PAL tool to apply the rules and thresholds of the Vital Signs workshop.

Shane Creamer Goes Deep on Performance Monitor!https://www.runasradio.com/default.aspx?showNum=81

Also, Mark Minasi (one of my IT heroes) did a very humorous and very informative session on IPv6 and Vista networking. Very enjoyable session.

Mark Minasi on Networking in Windows Server 2008!
https://www.runasradio.com/default.aspx?showNum=65

Finally, I will be doing another session on Run As Radio this Friday (published a few weeks after) on Windows Management Instrumentation (WMI). :-)

BizTalk Admin Console Fails with DTC Failures

Clint Huffman — Mon, 15 Dec 2008 20:56:40 GMT

Relevant Environment Information:

Microsoft BizTalk Server 2006 R2 (32-bit)

Microsoft Windows Server 2003 R2 (32-bit)

Symptoms: While doing a BizTalk Health Check with a customer, we encountered errors with the Microsoft Distributed Transaction Coordinator (DTC) while using the BizTalk 2006 R2 Admin Console. I, unfortunately don’t have the exact error any longer, but it mentioned something about unable to contact the DTC service which is a very serious issue because BizTalk is heavily reliant on the DTC service. The odd thing is that the BizTalk services continued to operate normally, so only the admin console was effected by this.

Looking at the event logs, I found a few errors similar to this one:

Category: ENTSSO
Type: Warning
Source: Enterprise Single Sign-On
Event ID: 10532
Description: "Failed to retrieve master secrets. Verify that the master secret server name is correct and that it is available.
Secret Server Name:
Error Code: 0x800706BF, The remote procedure call failed and did not execute."

This error is also a very scary error to get because just like the DTC service, BizTalk is heavily reliant on the Enterprise Single Sign-On service (ENTSSO).

We called up Microsoft Customer Support Services (CSS) to help us with this. Yes, I am a Microsoft field support professional, but I don’t know everything. ;-)

CSS had us restart the DTC service on the BizTalk Servers, then restart the Microsoft ENTSSO Master Secret Server. After that, it was working again. No errors in the BizTalk Admin Console.

Cause: The DTC service was restarted and the ENTSSO Master Secret service lost contact with it.

Solution: Restart the DTC service, the restart the ENTSSO Master Secret Service.

Lesson Learned: Always restart the ENTSSO Master Secret Service after restarting the DTC service.

More Information: The Enterprise SSO Master Secret key is extremely important. If your ENTSSO service becomes corrupted, then you lose everything. There are certainly *many* more reasons why the ENTSSO Master Secret Service may fail, so always make sure you backup the Master Secret key using SSOManage.exe, backup the key to some place you can get to and where it is not on the local machine, and finally, ensure you remember the restore password.