Edit
  • Article

[Newsletters Archive ^] [< Volume 4, Number 1] [Volume 4, Number 3 >]

The Systems Internals Newsletter Volume 4, Number 2

http://www.sysinternals.com
Copyright (C) 2002 Mark Russinovich

August 12 - In this issue:

  1. EDITORIAL

  2. WHAT'S NEW AT SYSINTERNALS

    • TCPView v2.21
    • LiveKd v2.0
    • PsInfo v1.33|
    • PsList v1.21
    • BgInfo v3.0
    • PageDefrag v2.2
    • Regmon v4.34 for Win64/Itanium
    • Sysinternals in PC World Magazine
    • Sysinternals at Microsoft

  3. INTERNALS INFORMATION

    • Mark and David Solomon Teach Internals in London
    • Inside Windows NT Development
    • When a Change Doesn't Count
    • .NET Server RC1
    • Loads of WMI Scripts
    • Using BootVis to Profile the Boot Process

  4. WHAT'S COMING UP

    • A Quick Introduction to Crash Dump Analysis

SPONSOR: WINTERNALS SOFTWARE

The Sysinternals Newsletter is sponsored by Winternals Software, on the Web at http://www.winternals.com. Winternals Software is the leading developer and provider of advanced systems tools for Windows NT/2K/XP. Winternals Software products include FAT32 for Windows NT 4.0, NTFSDOS Professional Edition (a read/write NTFS driver for DOS), and Remote Recover. Winternals is proud to announce Defrag Manager version 2.0, the fastest, most thorough enterprise-defragmenter available. Now you can manage defragmentation schedules across your entire Windows enterprise from a simple MMC snapin - without even having to install any client software on your NT or Windows 2000 systems. Visit http://www.winternals.com/es for more information or to request a free 30-day trial version.

Hello everyone,

Welcome to the Sysinternals newsletter. The newsletter currently has 35,000 subscribers.

You have probably heard of AMD's upcoming Hammer processor (now formally announced as "Opteron"), to be released late this year. With Hammer, AMD is taking its war with Intel from the 32-bit field to the 64-bit field. I think Intel should be scared, and I'll tell you why.

The Itanium is Intel's entry into the 64-bit space and includes x86 binary compatibility as a way to ease migration. The Itanium has been in the marketplace for almost two years, but has failed to make inroads, even with HP-UX, Linux, Windows XP 64-bit Edition, and Windows Advanced Server 64-bit Edition supporting it. Industry estimates project Itaniums as having only sold a few thousand units and my guess is that the recently released Itanium II won't improve the rate of deployment.

What are the reasons for Itanium's depressing showing thus far? I believe that the main factor is that Itanium and Intel's entire 64-bit strategy is based on new RISC-type architecture, and with the exception of x86 binary compatibility bolted on to it, Itanium is just another 64-bit processor among Alpha, PA-RISC, Sparc64, and PowerPC. And although Itanium supports existing x86 32-bit applications, those applications suffer significant performance degradation when compared to native 64-bit applications. This means that you need to make sure that any applications that you want to run with reasonable performance on Itanium are ported to 64 bits.

Hammer, on the other hand, is an evolution that naturally extends the x86 architecture and instruction set to 64 bits. It's capable of running 32-bit applications and 64-bit applications side-by-side at virtually the same performance levels. In addition, you can boot Hammer into full 32-bit mode, making it appear to be a standard x86 system capable of running a 32-bit OS. Not only that, but Hammer supports x86 16-bit modes and can use standard 32-bit x86 BIOS code even when you boot into a 64-bit OS. This means that you can have a dual boot system with a 32-bit OS as one selection and a 64-bit OS as another.

I believe that AMD's approach offers a huge advantage over the direction that Intel took with Itanium. AMD can begin selling Hammer as soon as it's available to customers that might not be quite ready to go to 64 bits, or that have only one application that can take advantage of a 64-bit machine. 32-bit applications can take full advantage of the performance innovations that AMD puts into Hammer, and if AMD prices Hammer close to 32-bit CPU prices the purchase decision could be a no-brainer. This should allow AMD to infiltrate the marketplace with its 64-bit processors from the low end. Contrast this to Intel's high-end approach- a strategy that's likely to be far riskier and take much longer. I believe that Hammer gives AMD an opportunity to sneak in and become the de facto 64-bit platform.

What about the performance edge that Intel might claim the Itanium's "clean" Explicitly Parallel Instruction Computing (EPIC) architecture provides over AMD's x86-64? We'll have to wait and see, but AMD has thus far given Intel a run for its money in the 32-bit arena. And, if you look at how Itanium II compares to the best x86 processors (see www.spec.org) you'll find them pretty evenly matched on integer performance, with Itanium having a 50-75% lead in floating point. This is despite that fact that Itanium is roughly twice as expensive. For example, the low-end Itanium II system from Hewlett Packard costs $5865 and a Hewlett Packard Pentium IV system running at 2GHz in an equivalent configuration costs just $2385.

Where did AMD get the idea to extend the x86 architecture rather than define a new one with x86 compatibility as a feature? When Intel moved into the 32-bit world, it did so by extending the 16-bit x86 architecture and instruction set to make 32-bit version that could also run 16-bit applications. As a result, 32-bit x86 processors can run 16-bit and 32-bit operating systems. However, Intel didn't use this approach with their 64-bit strategy. This may be because in the mid-90's it saw the ceiling fast approaching for the x86 in terms of performance given the design's complexity, and felt a fresh design could go farther into the future. Today both Intel and AMD have proven this assumption wrong with their astonishing x86 clock speeds - Intel has demonstrated a Pentium IV running at 4 GHz. Intel could pay the price for their miscalculation.

Please pass the newsletter to friends you think might be interested in its content.

Thanks!

-Mark

WHAT'S NEW AT SYSINTERNALS

TCPVIEW V2.21

TCPView is an application that's been on the Sysinternals site for several years, but until the recent 2.0 update it wasn't much more than a GUI version of the Windows netstat command-line tool. This major revision brings a slew of powerful features, the most significant of which is the display on NT 4, Windows 2000 and Windows XP of the process that owns each TCP and UDP endpoint. You can also view a process' version data and executable path, as well as terminate rogue processes. A feature you'll find useful if you want to stop an application from using a particular port is the ability to close connections on the fly without terminating the process.

In the UI area the latest version introduces automatic refresh at selectable intervals and difference highlighting. Difference highlighting, which helps you to zoom in on activity as its happening, describes the color highlighting of new and deleted items, something I carried over from Process Explorer.

Download TCPView v2.21 at
http://www.sysinternals.com/ntw2k/source/tcpview.shtml

LIVEKD V2.0

One of the tools David Solomon and I included on the CD for our book Inside Windows 2000 (see http://www.sysinternals.com/insidew2k.shtml ) is LiveKd. LiveKd is a front end to the Microsoft kernel debuggers that enables you to run the debuggers to interact with a live system. Using a debugger to look at a system without having to use a second computer and a serial debugging connection makes exploring internals easy and can even serve as a troubleshooting aid. The first release of LiveKd only supported NT 4 and Windows 2000-LiveKd version 2.0 works on Windows XP.

Although Microsoft followed my lead and added live kernel debugging capabilities to Windows XP, a few debugging commands don't work when run on a live system, which is why I went to the trouble of updating LiveKd to work on Windows XP.

One example is the stack output from the !process or !thread commands. LiveKd shows the complete kernel stack, whereas using the built-in XP live debugging capability does not. Another example is the !stacks command, which you can use to see where threads are currently executing and where they might have started executing. The !stacks command is useful when you are debugging a problem that involves system threads and want determine their purpose - knowing the driver in which a thread is executing can be the clue you need to fix a problem.

Download the LiveKd v2.0 patch at
http://www.sysinternals.com/insidew2k.shtml
Note that you'll need a copy of LiveKd from the book CD to which to apply the patch.

PSINFO V1.33

PsInfo, one of the dozen or so tools that make up the PsTools suite of command-line, remotely capable, Windows NT/2000/XP administrative tools, continues to evolve based on user feedback. Version 1.33 adds the ability to list installed hotfixes and applications, as well as to report label, size and usage data for a computer's disk volumes.

Download PsInfo v1.33 at
http://www.sysinternals.com/ntw2k/freeware/psinfo.shtml
Download the entire PsTools package at
http://www.sysinternals.com/ntw2k/freeware/pstools.shtml

PSLIST V1.21

If you like using the command-line or have ever wished you could get a process list for a computer (local or remote) with CPU usage data then you'll be interested in the latest version of PsList, another member of the PsTools suite. PsList v1.2 introduces command-line switches that run it in a task-manager mode in a command window, updating at a configurable interval and displaying CPU usage. Another new switch has PsList use indenting to show process parent-child relationships, which is sometimes useful in understanding the purpose of processes you're having trouble identifying.

Download PsList v1.21 at
http://www.sysinternals.com/ntw2k/freeware/pslist.shtml
Download the entire PsTools package at
http://www.sysinternals.com/ntw2k/freeware/pstools.shtml

BGINFO V3.0

Managing more than one system usually means having to deal with the annoyance of using sticky notes or opening system information utilities to determine a system's specifications and configuration. BgInfo is a utility that displays standard and custom system information you specify right on the desktop, giving you at a glance all the data you consider important.

BgInfo is already versatile, allowing you to specify any of a pre-defined set of values or direct BgInfo to obtain values from the Registry or from files. BgInfo v3.0 adds even more customization capability by introducing support for Visual Basic scripts so that you can craft scripts that capture custom configuration settings and report them to BgInfo for presentation.

Download BgInfo v3.0 at
http://www.sysinternals.com/ntw2k/freeware/bginfo.shtml

PAGEDEFRAG V2.2

Have you found your paging file, Registry files or event log files becoming more and more fragmented? The built-in defragmenter doesn't handle these files because they must be defragmented at boot time before the operating system is running. PageDefrag is a light-weight defragmenter with the single purpose of defragmenting these critical system files.

PageDefrag version 2.2 includes Windows XP support and allows you to specify a countdown timer that PageDefrag displays before defragmenting. The timer works like the one chkdsk implements where pressing any key during the countdown aborts the defrag.

Download PageDefrag v2.2 at
http://www.sysinternals.com/ntw2k/freeware/pagedefrag.shtml

REGMON V4.34 for WIN64/ITANIUM

Last year Microsoft lent me an Itanium system so that I could port Sysinternals tools to Win64. Not all tools require porting since most 32-bit applications run without change on an Itanium running Windows XP 64-bit Edition by using the Itanium's 32-bit binary compatibility support. However, Windows XP 64-bit Edition will not load 32-bit drivers, which means that any Sysinternals tool that has a driver component must at least have the driver ported to 64-bits.

The first utility I ported was Filemon. Filemon's port was relatively simple because all it required was changing the driver and GUI source to conform to 64-bit data type rules and then recompiling. I described the porting process in the last newsletter. Regmon's port included the same data type issues, but included other issues that made it much more difficult to port.

Regmon relies on a driver that patches the system call table in the kernel with Regmon interception routines so that it can monitor Registry activity. To make a long story short, the system call interface on the Itanium has processor-dependent characteristics that make the approach I used in the 32-bit driver unusable. I therefore had to use some difficult coding to get system call interception to work on 64-bit XP.

Fortunately, Microsoft has implemented a Registry activity callback interface in the Windows .NET Server kernel that makes the technique Regmon uses to watch Registry activity unnecessary. Look for a Regmon update in the near future that demonstrates the new Windows .NET Server API and a description of the API in the newsletter.

Download Regmon v4.34 at
http://www.sysinternals.com/ntw2k/source/regmon.shtml

SYSINTERNALS IN PC WORLD MAGAZINE

Regmon and Filemon both got nice plugs in the August issue of PC World, a high-circulation computer magazine, as recommended troubleshooting tools. Read more at http://www.pcworld.com/features/article/0,aid,101750,00.asp

SYSINTERNALS AT WWW.MICROSOFT.COM

Here's the latest installment of Sysinternals references in Microsoft Knowledge Base (KB) articles released since the last newsletter. This brings to 38 the total number of KB references to Sysinternals.

  • HOW TO: Create Custom Administrative Templates in Windows 2000
    http://support.microsoft.com/default.aspx?scid=kb;EN-US;q323639

  • Logon Delays Over Slow Links if Oplock Is Not Granted for the Policy File
    http://support.microsoft.com/default.aspx?scid=kb;EN-US;q319440

  • XCON: The MTA Process Is Leaking Memory with a High Handle Count http://support.microsoft.com/default.aspx?scid=kb;EN-US;q313735

In addition, Microsoft has honored Sysinternals as the currently featured Windows XP Embedded community site! Check out http://www.microsoft.com/windows/embedded/community/default.asp

INTERNALS INFORMATION

MARK AND DAVID SOLOMON TEACH INTERNALS IN LONDON

Hear me and David Solomon present our 3-day Windows 2000/XP/.NET Server internals class in London September 23-25 (the next class is December 9-11 in Austin, TX). Based on "Inside Windows 2000, 3rd Edition", it covers the kernel architecture & interrelationship of key system components & mechanisms such as system threads, system call dispatching, interrupt handling, & startup & shutdown. The internals of key subsystems explained include processes & threads, thread scheduling, memory management, security, the I/O system, NTFS, and the cache manager. By understanding the inner workings of the OS, you can take advantage of the platform more effectively and more effectively debug and troubleshoot problems.

For more information see http://www.sysinternals.com/seminar.shtml

INSIDE WINDOWS NT DEVELOPMENT

This presentation is a couple of years old, but is nevertheless an interesting look inside the development history of Windows NT 3.1 through Windows 2000. Mark Lucovsky, one of the original NT software engineers, delivered the presentation at the USENIX Annual Technical Conference in 2000. He describes the size of the development teams, defect rates (of the code they wrote, not the members), the source control system, and team development challenges.

See Mark Lucovsky's talk on NT development history at http://www.usenix.org/events/usenix-win2000/invitedtalks/lucovsky_html/

On a related note, here's Microsoft's own history of Windows, from Window 1.0 through Windows .NET Server: http://www.microsoft.com/windows/WinHistoryIntro.asp

WHEN A CHANGE DOESN'T COUNT

Windows keeps track of several timestamps for files and directories. They include create, last access, and last modified. The last modified timestamp is documented as recording the last time that a file was written to. Unfortunately, an application that relies on this being true will be in for a nasty surprise.

If an application opens a file using the CreateFile API and then writes to it via WriteFile or its variants, then the last modified time does reflect the time of the file's most recent modification. If the application opens the file for memory-mapped I/O using CreateFileMapping and MapViewOfFile, then modifies the file by changing the mapped contents, the data gets eventually written back to the file by the Memory Manager and the on-disk version is modified. The problem is that file system drivers don't update a file's last modified time when a modification occurs as a result of a memory-mapped write.

Why not? I assume the developers cut the corner in favor of performance, but the behavior isn't documented so I can only speculate. I guess all writes aren't created equal.

.NET SERVER RC 1

Windows .NET Server RC 1 has shipped and there's a chance that Microsoft will cut it's RTM (Release to Manufacturing) build of the new server OS by the end of the year (about the same chance as getting bit by a snake and a shark in the same year). From a kernel perspective there's not major visible change from Windows XP. There aren't any major new kernel subsystems, Win32 or kernel APIs - just tweaks that enhance performance on large systems and a handful of APIs that round out existing API sets.

There are, however, big changes in user-space. The biggest is of course the .NET framework, but there are also new server-oriented services. The Volume Shadow Copy service, introduced in Windows XP for backup, is utilized to provide the ability for client users to rollback files on server shares to previous versions. No administrator interaction is required because a user can simply select the desired entry in a file history list shown in a new Explorer file property tab.

A server networking enhancement is Winsock Direct, an infrastructure that allows application on a server in a multi-tiered configuration with specialized network interconnection, a configuration called a System Area Network (SAN), to talk directly to the networking hardware, bypassing the TCP/IP stack and the kernel.

Look for an upcoming article by David Solomon and I that describes the .NET Server system-level changes. We are also covering .NET Server in the next revision of Inside Windows 2000, to be called Windows Internals, due for publication next spring.

LOADS OF WMI SCRIPTS

If you like the PsTools suite from Sysinternals, then you are probably a scripting junky, and if your systems are running Windows 2000 or XP, or have the NT 4 WMI core components installed, then you should head to Microsoft's TechNet site and grab the free WMI sample scripts they've posted there. There's a large collection and they are divided into categories like Computer Management and Networking and you can download them individually or all at once.

Get free sample WMI scripts at http://www.microsoft.com/technet/treeview/default.asp?url=/technet/scrip tcenter/sampscr.asp

Speaking of freebies from Microsoft, you can find a subset of the Windows 2000 Resource Kit tools posted for download. Visit http://www.microsoft.com/windows2000/techinfo/reskit/en-us/default.asp

USING BOOTVIS TO PROFILE THE BOOT PROCESS

I periodically get requests to develop a tool for Sysinternals that monitors the boot process so that a user can determine what causes delays they might experience. Fortunately, I waited long enough that Microsoft beat me to it. The tool they wrote is called BootVis and it served an important role in the boot performance optimizations they implemented for Windows XP. You can download it from http://www.microsoft.com/HWDEV/PLATFORM/performance/fastboot/BootVis.asp . BootVis, which only runs on Windows XP, uses XP's WMI kernel tracing facility to collect a detailed trace of the boot process, including CPU usage, disk I/Os, process startup and driver I/Os. Calling the help file that comes with BootVis terse is being kind, so I'll walk you through how to use the tool.

If you want to be able to see precisely what functions with drivers and modules account for CPU time then open the Tools|Options dialog and point BootVis at your installed Windows XP symbols. Next, have BootVis capture a trace of the next boot, including driver delays, by selecting Trace|Next Boot+Driver Delays. BootVis will countdown and then reboot your system.

After the system has booted log in and run BootVis again. Select Trace|Stop Trace and it will extract the trace that's being collected from the kernel and save it to a file, which it then loads for display. You'll see four charts. In Cpu Usage you see a track of how much CPU processing was consumed through the boot. The Disk I/O bar chart shows the number of disk reads (red) and writes (blue) that occurred during each second. In the Driver Delay graph you'll see rectangles that represent driver I/Os that had durations larger than the default BootVis cutoff of 100 milliseconds - such delays are considered significant. Finally, the Process Creates chart shows labeled rectangles where processes started.

BootVis doesn't necessarily show all the trace data collected when you start it, so use the zoom-out button until you start to see the right edge of graphed data. Now take a look at each of the graphs. If you see a CPU or disk spike or want to zoom in on process startup activity use the zoom-in toolbar button. Note that the starting time shown will remain 0 as you zoom in, so after zooming manually enter a time near the activity in which you're interested in the Display Time edit box.

The CPU Usage, Disk I/O and Driver Delay graphs allow you to view more information than is immediately visible. Select a region in the graph of interest by moving the cursor, pressing the left mouse button and holding it down, and then dragging. Then right-click to get a menu. The interesting menu option in the CPU Usage graph is Show Summary, which pops open a dialog listing a breakdown by process of CPU usage for the selected time interval. If you double-click on one of the processes another dialog opens that shows a breakdown of CPU usage by module (e.g. the DLLs in the process). If you have symbols configured you can even double-click on a module and obtain a dialog showing a breakdown of that module's CPU usage by function.

Two menu items appear when you right click on a selected region in the Disk I/O graph: Show Summary Table and Show Detail Graph. Select Show Summary Table to see an accounting, broken down by module, of all the I/Os that occurred during the interval. Choose Show Detail Graph to display a line chart representing the actual sectors within the disk that were accessed over the interval. The detail graph even associates the I/Os with the processes that performed them and lets you filter out processes by deselecting them in the list on the left. You can select a region within the detail graph and access another summary chart for that subinterval by right-clicking and selecting the resulting menu item.

Choosing Show Summary Table from the Driver Delay graph's context menu brings up a list of the I/Os shown in the selected region that includes the name of the driver that performed the I/O, the begin, end, and total time of each I/O, and the I/O function performed.

That brings me to the end of my BootVis tutorial. Although it's not the most polished tool, after playing with BootVis for a while you'll no doubt agree with me that the level of detail viewable with its graphs is amazing. The next time you have a long boot and want to know what's responsible you'll know where to turn for the answer.

WHAT'S COMING UP

A QUICK INTRODUCTION TO CRASH DUMP ANALYSIS

Microsoft's recent enhancements to its kernel debugging tools have made crash dump analysis virtually painless. Next time I'll show you how to become a dump analysis pro in under an hour.

Thank you for reading the Sysinternals Newsletter.

Published Monday, August 12, 2002 7:00 PM by ottoh

[Newsletters Archive ^] [< Volume 4, Number 1] [Volume 4, Number 3 >]