1. Unplug the system from the network
2. Run Process Explorer, click Options, Verify Image Signatures, and Options, VirusTotal, Check VirusTotal.com.
3. Add the Verified Signer  and VirusTotal columns if not already visible.
4. Suspend any process that is unverified and/or ranks high (red) in the VirusTotal column. Suspend is necessary at first since many unwanted software will have a "brother" that will re-spawn the other.
5. Kill all of the processes you just suspended.
6. Follow up with Autoruns.exe to "root out" the unwanted software.
7. Reboot and repeat as necessary.

$37.46 at https://store.elsevier.com/product.jsp?isbn=9780124167018&_requestid=465338

$45.42 (Prime discount) at https://www.amazon.com/dp/0124167012/ref=wl_it_dp_o_pC_nS_ttl?_encoding=UTF8&colid=12JDCG3UP69SD&coliid=I2TOVTYHI6HDHC

I certainly don’t know everything about Windows performance analysis (no one ever will), so *many* of my friends and colleagues are named in the book as people who have helped me along the way.

A special thanks to Microsoft Premier Field Engineering (PFE), Jeff Stokes, Yong Rhee, Ed Wilson, and Mario Hewardt for helping to make this book possible.

This article is a walk through of how to create a performance counter data collector set template with all of the performance counters used by industry experts, how to import the template onto a local or remote system, and how to automatically start the data collector set at Windows start up – Performance Monitor does not have a feature to automatically start a data collector set after a reboot.

This procedure was tested on Windows 8.1 and Windows Server 2012 R2, but should work fine on Windows Vista, Windows Server 2008, and later operating systems.

Create a “black box” performance counter data collector template

First, determine the Microsoft products and services installed on the target system. In this example, I have a customer who needs a data collector set template for a Microsoft Windows Server 2012 R2 system running a named instance of Microsoft SQL Server 2012. Therefore, I want to target performance counters for both Windows Server 2012 R2 as well as SQL Server 2012.

Arguably the easiest way to do this is to install the Performance Analysis of Logs (PAL) tool – an open source project/tool of mine. It requires Powershell 2.0 and the Microsoft Chart Controls for the .NET Framework 3.5 – both are free products from Microsoft. Once installed, run the PAL Wizard from the Windows Start menu and navigate directly to the Threshold File tab. Select a threshold file or combination of threshold files (they can be mixed and matched through the inheritance pane, and click the Export to Perfmon template file button.

In this case, I selected Microsoft SQL Server 2012 which already has an inheritance of System Overview and Quick System Overview which are good for all Microsoft Windows operating systems.

Since SQL Server might have named instances, the PAL tool detected SQL counters and prompts me to name any SQL named instances.

In this case, I specified “SQL2” and saved it as an XML file which is the format that Windows Server 2008 and later uses for counter log templates. Once the XML file is saved. Optionally, open it in Notepad to verify that it has the counters that you expect to see.

This template is a good start, but will require additional modifications to make it a circular log.

Create a performance counter data collector using the template

Copy the XML template that you created in the previous step to a file system or network share location that is accessible to your domain account. This could be the a local disk drive on your workstation, a local disk drive on the target system, or a network share that your workstation or the target server has access to.

On your workstation or at the console of the target system, open an Administrator command prompt. This procedure requires administrator rights on the target system.

Run the following command to create a data collector using the template:

logman import <NameOfDataCollectorSet> -xml <PathToXmlTemplate> -s <NameOfTargetSystem>

Here is an example of me creating the data collector set named, “SQL2_Blackbox_darksteel” (the %computername% resolves to the local computer name using the template I created earlier, “Sql2Template.xml”, on a remote system called, “darksteel”.

Run the following command to check if the data collector set was created:

logman query

Note that the data collector set has not been started yet. Do not start it just yet.

Update the data collector set to be in binary circular format with a log file maximum size of 300 MB or a size that you prefer.

logman update <NameOfDataCollectorSet> –f bincirc –max <SizeOfLogInMB>

Now, the data collector set can be started and left to run forever without ever being larger than the size specified.

Optionally, the output location can be changed using the following sample command:
logman update SQL2_Blackbox_DARKSTEEL -o D:\Perflogs\SQL2_Blackbox_DARKSTEEL.blg

This file path is in respect to the file system on the target system – not the workstation from which the command might be running.

Important: Ensure that the output directory location does not already contain data. Performance Monitor’s Data Management features might delete data at that location.

To start the data collector, run the following command:

logman start <NameOfDataCollectorSet>

Setup the data collector set to start on the startup of Windows

Unfortunately, Performance Monitor does not provide a way to automatically start the data collector set after a reboot. This means all of your hard work of creating a “black box” data recorder will be for nothing after someone reboots the system. In this step, we will setup the data collector set to automatically start when Windows starts by using Task Scheduler.

Run the following command in order to have the data collector set automatically start after a reboot:

schtasks /create /tn <NameOfDataCollectorSetOrAUniqueName> /sc onstart /tr "logman start <NameOfDataCollectorSet>" /ru system /S <NameOfTargetSystem>

Note: The /S parameter of schtasks can be used to create this task on a target, remote, system.

Now the data collector set will automatically be started when the system starts.

Setup the data collector set to delete the oldest log file when low on disk space

By default, data collector sets will create a unique counter log each time it is started. This is to prevent the deletion of the existing logs. This is a nice feature, but this can result in several log files that might just be taking up too much disk space.

To setup the data collector set to delete the oldest log file when the disk space is low, open Performance Monitor (Start, Run, Perfmon.exe). Expand Data Collector Sets, expand User Defined and select the data collector set you wish to target for this. Right click on the data collector set and click Data Manager.

In this case, I setup the data collector set to delete the oldest counter log file.

Important: This setting can potentially delete existing data in the output directory location. Ensure that the output directory location *only* contains counter log files for this data collector. Avoid paths such as the root of a disk drive.

Conclusion

After following these steps you will have a data collector set that can run indefinitely without running the system out of disk space – it stays at the maximum size or smaller. Just stop the data collector set (the data collector set must be stopped before moving the counter log file otherwise it can result in file corruption) and analyze the counter log with the Performance Analysis of Logs (PAL) tool.

Want more? These procedures and *much* more is covered in my book, “The Windows Performance Analysis Field Guide” due out in August 2014.

A relatively quick and easy way of validating the drivers of a system is to use the Sysinternals (Microsoft owned) tool, Autoruns. Autoruns is a free tool that does not require installation. Just download and run. For more information on Autoruns go to https://technet.microsoft.com/en-us/sysinternals/bb963902 and consider reading the book, “Windows Sysinternals Administrator’s Reference”.

1. Download and launch Autoruns.exe

1. Download Autoruns using one of the links above or click here and save it to your desktop.
2. Double-click on it to run it. 
   Note: Administrator rights is not required unless you intend to make changes to the system. Also, you might be prompted for Administrator credentials or other User Access Control (UAC) prompts depending on your settings.
3. Autoruns will begin to gather system information. For now, press the ESC key to stop the gathering of data since we will be rescanning the system shortly. Otherwise, wait for the Status bar to reports, “Ready”.

• Filter Autoruns: This step will remove common entries that are relatively safe and enable verification of the signatures of the drivers, EXEs and DLLs of software targeted for startup on the system.
  Note: Microsoft drivers are certainly not immune to problems, but they are generally written well enough to not be on the list of usual suspects.

1. In the menu, click Options, Filter Options.
2. Enable/check Verify code signatures, Hide Microsoft entries, and Hide Windows entries.
   
   
   
3. Click the Rescan button and wait for the hour glasses to go away. Again, this might take a few minutes depending on the resources of the system.

• Validate drivers: This step checks non-Microsoft drivers. According to the Windows Sysinternals Administrator’s Reference, “Verifying a digital signature associated with that file gives a much higher degree of assurance of the file’s authenticity and integrity.”
  Note: When a driver is verified, the Publisher field changes from the company name to the name on the signed certificate.

1. Click the Drivers tab and look for drivers that are “Not Verified”. This will show up in the Publisher field.
2. In my case, I found one driver that is not Microsoft, but it was verified to be a virtual networking vendor.
   
    
3. If any of the drivers are highlighted and come up as “Not Verified” in the Publisher field, then the driver does not have a digital signature.

Drivers that are “Not Verified” do not have a digital signature or have been modified. It doesn’t mean that the driver is malware. It just means that the driver cannot be verified to be from the publisher that is claims to be from. For example, the driver might have been written by a small company that didn’t bother to have their driver signed.

Note: 64-bit versions of Windows and Windows Server require all drivers to be signed when loaded, but this policy can be by-passed.

With that said, malware commonly installs a driver to gain access to the system and it could be exposed through this procedure. My advice is if the system is not functioning properly, then I would uninstall the unverified drivers.

Beyond driver validation, Autoruns is great for fixing performance issues as well. I simply go to all of the other tabs and uncheck/disable all of the software that I don’t recognize. Be careful when doing this and disable only the entries that you recognize and don’t want. If you accidentally disable something that you need, then its okay because Autoruns makes a backup of the entry and it can be re-enabled by running the tool again.

I hope you find this helpful.

With that said, if you deal with many unique counter and don’t want to create your own threshold file yet, then consider using the script discussed in this blog entry as a starting point.

Due to popular demand, I created a Powershell script that will convert a performance counter data collector template into a PAL threshold file. The script is called a non-imaginative PerfmonTemplateToPalThresholdFile.ps1. Be forewarned that the threshold file produced from this will not have any thresholds in it.

For now, I consider it to be beta, so it can be downloaded from:

https://aka.ms/clinth, then go to PAL/BetaTesting

or

https://skydrive.live.com/?cid=e6360c54b48a891b&id=E6360C54B48A891B%21428&authkey=!APQbXtD8U-HExzE#cid=E6360C54B48A891B&id=E6360C54B48A891B%215119&authkey=%21APQbXtD8U-HExzE 

In a few weeks or so, I will include it with the PAL tool – likely v2.4.1.

How to use it

1. Ensure Powershell is set to unrestricted – WARNING this can be a potential security risk
1. At an administrator Powershell session run: Set-ExecutionPolicy unrestricted
2. Place a performance counter data collector template file (it must be in XML format) in the same directory as the script.
3. Unblock the script file. Files downloaded from the internet are considered high risk, so go to the properties of the script file and click the Unblock button. Otherwise, the script will not be permitted to run.
4. Run the script
1. Start a Powershell session and change directory to the script location.
2. .\PerfmonTemplateToPalThresholdFile –PerfMonTemplate .\SysTemplate.xml
3. A PAL threshold file will be created in the same directory. It will use the DataCollector Name attribute in the template file as the file name.
5. Copy the PAL threshold file to the PAL installation directory
1. Once the PAL threshold file is created, copy it to the PAL installation directory.
2. Run the PAL Wizard and you should see the file in the list of threshold files.

Considerations

• This script is designed for PAL v2.0 and later.
• The PAL threshold file produced will only create statistics only – no thresholds.
• If you goal is to have all of the counters in a counter log to be represented in the PAL report, then use the AllCounterStats feature.

The only real way of knowing if a page file is actually being “read from” is to get a file IO trace. This can be collected and/or viewed with tools such as the Microsoft Performance Recorder/Analyzer, Microsoft Resource Monitor, or Sysinternals Process Monitor.

Using Resource Monitor

Resource Monitor is built into the operating system and can be launched from the Performance tab of Task Manager.

The Disk Activity pane in Microsoft Resource showing reads and writes to pagefile.sys.

The Disk tab shows the processes and files involved in live disk activity. This data comes from Event Tracing for Windows (ETW) and shows much more data than what performance counters can provide. In this case, we can see the file C:\pagefile.sys being written to by the System process.

Using Windows Performance Recorder/Analyzer

Microsoft Windows Performance Recorder/Analyzer is part of the free Windows Assessment and Deployment Kit (ADK) and can capture activity related to hard page faults and the processes and files associated with them. In this case, I used WPR to record Disk IO and File IO activity while forcing the system to use the page file. In the screenshot I am showing the Hard Faults chart within Windows Performance Analyzer (WPA) and I aggregated the data to show the files most involved with hard page faults and in this case it was C:\pagefile.sys. This is proof that most of the hard page fault activity was related to the page file in this case.

Using Sysinternals Process Monitor

Sysinternals Process Monitor (Procmon) can also show page file reads and writes. This can be done by enabling the Advanced Output option under Filter, Enable Advanced Output. Once enabled, set the filter to only show events where “Path is C:\pagefile.sys” or similar for other page files.

After capturing data for a few minutes, click Tools, File Summary to get more details of the number of reads and writes to the page file.

Hard page fault related performance counters

The performance counters \Memory\Page/sec, \Memory\Page Reads/sec, and \Memory\Page Inputs/sec measure hard page faults (faults that must be resolved by disk) which *may* or *may not* be related to a page file or a low physical memory condition. Hard page faults are a normal function of the operating system and happen when reading the portions of image files (DLLs and EXEs) as they are needed, when reading memory mapped files, or when reading from a page file. High values for these counters (excessive paging) indicate disk access (generally 4 KB per page fault on x86 and x64 versions of Windows and Windows Server), but again, they may or may not be related to page file activity, but they contribute towards disk usage which can increase the likelihood of system-wide delays assuming the related disk(s) are overwhelmed. Therefore, it is recommended to monitor the disk performance of the logical disks hosting a page file in correlation with these counters.

Note: \Memory\Page Writes/sec and \Memory\Page Output/sec do measure only page file writes.

In addition, the counter \Paging File(*)\% Usage provides us the usage, but not how often a page file is actually being accessed. In addition, hard page faults just mean disk access.

This is the reason why the Performance Analysis of Logs (PAL) tool measures a low memory condition that could cause system wide delays by taking many performance counters into consideration. The PAL tool has an analysis called Physical Memory Overwhelmed and creates a factious counter called \Memory\Physical Memory Overwhelmed. In this analysis, it analyzes for low physical memory (\Memory\Available MBytes), page file usage (\Paging File(*)\% Usage), and disk counters related to the page file(s) to determine if they are overwhelmed when the system is low on physical memory.

One of the hardest parts about scripting is getting to know the properties and methods of objects and state of a script during execution.  In this procedure, I show you how to modify Microsoft Visual Studio 2005 for full debugging of VBScripts. I have used this procedure for many years starting with Windows XP. This time, I recently did this on Windows 8.1.

Install and run Visual Studio 2005

Yes, it must be a full version of Microsoft Visual Studio 2005 (Express editions do not work) and it can’t be older or a newer version of Visual Studio, so break out that old archive of software and install it.

During installation, the only option that is needed is “Visual Basic”. In my case, I cleared all other checkboxes and only checked Visual Basic. Then finish the installation wizard.

On Windows 8.1, I had to install an older version of the Microsoft .NET Framework and run Visual Studio 2005 with Administrator rights in order for it to create some starting folders and files.

Configure Visual Studio 2005

Once Visual Studio 2005 is installed, you can immediately run your scripts through the Visual Studio 2005 debugger by using the “//X” argument in CScript.exe or WScript.exe. This prompts the scripting engine for a debugger. I will show how to do this later. In the meantime, let’s configure Visual Studio 2005 a bit more to make it better suited for VBScript execution and editing.

Visual Studio will look like this initially.

In the menu, click Tools, External Tools.

Optionally, delete the “Dot&fuscator Community Edition” entry. In my case, I removed it.

For CScript execution

Click Add and set the fields to these values:

Leave all other entries at default settings and Click Apply.

Note: Optionally, use WScript in the command path if you want to have WScript.exe as the VBScript execution engine.

For CScript debugging

Click Add and set the fields to these values:

Leave all other entries at default settings and Click Apply.

Note: If you want to prompt for arguments to the script before execution, then check the “Prompt for arguments” check box.

Click OK on the “External Tools” dialog box when finished adding entries.

Add the scripting buttons to the toolbar

Right-click the Visual Studio 2005 toolbar (the bar with File, Edit, View, and so on) and then click Customize. Click the Commands tab and select Tools from the Categories menu. In the Commands menu, scroll down until you find “External Command 1”. Click, hold, and drag “External Command 1” to the menu bar of Visual Studio 2005. You can put it anywhere you want.

Note: The button will initially show up as “External Command 1”, but will change to the external command title we provided earlier once the toolbar Customize dialog box is closed.

Repeat this for all of the entries in the External Command window that you created earlier such as “External Command 2” and “External Command 3” and so on. Click Close on the Customize dialog box when finished.

In my case, CScript and CScript Debug show up on my toolbar.

Show line numbers

When a VBScript throws an error, it provides the line number and character number of where the error occurred. For this reason, it is convenient to have the line numbers shown when editing the scripts.

Click Tools and select Options. The Options dialog box will show. Check “Show all settings”.

In the tree view control on the left, navigate to Text Editor and select All Languages. Check “Line numbers” on the right and then click OK.

Executing VBScripts in Visual Studio 2005

To open a script within Visual Studio 2005, simply right-click on the script, and Open with Visual Studio 2005. In this case, I opened up test.vbs.

To execute the script, I just click the CScript button. In my case, my script output automatically showed the output in the Output window below.

Debugging VBScript in Visual Studio 2005

Now for the best part! Debugging the script. When a script loaded into Visual Studio 2005, click the CScript Debug button (or whatever you named it). If it throws an error of not able to find the script, then click anywhere in the script editor pane and try it again. The script has to have the focus.

When the debugger starts, it prompts if you want to debug the script from within the same instance of Visual Studio or a new instance of Visual Studio. Choose the existing instance of Visual Studio especially if you already have break points set. Then click the Yes button.

This starts off the script in the debugger and it stops at the first line of code. The yellow arrow and highlight indicate the line of execution.

At this point, you can do debugging as you normally would in Visual Studio such as setting break points, run to cursor, show locals, have a watch window and so on. Use the commands in the Debug menu and/or Debug icons to debug the script.

Showing the output when debugging

A quirk of the debugger is that you have to add the output window again by clicking on down arrow of the icon of a window with a red sphere, then selecting Output.

Change the output window to the debug output.

Now, you should see the WScript.echo output from the script when debugging.

Using Locals and/or the Watch window

I like looking at the variables of my scripts using the Locals and Watch windows. While in the script is paused in the debugger, click the “Watch 1” icon in the Visual Studio tool bar. This will replace the Output window with the Watch 1 window.

In this case, I can select variable “b” and drag it to the Watch Window.

I can do this with any variable or object within a script. This technique is especially helpful with objects and arrays because the Watch window allows you to explore the it and see the values.

Note: You cannot edit the script while it is running in the debugger and the script must be saved in order to run what you are seeing in the editor.

Intellisense and quirks

Visual Studio tries to provide “intellisense” to objects, so that you can just type an object instance name and type, “.” to have it provide you with the properties and methods of the object. Unfortunately, Visual Studio 2005 only provides intellisense for a few of the VBScript objects. Also, please understand that Visual Studio 2005 actually thinks that the VBScript is VB.NET code, so there will be a few inconsistencies.

Also, remember that the debug output has to be enabled. See “Showing the output when debugging” discussed earlier.

For a richer intellisense experience with VBScript editing, consider third party tools such as Sapien Technologies Primal Script.

Conclusion

There are certainly other editors out there for debugging VBScripts, but this one seems to be the easiest and richest experience in regards to VBScript debugging. I’ve used this technique for years for my own VBScripting needs and to teach students the basics of VBScripting. I hope you find this useful.

The largest number that can be represented by 32-bit is 4 GB. That 4 GB is divided into user mode (process memory) and kernel mode. By default, this gives the kernel 2 GB to work with. The kernel memory is then divided further into a Page Frame Number (PFN) database, Pool Paged, Pool Nonpaged, and the rest of the free memory is given to Free System Page Table Entries (PTEs). In Windows Server 2003, Pool Paged and Pool Nonpaged have a specific size. This means that the system can run out of either pool and hang when there is still plenty of free kernel virtual memory elsewhere. This is why 32-bit versions of Windows Vista and Windows Server 2008 and later allow the full kernel virtual address space to be used.

Instead of me going into all of the complexities of how to detect this problem and how to troubleshoot it, I will just go into these recommendations if you suspect the system it out of kernel pool memory.

1. Migrate to 64-bit Windows Server: 64-bit has 8 TB of virtual address space for kernel memory which will effectively solve this issue, but I realize that it might be difficult to go to 64-bit Windows Server. All drivers on a 64-bit version of Windows Server must be 64-bit, but 32-bit applications can run on 64-bit Windows Server.
2. Upgrade to 32-bit Windows Server 2008: If it is impossible to go to a 64-bit version of Windows Server, then at the very least consider upgrading to 32-bit Windows Server 2008. This version of Windows Server allow Pool Paged and Pool NonPaged to expand to up to just less than 2 GB assuming there is more than 2 GB of physical memory installed – Pool NonPaged’s maximum is 75% of physical memory (RAM).
3. Disable Dynamic Memory (hot-add): If the physical system has the capability to add more physical memory (RAM) to the system, then the Page Frame Number (PFN) database in Windows is likely larger than needed. To reduce the PFN database to a smaller size, set the DynamicMemory registry key to 1 which means 1 GB. The system realizes it is too small and resets it to the actual memory installed. This will likely increase the size of Pool Paged Bytes by 100 MB which could make the Pool Paged size to 169 MB.
   How to Configure the Paged Address Pool and System Page Table Entry Memory Areas
   https://support.microsoft.com/kb/247904
   The DynamicMemory registry key is located under the following registry key:
   HKLM\System\CurrentControlSet\Control\Session Manager\Memory Management
4. View the actual amount of Pool Paged maximum size: Use Process Explorer with the Debugging Tools for Windows (point Process Explorer to the dbghelp.dll file that comes with the Debugging Tools for Windows) and a symbol path of “SRV*C:\Symbols*https://msdl.microsoft.com/download/symbols” (internet access required) to see the maximum size of the kernel pool paged.
5. Look for kernel insufficient memory events: In the System event logs, look for Event ID 2019 and 2020 messages. If the system has these events, then it indicates that the system ran out of kernel virtual memory.
6. Consider adjusting PoolPagedSize with CAUTION: Be *very* careful with this!!! The PoolPagedSize registry key (see the link for more information) can be set to a larger amount than what it is, *but* this will take away from the System PTEs and your system is borderline for running out of those as well. Only do this if a Microsoft Support Professional has recommended this.
7. Use Poolmon.exe: If the pool paged memory usage is too much, then use Poolmon.exe to identify which drivers are consuming the most. The download and usage of Poolmon is described in this blog entry by Mark Russinovich (author of the Windows Internals book and creator of the Sysinternals tools).
   Mark Russinovich’s blog entry on Windows pool memory troubleshooting:
   https://blogs.technet.com/b/markrussinovich/archive/2009/03/26/3211216.aspx?wa=wsignin1.0.