Likely problem?

https://www.microsoft.com/en-us/download/details.aspx?id=55080  

Yes. While this is not debugging using windbg, procmon has the capability to aid us in debugging as long as we have the necessary symbol files.
The intention of this blog is to reveal some powers of procmon tool. We use it on various troubleshooting occasions when we don't know what is really happening behind the scenes.

In my opinion, procmon is the most powerful troubleshooting tool (at least, I use it for resolving most cases)
Thought of writing this blog with an example of an issue that I happen to solve for a customer using this tool.
Recently, I worked on an issue with a customer who had this particular scenario - Multiple processes would crash on launching. This seems to be affecting hundreds of machine and has been happening for over a few months. Some examples as below:
‘Bit Locker Encryption Options’ from the control panel:
--------------------------------------------------------------------------

Configuration Manager or SCCM’s ‘Software Center’:
-----------------------------------------------------------------------

Typically, it is supposed to show some more stuff under this, however before it could completely load everything, it would end up in crashing for some unknown reasons... Well not really unknown in the end of this blog. 
For now, I would take the normal approach here.

Pretty confident that there has to be something interesting in the event logs.
Sure enough, you would find the classical application crash events in the event viewer.

Event related to Bit Locker Encryption Options crash:
-----------------------------------------------------------------------

Log Name:      Application
Source:        Windows Error Reporting
Date:          24-12-2014 12:12:42
Event ID:      1001
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      ABC.XYZ.COM
Description:
Fault bucket 85905360500, type 4
Event Name: APPCRASH
Response: Not available
Cab Id: 0
Problem signature:
P1: MBAMControlUI.exe
P2: 2.1.117.0
P3: 5271df47
P4: unknown
P5: 0.0.0.0
P6: 00000000
P7: c0000005
P8: 0000000000000000
P9:
P10:

Log Name:      Application
Source:        Application Error
Date:          24-12-2014 12:12:29
Event ID:      1000
Task Category: (100)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      ABC.XYZ.COM
Description:
Faulting application name: MBAMControlUI.exe, version: 2.1.117.0, time stamp: 0x5271df47
Faulting module name: unknown, version: 0.0.0.0, time stamp: 0x00000000
Exception code: 0xc0000005
Fault offset: 0x0000000000000000
Faulting process id: 0x%9
Faulting application start time: 0x%10
Faulting application path: %11
Faulting module path: %12
Report Id: %13
Faulting package full name: %14
Faulting package-relative application ID: %15

Events related to Software Center’s crash:
--------------------------------------------------------

Log Name:      Application
Source:        Windows Error Reporting
Date:          24-12-2014 12:03:12
Event ID:      1001
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      ABC.XYZ.COM
Description:
Fault bucket -329875240, type 5
Event Name: BEX
Response: Not available
Cab Id: 0
Problem signature:
P1: SCClient.exe
P2: 5.0.7958.1000
P3: 5230d2b7
P4: StackHash_5861
P5: 0.0.0.0
P6: 00000000
P7: PCH_E4_FROM_ntdll+0x0003C1AC
P8: c0000005
P9: 00000008
P10:

Log Name:      Application
Source:        Application Error
Date:          24-12-2014 12:03:06
Event ID:      1000
Task Category: (100)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      ABC.XYZ.COM
Description:
Faulting application name: SCClient.exe, version: 5.0.7958.1000, time stamp: 0x5230d2b7
Faulting module name: unknown, version: 0.0.0.0, time stamp: 0x00000000
Exception code: 0xc0000005
Fault offset: 0x00000000
Faulting process id: 0x1404
Faulting application start time: 0x01d01f4374dffc6d
Faulting application path: C:\Windows\CCM\SCClient.exe
Faulting module path: unknown
Report Id: b785c36e-8b36-11e4-be97-28e3471721ed
Faulting package full name:
Faulting package-relative application ID:

Log Name:      Application
Source:        .NET Runtime
Date:          24-12-2014 12:03:03
Event ID:      1026
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      ABC.XYZ.COM
Description:
Application: SCClient.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: exception code c0000005, exception address 00000000

As you can see above, the faulting module is showing as unknown. So, we don’t know for sure what the culprit here is.

Typically in these scenarios, we would capture a memory dump when the crash occurs and have the customer send the dump file to us. We would then debug it to understand the cause of the crash.
However, that’s a lengthy and tedious process.
First of all, we would need to look at any logs if available for the component that we are troubleshooting. If not, we go with memory dump.

Before I get onto the memory dump procedure, this time though; I decided to take my favorite approach – The procmon approach. 
Usage of procmon is pretty simple. Procmon has evolved and has been evolving with new great improvements, advanced filters and additional activities.
Though many of us in the system admin world already knows about it, I’m posting the capture procedure for the benefit of others.
Once you download and run procmon.exe tool, it would immediately start capturing file system, registry and process/thread activities.

The Capture button which looks like a magnifying glass (mouse is hovered in the picture) is same button used for both starting as well as for stopping the captures. Kind of like ON / OFF switch. You could also use the CTRL + E button combination to issue a button stroke if you would like.
As you see in the bottom of the screenshot, it has already captured so much of events though I stopped the capture within 2 seconds after I opened it… Those are the things happening in the background whether you like it or not.
The very fact that it takes too much of information can be chatty as well and procmon log analysis requires that we put in our logic for the issue that we are troubleshooting. Many events can lead us to go in different directions of analysis. So, the first and foremost technique in data capture using procmon is to keep it as short as possible. While there are many ways using filters to take only necessary data, I’m not discussing about those at this time.

Let’s do the data capture now.
Download link here:
http://technet.microsoft.com/en-us/sysinternals/bb896645.aspx
By default, once we open the tool, it would start capturing data automatically.
So, we click on the capture button to stop the capture first.
Next is to click on the clear button or (use CTRL + X button combination) to clear everything that it has captured till now.

Looks nice and clean now.
If possible, close all applications that you can to avoid getting chatty.
Be ready to reproduce the issue. Most of the times, we keep procmon tool and the problem window resized in the same screen so that we can repro the issue as well as click on the capture button in Procmon. No time to waste a single second. You will see why in a bit. 
Switch ON the capture button.
Immediately reproduce the problem (In this case, I simply tried to launch the software center).
Waited for the application to crash.
Switch OFF the capture button as soon as possible.
Phew! Log capture is done.
You can now save the file (be sure to save it as ‘PML’ format and choose ‘All Events’).
 

Zip the file and send to Microsoft for analysis. If you like a little adventure, then start doing the analysis. 

Typical analysis of procmon data may not need symbols.
However, in this case that would be much required. Thankfully, Mark Russinovich had already configured it for user’s convenience. As long as you have internet access, you are good to connect to the Microsoft public symbol server.
You can see it being configured as below.
BTW, don’t forget to note that number of events that it has captured shown with the arrow. Trust me, this data capture was run only for less than 3 seconds and it already has 600 thousand + events. This is why we want to data capture to be super-fast and not to waste a single second. I’m glad you understood this. 

 
Note: For the sake of this blog, I had switched from MS internal to MS public.
In my personal experience, I have seen many times that the built-in dbghelp.dll doesn’t work.
I normally use the latest MS internal version to get around that.
Should the need arise, you may want to download and install the latest “Debugging Tools for Windows” from Microsoft. It is available in the SDK as well.

In this case, we know the name of the EXE that crashed. So, I will first filter it to show only activities by SCClient.exe (software center).

One of the most useful buttons is the Filter button (CTRL + L combination).

Add this filter and click on OK.
Now, it will show ONLY the events pertaining to SCClient.exe.

If we double click on an event anywhere, we can get into more details of that event.
Sample below:

Note that you can see the ‘Event’ tab as well as the ‘Process’ tab and the ‘Stack’ tab.
The Stack is the one that will be useful only if we have symbols in place.
Of course, it is not easy to point to the particular event that led to the crash. It takes some hit or miss logics to be tried (at least for me). Finally, I have the point of failure.

Caught the event in the list which shows the unhandled exception in the stack.

The previous event to this shows normal activity as below:
 

While this appears to be normal, we need to look closely into this event to understand why the next event shows the unhandled exception. Wait…. From what I can infer from the stack, there appears to be some kind of drawing or something to do with display enumeration.
Perhaps something is wrong with the display driver??

This time, go back to the Process tab. Next, I choose to sort by the ‘Company’ field to find out the third party modules loaded within this SCClient.exe.
 

There appears to be 2 modules from LogMeIn and 2 modules from Intel.
I ignore LogMeIn modules because I installed it on the customer’s computer for me to be able to establish a remote troubleshooting session. Issue was still happening prior to me installing it.
Let’s take a look at the Intel modules to see what they really are?
We can just double click on the module name and it gives all possible details about it.
 

and

Okay. These two files have something to do with display graphics.
Don’t want to jump the gun right away.
Going back to procmon, I look at what is the previous module that was loaded before the crash.
Filtered with ‘Load Image’ and can see the following:
 

Now, I’m certain that I really would want to eliminate these 2 modules in the first place to see if the issue happens or not.
We un-installed the Intel graphics driver and voila! Software Center opens up just fine.
So does the Bit locker Encryption Option.

Case solved!!!

Remember that I first mentioned about looking at the associated log file for the component that you are troubleshooting before jumping to dumps. In this case, there is a log called scclient_<domain>@Username.log.
Unfortunately, we didn’t have much information on this and I expect this because the issue was a crash while rendering graphics and not likely, this is going to be logged.

Oh! If you don’t really know what the associated log file is, procmon is still your friend.
Just put a filter for ‘Path’ ‘contains’ ‘.log’.
 

Once you apply that filter, it should look like this:
 

As we can see SCClient.exe is writing to a log file.
The same applies for any application to understand if that is logging anywhere or not.

Read on if you are still interested for my debugging notes….
We installed the graphics driver again, I captured a memory dump using another awesome creation by Mark – Procdump.exe
http://technet.microsoft.com/en-in/sysinternals/dd996900.aspx

0:009> .lines
Line number information will not be loaded
0:009> |
.  0          id: 474   examine              name: C:\Windows\CCM\SCClient.exe
0:009> vertarget
Windows 8 Version 9600 MP (4 procs) Free x86 compatible
Product: WinNt, suite: SingleUserTS
kernel32.dll version: 6.3.9600.16656 (winblue_gdr_pres14.140305-1700)
Machine Name:
Debug session time: Thu Dec 25 17:40:28.000 2014 (UTC + 5:30)
System Uptime: not available
Process Uptime: 0 days 0:00:05.000
  Kernel time: 0 days 0:00:00.000
  User time: 0 days 0:00:00.000
0:009> .ecxr
Unable to load image C:\Windows\System32\igdumdim32.dll, Win32 error 0n2
*** WARNING: Unable to verify timestamp for igdumdim32.dll
*** ERROR: Module load completed but symbols could not be loaded for igdumdim32.dll
eax=00000000 ebx=00000000 ecx=076f9130 edx=00000000 esi=05a8892c edi=05a86600
eip=00000000 esp=0593f28c ebp=0593f2b0 iopl=0         nv up ei ng nz na po cy
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010283
00000000 ??              ???
0:009> kb
  *** Stack trace for last set context - .thread/.cxr resets it
*** WARNING: Unable to verify timestamp for igdumdim32.dll
*** ERROR: Module load completed but symbols could not be loaded for igdumdim32.dll
ChildEBP RetAddr  Args to Child             
WARNING: Frame IP not in any known module. Following frames may be wrong.
0593f288 64616e27 05999420 0593f2a0 05a86600 0x0
0593f2b0 6461e7ae 05a86600 0593f318 07700f40 igdumdim32+0xc6e27
0593f360 645f6d83 05a86600 077011d0 00000000 igdumdim32+0xce7ae
0593f3f8 64d1479f 00000003 0593f41c 0593f474 igdumdim32+0xa6d83
0593f458 64e0d8ac 0859ace0 00000000 00000000 d3d9!CD3DDDIDX10::Blt+0x7a
0593f49c 64df5eb1 0859ace0 00000000 00000000 d3d9!CD3DDDIDX10::SurfaceBlt+0x35
0593f4fc 64db8983 0859ace0 00000000 0859ae78 d3d9!CD3DBase::SurfaceBlt+0x44
0593f5fc 69ed1767 0599a720 059f6980 0593f67c d3d9!CBaseDevice::UpdateSurface+0x648
0593f620 69ed7c92 059f6980 0593f67c 059f6680 wpfgfx_v0400!CD3DDeviceLevel1::UpdateSurface+0x1f
0593f694 69ef57b8 085c4518 0593f6bc 085de5f0 wpfgfx_v0400!CD3DGlyphBank::RectFillAlpha+0x206
0593f6cc 69ef1787 0593f7b8 05a08450 00000000 wpfgfx_v0400!CD3DSubGlyph::ValidateAlphaMap+0xb3
0593f778 69eea0b9 0593f9dc 0000ff01 059ed288 wpfgfx_v0400!CD3DGlyphRunPainter::Paint+0x601
0593f8c8 69f393ba 05a08450 0593f9dc 0593f948 wpfgfx_v0400!CHwSurfaceRenderTarget::DrawGlyphs+0x188
0593f8dc 69e06aa2 05a08450 0593f9dc 059f6620 wpfgfx_v0400!CHwDisplayRenderTarget::DrawGlyphs+0x1a
0593f9a0 69e0697c 00c34e28 0593f9dc 05a0c4f8 wpfgfx_v0400!CMetaRenderTarget::DrawGlyphs+0xe6
0593fa00 69dfc7f3 08588bf8 00be33e8 085baf50 wpfgfx_v0400!CDrawingContext::DrawGlyphRun+0x30d
0593fab0 69dff4f7 05a0c4f8 05a0c4f8 085baf50 wpfgfx_v0400!CMilSlaveRenderData::Draw+0x43f
0593fac4 69df632c 05a0c4f8 0598e188 085baf50 wpfgfx_v0400!CMilVisual::RenderContent+0x26
0593fb8c 69df639b 0593fba8 00000000 05a0c4f8 wpfgfx_v0400!CDrawingContext::PreSubgraph+0x6d9
0593fbb8 69df7602 00c05d50 05a0c510 05a0c4f8 wpfgfx_v0400!CGraphIterator::Walk+0x37
0593fc00 69e11856 00c05d50 00be116c 0593fcf4 wpfgfx_v0400!CDrawingContext::DrawVisualTree+0x230
0593fc9c 69df7e26 00c05d50 00c34e44 00be116c wpfgfx_v0400!CDrawingContext::Render+0x3f9
0593fd0c 69df7c9b 0593fd33 00c3de28 0593fdcb wpfgfx_v0400!CSlaveHWndRenderTarget::Render+0x1a0
0593fd34 69df1a72 0593fdcb 00c3de28 00000000 wpfgfx_v0400!CRenderTargetManager::Render+0x34
0593fd4c 69df1906 0593fdcb 00c3a5a0 00c3de30 wpfgfx_v0400!CComposition::Render+0x1f
0593fdb4 69df199a 0593fdcb 00c3de30 00c3a5a0 wpfgfx_v0400!CComposition::ProcessComposition+0x12c
0593fdcc 69df19d7 0593fde7 00000001 00c3a5a0 wpfgfx_v0400!CComposition::Compose+0x3e
0593fde8 69df17e1 00c3de30 00be03e8 00c3a5a0 wpfgfx_v0400!CPartitionThread::RenderPartition+0x1b
0593fe00 69e0aee1 00000000 00000000 00c3a5a0 wpfgfx_v0400!CPartitionThread::Run+0x48
0593fe18 7734495d 00c3a5a0 0593fe68 77b098ee wpfgfx_v0400!CPartitionThread::ThreadMain+0x1c
0593fe24 77b098ee 00c3a5a0 cb763808 00000000 kernel32!BaseThreadInitThunk+0xe
0593fe68 77b098c4 ffffffff 77afe0cf 00000000 ntdll!__RtlUserThreadStart+0x20
0593fe78 00000000 69e0aec5 00c3a5a0 00000000 ntdll!_RtlUserThreadStart+0x1b

As you can see (read stack from bottom to top), there appears to be some drawing happening and finally ends with igdumdim32.

0:009> lmvm igdumdim32
start    end        module name
64550000 64cdd000   igdumdim32 T (no symbols)          
    Loaded symbol image file: igdumdim32.dll
    Image path: C:\Windows\System32\igdumdim32.dll
    Image name: igdumdim32.dll
    Timestamp:        Mon Jun 24 21:16:07 2013 (51C869BF)
    CheckSum:         00000000
    ImageSize:        0078D000
    File version:     9.18.10.3220
    Product version:  9.18.10.3220
    File flags:       0 (Mask 3F)
    File OS:          40004 NT Win32
    File type:        2.8 Dll
    File date:        00000000.00000000
    Translations:     0000.04b0 0000.04e4 0409.04b0 0409.04e4

Basically, igdumdim32.dll caused an access violation when trying to execute instructions from a non-executable address in the memory.

Case of crash – explained!!

Ajith Kumar

Support Escalation Engineer

How To Tackle Performance Issues (For System Center products like ConfigMgr / OpsMgr):

---------------------------------------------------------------------------------------

The intention of this blog is to help troubleshoot 'performance issues' on System Center products or at least identify where the bottle neck is from.

First of all, don't get carried away by the name 'performance issues'. The term performance can be interpreted in many ways.

Explain what he / she sees as performance issues. Always get clarity on which action is producing the so called 'performance issues'.

Mostly, performance issues should revolve around local server or SQL.

For Configuration Manager (ConfigMgr), there is a huge dependency for WMI as well. Normally there is an indication about performance issues with backlogging in the inboxes.

For Operations Manager (OpsMgr), just evaluate local server and SQL. Normally there is an indication about performance issues with Events like 2115.

Let's drill a bit down to local server first.

A server can experience sluggishness because of primarily the following 4 parameters.

1) Disk bottleneck

2) Memory starving

3) CPU over utilized

4) Network congestion

Other than this, you could also look at the possibility of WMI repository bloating. (chances are very less for this case though).

A quick check of the size of the repository folder could let you know. If you see that to be like 300 MB or greater than that, it's likely that something is not right there.

Engage windows performance experts if that's the case. (Do NOT blindly rebuild the WMI repository).

Also, look at the WMI memory quota if that is getting exhausted (especially for ConfigMgr provider / console).

Mark Ghazai had written an excellent post about that. - http://blogs.technet.com/b/askperf/archive/2008/09/16/memory-and-handle-quotas-in-the-wmi-provider-service.aspx

We will explore each of the above 4 parameters one by one here (Again, no intention of deep troubleshooting here).

All of these can be covered by taking a perfmon trace while the issue is being reproduced.

'How to take perfmon?'

Go to Start, Run and type "perfmon" (without quotation marks) and press OK.

Expand Data Collector Sets | User Defined

Right-click "User Defined" and select New | Data Collector Set

Give it a Name and click Next (make sure "Create manually (Advanced)" is selected)

Click Next

Choose “Create data logs” and enabled the checkbox for “performance counter”

Click Next

Click on the “ADD” button and choose all the below ones.

Please make sure that “All Instances” are selected.

               Cache

               Logical Disk

               Memory

               Network Interface

               OpsMgr*  - - For SCOM

               Paging File

               Physical Disk

               Process

               Processor

               Redirector

               Server

               SMS* - - For SMS

               System

               SQLServer:Buffer Manager - - For SQL server.

               SQLServer:Memory Manager - - For SQL server.

In the “Sample Interval”, please choose “10 seconds” and click Next.

Choose the location where we want the file to be saved and click Next.

Choose “Start this data collector set now” option and click finish.

Always keep in mind that perfmon logs are good for long-term trend analysis only.

That means, a live perfmon may not really be the proper way to catch something. Keep it running for a prolonged time and then do the analysis.

Once you have the perfmon logs, you can do a quick analysis by yourself to see if you can catch something obvious. I have pasted some sample perfmon graphs (not all) for better understanding.

Disk

-----

There are a couple of counters that are very important to look at when you suspect disk. The complete analysis / review of all of them is much beyond the scope of this session, however I've tried to put some of those to begin with.

Look at logical volumes versus physical disks. Having multiple logical volumes on a single physical disk doesn't do any good as far as performance is concerned.

Review the %Idle time for the disk. The greater the idle time, the merrier. :)

Typically it has to be 70% or higher.

There are many issues that we have seen where the disk idle time would be at 0% for a prolonged period of time.

Sample:

Take a look at the disk E:\ idle time. This is almost touching zero ALL the time. Typically it has around 90.

This explains that this disk is super busy.

Review the Avg Disk Queue Length counter.

Typically, this will be zero. However, you may end up seeing higher values. It is not a problem if this value goes high and comes back to zero. Prolonged higher values means that those many requests are still *waiting* to access the disk.

Sample:

The above is the combined graphs of Disk Q length + Read Q length + Write Q length.

Note that the curves for all three graphs are in sync.

The values are not visible in perfmon because, multiple counters are selected. However, the Q length value had an avg of 135 and max of 1268.

That means those many requests are waiting to access the disk.

Typically this value has to be less than double the number of spindles on the disk.

Review the Avg Disk sec / Transfer counter.

This counter reflects what an application is seeing because this tells the time taken for each i/o completion.

Typically, this has to be around 0.00x or atleast 0.01x.

Minute analysis of this requires you to know more details (write cache enabled controller, type of disk like SAN etc). That is beyond the scope of this.

For system center engineers, just look at this value and see if this is high.

Sample:

In the above graph, Disk sec / transfer value is avg 0.2 and max of 2.25

This is definitely a very poor value. Clearly, the disk is not capable enough.

Review the Split IO/Sec counter.

Normally, this will be close to zero. If not, that disk could be fragmented / low free space or NTFS block size was small.

OpsMgr does not create many small files, however this could be acceptable for around 5 or anything less than 10 for ConfigMgr servers because there will be too many small files under the inboxes.

Sample:

As you can see above this has an avg of 101 and max of 274. This volume could use a defragmentation to be run.

Memory:

-------

Review the Cache Bytes counter.

See if there is a fluctuation in cache bytes. That typically is a sign of disk bottleneck.

Sample:

Review the Available MB counter.

This way you can find out how much of RAM is available still.

No hard and fast rule on this, but typically we need 5% of total RAM to be free.

On 64 bit servers, you would need atleast 250 MB of this ALL the time, but starving condition is around 100 MB.

Starving condition is 4 MB on 32 bit servers.

Here is a sample to help understand how the available MB is proportional to the Total Working Set of processes.

As you can see when the working set goes up, the available MB goes down as expected.(Due to perfmon scaling, the top line is for working set and the bottom line is for available MB)

Rest of the memory counters are not generally needed by System Center engineers unless going in-depth.

You can also take memory dumps of offending process or the entire system if need be for further debugging.

Processor:

----------

Review the %processor Queue length counter.

This shows the number of threads in the processor queue. A sustained processor queue of greater than two threads generally indicates processor congestion.

Look for longer durations with higher queue values.

Review which process is taking the maximum spike and determine next steps from there.

You can use Process Explorer to drill a bit down on what is happening with the process. Further troubleshooting is beyond the scope of this blog.

Review the %processor time counter.

This counter lets you know the usage of processor time. You need to consider any sustained CPU time over 90%.

Sometimes, you may want to add the ‘Threads’ counter in perfmon in some cases. Do it ONLY in case of troubleshooting CPU spike issues, not otherwise.(Data will be extremely huge if you take this).

Sample:

If you note the above graph, the top most line in black is the %Processor Time. As you can see, it’s always at 100%.

Also note that there are gaps in between. The reason is because perfmon is a non-invasive process, so it would not collect the data if the system is extremely busy with CPU cycles.

Review the %Privileged time counter.

Privileged time is the time spent doing kernel work.

Sample scenario of smsexec.exe process consuming CPU cycles.

In this graph, I’m painting both the SYSTEM and SMSEXEC’s privileged time usage. The reason you should evaluate SYSTEM as well is because SMSEXEC is a user mode application. SYSTEM is the only process that you would see in the task manager that actually runs in the kernel. and oh! It always will have the PID of 4. (just fyi).

Network:

---------

Review the Output Queue Length counter.

Generally this would need investigation if this value is greater than 5.

Review the Packets Received Discarded counter and the Packets Received Errors counter.

Generally this suggests hardware problems or driver issues.

You should be able to at least identify where the bottleneck is from based on the above by now.

Let’s now get back to the SQL server.

Remember to take the normal perfmon counters + include two more SQL counters (MSSQL$:memory manager, MSSQL$:buffer manager) as given above.

SQL performance is again a very vast topic and will go way beyond the scope of this blog.

There are a couple of very basic things that we need to be aware of....

SQL server will consume almost ALL the memory that the operating system has.

This is not a bug, this is normal.

We should do the capping properly (of course, leave memory of OS as well. There is no pre-defined rule as far as I know, but will need to be observed based on perfmon baselining. On the other hand, it is generally safe to cap SQL with 6 GB max memory if the OS has total 8 GB of RAM considering there is no other heavy application).

The Operating system is also designed to trim the working set of SQL to gain back memory if there is a demand to the OS.

This will have a direct impact on SQL server performance. (page faults).

Use the following query to get information about SQL System Configuration:

Select * from sys.configuration

For SQL server as well, do the perfmon analysis like you did for the server.

Additional SQL specific things to consider for basic troubleshooting is as below....

Review the Page Life Expectancy counter.

This tells the time in seconds that a page will remain the buffer pool.

Higher values are good. Typically, this should not drop below 300.

Review the Total Server Memory and Target Server Memory counters.

These tells how much memory SQL Server is using for its cache.

Review the Buffer cache hit ratio counter.

This shows the pages in percentage that is in the buffer pool (ie.. No need to do a read from disk).

100% is excellent, but anything less than 80% is not good enough.

It simply means that SQL needs more memory.

Check the contig level using the following query:

Dbcc showcontig with fast

Once you have the output of the above query, look at Scan Density. Ideally this has to be greater than 80. Also look at Logical Fragmentation for that.

Troubleshooting techniques typically involve looking at the large tables and then identify the scan density and logical fragmentation of that table.

Kevin Holman has written a very great article on 'Useful SQL queries'. I normally use this to copy the queries while troubleshooting.

http://blogs.technet.com/b/kevinholman/archive/2007/10/18/useful-operations-manager-2007-sql-queries.aspx

For instance, in ConfigMgr if you see issues regarding one particular component (like state messages) then evaluate tables associated for that.

For instance, in OpsMgr, if you see issues with one type of workflow (like Alert), then evaluate tables associated for that.

Once you identify the table which is fragmented, You can choose to re-index just that one table or the entire database depending on the output.

This can really have a *significant* performance boost.

Sample example output of the query for a table named 'Test': D

BCC showcontig ('dbo.Test')

DBCC SHOWCONTIG scanning 'Test' table...

Table: 'Test' (546100986); index ID: 1, database ID: 8

TABLE level scan performed.

- Pages Scanned................................: 12360831

- Extent Switches..............................: 6491845

- Scan Density [Best Count:Actual Count].......: 23.80% [1545104:6491846]

- Logical Scan Fragmentation ..................: 50.04%

From the above, we can see that the scan density and the logical fragmentation values are poor.

This clearly indicates that a re-index of this table is needed.

Sometimes updates stats may also be needed. That improves the query plan and ensures that query is with latest statistics, but on the expense of having to recompile the query.

Query to update statistics:

Use <dbname>

Go

exec sp_msforeachtable 'update statistics ? with fullscan'

Go

Query to rebuild indexes:

Use <dbname>

Go

Exec sp_MSForEachtable 'DBCC DBREINDEX (''?'')'

Go

Another major parameter which is often neglected is to review the Maximum Degree Of Parallelism (maxdop).

This setting is to tell SQL how many processors to use for query execution.

MAXDOP setting of “0” means all available processors will be used. A setting of 0 may be fine for servers with eight or fewer processors.

For servers with more than eight processors, the time it takes SQL to coordinate the use of all processors may be counterproductive. So for servers with more than 8 processors, don't leave it to 0. Change it to 8.

For Configuration Manager, we don't do much of parallel processing. So, keeping this value to 1 or the total number of *physical* processors should be fine.

For Operations Manager, we may need parallel processing especially for Data Warehouse database. So, keeping this value to 8 or the total number of *physical* processors should be fine.

Note that we are *not limiting* SQL to use only one CPU here instead of all the CPU’s, but this is a mechanism to tell SQL how many CPU to use for executing any given single query.

You could also use the following query to find out what is the optimal maxdop setting for SQL.

select case

when cpu_count / hyperthread_ratio > 8 then 8

else cpu_count / hyperthread_ratio end as optimal_maxdop_setting

from sys.dm_os_sys_info;

What other than these???

Process Explorer is an excellent tool for troubleshooting CPU spike issues. You can look at which thread is consuming CPU cycles on a process and even look at the stack of that thread.

While troubleshooting performance issues, we take memory dumps as well for digging in-depth should the need arise.

Often, many admins come up with a question on when to take memory dumps? What criteria?

To understand about it, a bit of how windows works in the background is beneficial.

To keep it short, just understand, all that a process sees is the Virtual Memory. Windows does the manipulation of physical memory to virtual memory and vice-versa.

When a memory dump is captured for a process, that is the actual virtual memory of that process.

Two types of memory dumps you may want to know about. (hang dumps and crash dumps).

When you suspect something is leaking or if you suspect the application is not doing what it is supposed to do, you tend to go for a hang dump.

When a process is terminating abnormally, you tend to go for a crash dump.

Use tools like adplus or procdump (my personal choice) to generate memory dumps of the suspect process, so that it could be debugged later.

BTW, if you feel perfmon analysis is complex, don't worry. Tools make life easier. Find PAL (http://pal.codeplex.com)

Hope this helps!!

Cheers!

Ajith Kumar

System Center Support.