I've spent years watching some of the absolute best problem solvers in action.  These guys/girls never cease to amaze me in their ability to slice and dice a complex problem down to its root cause in the most efficient way possible.  So when I started thinking about what personality traits make them such experts  a pattern started to emerge:

-Ability to think holistically – See the big picture in terms of impact, time restraints and value.

-Ability to visualize  the workflow – to see how systems are built upon each other to accomplish a goal.

-Familiarity with features – You need to know what you are troubleshooting.

-Knowledge of Support Boundries – Knowing what you can and can’t do.

-Knowledge of Resources – Knowledge Bases, Repro Labs…ext.

-Not Ego Driven – Knows when to ask for help or bring in other resources.

-Natural Curiosity and Skepticism – Gets that feeling when something doesn’t sound right.  Double checks things.

An expert troubleshooter uses their experience before breaking down a problem.  They first act upon any hunches they have gathered from the problem statement that relate to previous experience or similar issue.  This is a very important step since a seasoned troubleshooter may be able solve the problem quickly and efficiently. 

This is a best case scenario, but limited time should be allocated to it if the quick fix doesn’t work.  A common pitfall that troubleshooters hit is that they spend too much time focusing on something familiar and stop listening to the customer who may be giving them valuable data. 

Important Tips:

-Don’t hold onto a hunch when other options are presented.

-Don’t try to solve multiple problems at once.  Pick one and work it until you hit a road block.

-What works for one customer may not work for all customers.  Every IT environment is different and have different needs.

-Don’t give a definitive statement without data to back it up.

-Hit up your resources whenever possible.  Just because you’ve never seen something doesn’t mean they haven’t.

The first question that needs to be asked after you have your problem statement is “What is the Expected Behavior?”

When dealing with a technical issue you can be dealing with one of three things.

1.       Its Broken

2.       Its Misconfigured

3.       It’s Slow

A broken application/server doesn’t work at all.  The hardware or software process constantly and consistently crashes or generates a hard fault.  A blue screen is a good example.

A misconfigured application/server continues to run, but doesn’t generate the expected behavior.  These are the hardest to troubleshoot.

A slow application/server give the expected results, but does so in an unacceptable amount of time. 

It’s Broken

When an application or piece of hardware is truly broken that means there is a bug in the software or a malfunctioning piece of hardware.  The most common symptom that is associated with a broken application is an exception.  An exception is the response you see when an error has occurred.  There are 2 types of exceptions:  1^st and 2^nd Chance Exceptions.

Think of it like taking a test at school.  You spend days studying for the test and programming your mind.  When you take the test, that programming successfully gets you through a few questions until you get to one that is completely unfamiliar. 

At that point you can either skip the question and come back to it or completely freak out, quit the test, quit school and move back in with your parents.  A crash is a second chance exception.

A first chance exception would be skipping the question and coming back to it.  You may get the question the second time around or you may freak out again. 

A 2^nd chance exception can also be referred to as an “Unhandled Exception.”  When these occur we would generally expect that either a crash dump is created or an error is thrown.  These are crashes and they need a specific kind of troubleshooting.

How to determine if you are dealing with an Exception:

1.        Look for Error Events.  You could see an error in various logs including application specific logs or the default Windows Logs. 

2.       Does the Process ID of the application change?  If the PID of a process changes, that means it was stopped and restarted. 

3.       Look for a dump file.  Under many circumstances, a crashing application will dump its memory out to Dr. Watson.

4.       There are rare circumstances where you see a “silent exit” in while a process terminates without generating any errors or dump files.  These are the most difficult to find.

5.       Is there a memory leak crashing an application?

Troubleshooting an Exception or a Crash.

Once it has been determined that there is a genuine exception or crash, there is only so much troubleshooting that can be done outside of code debugging.  This debugging can be done through very specific logging or memory dumps.  Memory dumps are the preferred method for Root Cause Analysis.  Therefore, the first goal is generate a dump file if one wasn’t automatically generated.  You can also get an iDNA trace if the process is crashing on startup.

Configure Dr. Watson to generate full memory dumps.

Start-run -> drwtsn32.exe on 32 bit machines. 

Download the debugging tools for windows in you need to take specific dumps.

https://www.microsoft.com/whdc/devtools/debugging/default.mspx

You also NEED performance monitor data when debugging is many situations.  It is always better to have too much information than not enough and often, this data is critical for finding root cause.  Take the extra time to setup a Counter Log with All Counters and Instances. 

Other Options

If the crash is reproducible, you may be able find the problem based on what is happening right before the crash.

Example:

In the Exchange world, it is sometimes possible for a malformed message to crash Transport.  The message will either be sitting in the Mailroot\VSI\1\Queue directory in Exchange 2003 or in the Poison Queue in Exchange 2007.  Turning up Diagnostic Logging will generate a warning or error pointing to this message.  Thus, this is an exception you can control by finding a way to prevent that message from reaching the servers.  You can remove that message from the queue and stop further occurrences  through a firewall or through built in Spam prevention.

The downside to preventing that technique is that it doesn’t give us full Root Cause.  It tells us what  caused the crash, but not why.  To get the why you may need the memory dumps as well.

Misconfigured

A configuration error is the most common cause of outages in an IT environment.  Often there are so many people making changes that one hand doesn’t know what the other hand is doing.  Configuration errors can cause exceptions or crashes, but they generally result in the application/server not generating the desired outcome.

Problems are most easily identified as configuration related by asking the questions:

Has this ever worked?

What changed?

If the server or application has never given the expected results, then you are probably dealing with a setup issue.  These problems simply require replaying the steps the customer has done to find the problem.

If the application/server has worked in the past, then next step is to determine what changed.  This is where the bulk of IT troubleshooting is done.  There are two approaches to tackling this issue:  Discover what changed or troubleshoot based on symptoms.  Both of these approaches have distinct advantages and problems.

Finding what changed is a big picture approach.  It involves gathering all the involved parties and running through change control data.  There are utilities like MOM, MPS Reports and ExBPA to help gather change information.  The goal with the discovery process is that someone will know what changes have been made and you can determine if they are possible causes.  The problem is that under most situations, the people driving the problem simply don’t know what has changed.  If they did know, they probably don’t need your help fixing it. 

So the main advantage here is that this approach can very quickly narrow down the issue if the right people are involved.  The main disadvantage is that you can spend a huge amount of time looking for changes and not finding any.

Examples of Big Picture troubleshooting techniques are the 5 Why’s and Kepner-Tregoe.

Symptom based troubleshooting can be extremely efficient depending on the knowledge level of the people troubleshooting and access to resources.  In Symptom based troubleshooting, the goal is to take the symptoms of the issue at hand and rule them out one by one. 

For example, the age old Help Desk call of “My computer won’t start”

Possible causes:

-It’s not plugged in.

-The monitor isn’t hooked up.

-there’s a Bios problem

-probably a lot more causes that I can’t think of right now

The help desk engineer would generally start eliminating causes.  They would make sure the monitor is plugged in and then make sure the monitor is plugged in and turned on.  If the problem still persisted they would look at the BIOS.  If they found it was a problem with the BIOS, they would then start eliminating possible causes from the BIOS until they found root cause and fixed the issue.

Most engineers take this approach because it allows them quicker access to the immediate problem.  The main drawback to Symptom based troubleshooting is that it is dependent on the knowledge level of the person doing the troubleshooting.  A lot of time can be lost gathering data for things that aren’t related to the issue at hand.

Let’s say, for example, we take the “My computer won’t start” problem and say it was caused by a hardware update that is documented in the company’s change control records.  Because the engineer immediately went to troubleshooting the symptoms, they have potentially wasted a huge amount of time looking at symptoms that aren’t related to the issue.

As you can imagine, the best way to troubleshoot is a combination of the Big Picture and Symptom based approaches.

It’s Slow

A slow application/server is generating the correct results in an unacceptable amount of time.  These problems are rarely caused by an exception but can be caused because of a configuration error.  For the purposes of this paper we will concede that we are dealing with pure performance based issues.

Before you can start to troubleshoot slow applications/servers, its best to ask what it is being compared to.  Slow and fast are both perceptions that are relative without concrete benchmarks. 

However, troubleshooting slow systems has a specific troubleshooting path:

Establish a Performance Monitor baseline on the problem server and another server to isolate differences.  Through this, you can find whether you are dealing with a disk IO issue, network issue, memory issue and so forth. 

Every good problem solver needs a good Broadsword technique in their arsenal.  A Broadsword fix is method that eliminates several potential causes in one fail swoop. 

A good example of a Broadsword in the Exchange realm is what we dubbed “The Ninja Trick.”  The Exchange 2000/2003 metabase file is easily corrupted by disk issues or anti-virus.  Once that corruption occurs, the fix is to fully uninstall IIS (removing the metabase) and then reinstall IIS and Exchange over the top.  Depending on how available the customer’s media was, this process generally took under 90 minutes.

What we discovered is that the Ninja Trick solved all sorts of issues that weren’t necessarily metabase related.  That is because the entire process not only reset the corrupted file, but it also reset the core Exchange files, removed any hooks into Transport and reapplied service packs.  This essentially made the technique a catch all for a large number of common issues that customers were seeing. 

Other basic examples of Broadswords:

-FFR (FDisk, Format and Reinstall)

-Migrating Users/Data off a problem server

-Re-installing (Starting Over)

-Replacing Hardware

Time vs Results

In today’s IT culture time is at a premium.  SLA’s continue to demand greater uptime as High Availability solutions aim to make disaster recovery seamless.  However, as these environments become more complicated with more moving parts, they often become harder to fix when things go wrong.

Broadsword techniques are valuable in some situations but they often cause unexpected problems as well.  The Ninja Trick solved many tough cases, but any customizations that were made to the server were removed when it was used.  Anti-virus, webpages, disclaimer software and various other things needed to be reconfigured to get the server back into full production.

So the question becomes:  “When is it appropriate to use a Broadsword?”

The answer is:  When the problem will take longer to troubleshoot and fix the issue then to use the sword and clean up the collateral damage.

A friend of mine recently ran into a major problem with a High Availability solution he had designed for his environment.  The requirements of his solution were:

1.       Migrate his 160 users from Exchange 2003 to Exchange 2007

2.       Provide uninterrupted email service in the event of a server failure

3.       Provide High Availabilty to a critical folder which shared financial data

4.       No single point of failure

5.       $40,000 budget including software and user CALs.

Our solution was:

1.       2 new HP DL180’s with 500gb of storage in a RAID 10 and 8gb of RAM

2.       2 copies of Windows 2008 STD and 2 copies of Exchange 2007 STD

3.       CA XOSoft HA and ArcServe backup

4.       1 tape backup

5.       Final price was just over $40,000

In this solution, XOSoft would be responsible for replicating the Exchange data and file share between the nodes.  In the event of a server failure the XOsoft software would fail the solution to the second machine which contained all of the current data.  Arcserve would provide backup.  This solution filled all the requirements for the project.

However, after everything was built and testing began he found 2 major issues.  Failover between the nodes took over 30 minutes and one of the nodes sporadically rebooted itself without warning.

Thus began the 3 weeks of constant troubleshooting that completely engrossed my friend and kept him from his regular duties for his company.  For the most part, he did everything right during his troubleshooting.  He tried to make the server generate warnings and errors.  He contacted Microsoft and Computer Associates to open trouble tickets.  He researched other known issues and dug through the servers with a fine tooth comb.

About 3 days after the problems started he called me to get my take on the situation.  Since XOSoft had only recently released a version that was supported on Windows 2008 I recommended a Broadsword of rebuilding the whole solution on Windows 2003.  Unfortunately, he was dead set on his original solution and continued to fight it for another 2 weeks.

I went by to help him and found that the failover problem was being caused by TCP/IP settings on the Domain Controller and that Arcserve was crashing the passive node due to an incompatibility with Windows 2008.

This case study is a perfect example of the Time vs. Results problem that many engineers and IT professionals are subject to.

My friend dedicated countless hours of troubleshooting and frustration to this problem.  He was constantly harassed by his management and fell way behind on his duties because his ego wouldn’t allow him to start over with a better tested OS version. 

People who are good troubleshooters often find it difficult to STOP working a problem until they find the solution.  They take it personally and refuse to accept that the root cause may be out of their control. 

In most cases, this is good thing!  It is that unwillingness to admit defeat that drives them to learn new things and find unique solutions to otherwise unsolvable tasks.  In fact, when hiring a technical person, that is the single most important trait that a company will look for. 

Ego is what makes a good troubleshooter.

Being able to let go of one’s ego is what makes a GREAT troubleshooter.

Take this following scenario:

There are 4 Edge servers in a DMZ.  It takes one of these servers 4 times longer to deliver mail than the other 3.  You log into the server and there is nothing obviously wrong.  You turn up logging, enable performance monitoring and monitor the server for an hour.  You find some events that may or may not be related and spend 2 hours researching them before realizing they are expected errors.  Now, you start comparing this server to the others to see what is different and again find nothing obvious. 

At this point you have probably spent over 3 hours troubleshooting an Edge server that can easily be rebuilt on the same (or new) hardware and re-integrated into your environment.  It is time to stop troubleshooting this issue and use a broadsword. 

-  There is no guarantee that spending more time troubleshooting will provide any new information.

-  The broadsword will probably take under 2 hours to complete.

-  There is minimal collateral damage in this case.

-  If the issue returns on the same hardware you have essentially ruled out software as the problem.

-  If the issue returns on different hardware you have ruled out hardware as the problem.

So by using a broadsword technique of rebuilding the server, you have either fixed the issue or cut the number of potential causes in half.  This is most likely far more than could have been accomplished by additional troubleshooting in the original state.

Downsides

There are other potential downsides to using a Broadsword.  The main one being that it makes definitive Root Cause Analysis impossible or far more difficult.  This is usually the case when multiple changes are made at the same time.  In some situations it is more important to get a fix and get it fast.  While in others the need to understand why the issue occurred in the first place takes precedence. 

So whether or not RCA is required should always factor into the decision to make a sweeping change. 

1.       Be polite.  When you join an existing conference call you will be the new person.  Some people on the call may have been on it for hours and are exhausted, while others may be looking for reassurance from you.  Some may even be looking to discredit you so it’s important that you don’t step on any toes.  Make sure everyone knows that you are there to help.

2.       Announce yourself.  When you join, tell everyone who you are and what your role is.  Be specific and try to speak with authority. 

Depending on your particular conversation style and the nature of the call, this can be a good time to break the ice.  Sometimes, asking a simple question like “How’s everyone doing?” can elicit an offhanded remark or joke which can be a good segway into gathering background information.

3.       Get the Lay of the Land.  Asking for background about the problem here is an effective way to start scoping the issue and discovering who the main players on the call are. 

Write down names and what they do.  I can’t stress this enough.  It's polite and when you need something from that person it cuts down on confusion.  Repeatedly having to ask for the Networking Guy’s name shows the whole call that you’re too lazy to remember.

Get as much background as you can.  This is the best chance to scope the issue and have everyone on the call agree to it.  If there is disagreement here between other people on the call it isn’t necessarily a bad thing.  The more they talk and the more you hear the story from a different perspective the more information you have.

Find out who is leading the call.  A large conference call needs to have someone running it.  It may be you and if that’s the case you probably need to know.

4.       Set the agenda.  Once you have the scope of the issue make sure everyone on the calls knows what it is.  When you are trying to solve a technical problem the people involved need to know that is purpose of the call and nothing else.

5.       Pre-Troubleshooting cleanup. 

Get rid of static or noise on the line.  If somebody’s line is generating static or there are dogs barking the in background you can politely ask them to mute their phones.  Let people on speaker phones know if you can’t hear them.

Often, the biggest deterrent in a technical conference call is having to give updates and explain the details of what you’re doing to non-technical people.  In an ideal situation, there will be a TAM or other liaison who will give those updates to the people who need to know. 

Keep in mind that all the members of the call have a reason for being there and may feel a need to understand the discussion.  However, continually having to stop the troubleshooting process to update and explain the situation often does more harm than good.  If phrased correctly and politely, it isn’t bad to ask the managers to exit the call or mute.  The best case scenario is they let the tech people be tech people.  The worst thing that can happen is that they say no.

Conference Call Mistakes

1.        The Mute Button.  If you go on Mute, make sure it doesn’t blare music to the rest of the call.

2.       Eating/Drinking.  If you are going to eat or drink while on the call, ensure you are properly muted.  Nobody wants to listen to somebody else eat a bag of Cheetos and swig a Mountain Dew.

3.       Disappearing from the call.  There are some pretty good reasons you need to drop off the call.  Just make sure everyone knows you have left and give an estimation of when you will come back.  When you return, announce that you’re back.

4.       The Blame Game.  You know it’s a vendor’s problem.  The vendor knows it too but won’t admit it.  Don’t get into a word fight.  Find a way to get evidence. 

On a similar note, try not to make anyone at the customer’s organization look bad.  The vast majority of technical problems are caused by someone making a change to the infrastructure and not realizing or admitting it.  Assigning blame is for the managers.  Fixing the problem is for the engineers.

5.       Veering off the road to resolution.  Always remember the problem statement.  If you find other issues that aren’t relevant to your problem don’t get distracted trying to track them down.

https://msexchangeteam.com/archive/2004/03/10/87622.aspx

https://support.microsoft.com/?kbid=842026

The following link shows a very comprehensive guide to troubleshooting Link State:

https://technet.microsoft.com/en-us/library/aa996974(EXCHG.65).aspx

However, one of the most common reasons we see Routing Group Propagation failures is because of a “Deny” being set on a group in AD.  The guide above gives a quick blurb:

“If the routing group master or any routing group members are missing the SendAs permission, WinRoute will show the server as Am I connected to the Master?: NO. Verify that this server or the groups it belongs to are not explicitly denied the SendAs permission on the routing group master.”

 But this doesn’t give you an effective way to see if this is your problem which is the purpose of this blog.

The easiest way to determine if you are dealing with a permissions issue is to:

1.        Take a Winroute.exe snapshot of the Routing Group Master and verify that it can’t connect to itself. 

2.       Get a quick Regtrace.exe to verify it is a permissions issue.

3.       Walk the permissions tree until you find the problem.

Step 1

If the Routing Group Master can’t connect to itself it’s a pretty good bet you’re dealing with a permissions issue.  It’s possible that another application has grabbed port 691 and you can test that by stopping the Routing Engine and running Netstat –an.  If you see something listening on port 691 then you know you’re dealing with a port conflict.  If not, its probably permissions.

Otherwise you will expect to see something similar to the following:

 Step 2 – Take a Quick Regtrace

Regtrace is built into Windows 2003 and simply needs to be configured to gather the correct information.  This process takes about 2 minutes and can help you save hours of troubleshooting time.

Add the following registry key to enable Regtrace:

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MosTrace\CurrentVersion\DebugAsyncTrace]

"AsyncThreadPriority"=dword:00000000

"AsyncTraceFlag"=dword:00000000

"EnabledTraces"=dword:00000044

"MaxTraceFileSize"=dword:0c800000

"OutputTraceType"=dword:00000001

"TraceFile"="c:\\1.atf"

"Modules"=hex(7):52,00,45,00,53,00,56,00,43,00,00,00,00,00

This will create the c:\1.atf file which is tracing the Routing Engine and has a max size of 200mb.  It generally won't get over a few hundred k.  You can copy the above data into notepad and save it as an *.reg file if you would like to simply import it that way.

Now, restart the Routing Engine Service to gather startup information.  The “Send As” check is made on startup.

Next you want to disable tracing setting EnabledTraces and OutputTraceType keys to zero in the registry.  Then remove the Modules key.

You always want to disable regtrace when you're done with it because it does cause a performance hit.

Open the 1.atf in notepad and you will see a whole lot of gibberish.  Generally, these files need to be “fixed” and viewed with a Microsoft internal application, but for the purposes of testing “Send As” you simply need to do a search in that file for “Other side does not have full authorization”

If you find that string, you are dealing with a deny on “Send As.”

Step 3 – Walking the Permissions Tree

“Other side does not have full authorization” tells you that the Routing Group Master does not have “Send As” permission on its own Server object in Active Directory.  By default, it should inherit this permission from the Organization object.  Most often, people will add an account to the “Exchange Domain Servers” group to give it full access to all mailboxes.  However, once a user or group is added to a built in Exchange group, that group will now carry all of the permissions associated with that account.  Permissions are cumulative.  This can make identifying permissions issues difficult and ExBPA does not identify this issue.

To walk the tree you will use a combination of ADSIEdit and Active Directory Users and Computers.

Start by getting the permissions on the server object of the RG Master.

If you’re lucky, you’ll find either an explicit or inherited deny on “Send As” for either the Exchange Domain Servers or ServerNameC$ account. 

If you find it there, simply remove it, restart the Routing Engine and wait.  If that doesn’t allow the master to connect you need to continue walking the tree.

If the permission isn’t obvious, I generally start by pulling group membership starting with the server in Active Directory Users and Computers.

The server should only be a member of Domain Computers and Exchange Domain Servers.  In this example there is an Exchange 2007 environment so it is also a member of ExchangeLegacyInterop.

Next, double click on Exchange Domain Servers and check its members.  You should ONLY see Exchange 2003 servers here.  Exchange Domain Servers should only be a member of Exchange Enterprise Servers and Pre-Windows 2000 Compatible Access.  Exchange Enterprise Servers should only have Domain Servers as a member and shouldn’t belong to anything.  This is the most common place that permissions get misconfigured.  If there are other users or groups here, that is most likely the problem. 

You can verify this is the problem by going back to permissions on the server object in ADSIEdit.  Find the group or user that was added and see if it has “Send As” permission.  If it’s a user, you should see it immediately.  If it’s a group, often you have to check membership of that group and look up each one.

I will often just highlight the top listing in the security box, scrolling the bottom box down to the “Send As” right and then checking each user/group. 

If “Send As” is an inherited Deny you can use the Advanced Tab to see where it is inherited from:

In this example you can see it is inheriting the deny from the “First Administrative Group” level.

Hope this helps!

Why Troubleshooting?

For many IT Professionals and Support Engineers, the ability to effectively troubleshoot an issue is becoming a lost art.  And why shouldn't it be?  Exchange 2003 has become a very mature product where the odds of running into a problem that no one has seen are slim.  Exchange 2007 is quickly approaching maturity as well with SP2 and the 2010 on the horizon.  In truth, most problems can be quickly solved by a quick search through the Knowledge Base or a Bing query.  After all, that's what they are there for.  So what happens those queries come up blank?  It could be that you've come across a brand new problem.  It could be that you didn't query on the right words.  Most often though the answer is that you don't really know the nature of the problem you are dealing with.

The Importance of a Clear Problem Statement

When a support incident is opened at Microsoft, the most important question that gets asked is:  "What is the problem you are experiencing?"

Everything that happens with the call, direction and troubleshooting from that point on is directly related to that answer.  A vague problem statement can, at best, get you routed to the wrong group where you have to go through the whole definition again.  At worst, it can completely confuse the people who are there to help and send the case in the wrong direction.  When that happens you can expect the time to solution to at least double or triple.  Nobody wants that.

I recently worked a long running, high profile case where the problem statement was "Issue with Exchange Server."  I jumped on a conference call with the customer and the topics of conversation were ranging from message traffic, to server performance to Outlook service packs.  There was no real plan or direction for the troubleshooting to go because the initial statement was so vague.  Of course, without a plan there's no attack.  With no attack, there's no victory!  (points to whoever gets that reference!)

The very first thing we did was to hash out the actual problem the customer was experiencing.  In this case, it was similar to "All Outlook Clients Randomly Disconnect From 3 Exchange 2003 Active/Passive Clusters." 

Things to note in that problem statement: 

1.     ALL Outlook Clients.  Not a particular version.  Not a particular segment on the network.  Not OWA clients. 

2.     "Randomly Disconnect." - The issue is not reproducible at will and transient in nature.

3.     This affects multiple Exchange 2003 servers. 

4.     No mention of errors or warnings because there weren’t any.

This is the kind of issue where you can’t just plug an error code into a browser and hope for the best.  For this issue you have to know how to troubleshoot.

Based on that problem statement we were able to draw up a solid action plan.

Since its all clients, that narrows us down to Network or something Server side.

Since the clients randomly disconnect, that further indicates Network or something Server Side.

Since it affects multiple servers, that indicates Network or Active Directory Connectivity.

After knowing these details the action plan became simple:

1.     Concurrent Netmon traces of a client and server while the issue was occurring.

2.     All performance counters from the servers while the issue was occurring and before that to establish a baseline.

3.     Memory dumps of Store.exe and Lsass.exe while the issue was occurring.

I won’t get into specifics here since this blog is focused on defining the problem.  However, from that data we were quickly able to identify the problem as an overworked Global Catalog server and implement some changes to increase performance. 

I know this whole blog sounds simple and a bit like a self-help book, but my experience is that long running problems are most often ill defined at the outset.  When you define a problem statement, you have to be precise and clear. 

Who or What is being specifically affected?

How are they being specifically affected?

How often are they being specifically affected?

Its fine if you don’t know immediately know the answers to these questions.  Many times you won’t.  Getting the answers is part of the troubleshooting process.  The point is that when troubleshooting a problem the absolute first thing you have to do is agree to a clear, well defined problem statement. 

More to come later! 

 A Mail Storm occurs when Exchange cannot process the mail it has queued fast enough to also deal with the incoming load.  The queues will show thousands of messages destined for various remote and local recipients.  You may see all, or any combination of following queues backed up:  Pre-Submission, Messages Awaiting Directory Lookup, Local Delivery, Messages waiting to be routed, and Remote Delivery  Queues.  The basic, overall symptom is that while you do see some mail processing, you will also see these queues continue to grow.   You can think of this as a situation where more load is put on a Exchange Server  than it's suppose to handle,  i.e. an overloaded Exchange Server . 

There are numerous things that can cause a Mail Storm.  Here is a quick list of some of the more prevalent ones:

-A Spam attack or Open Relay within your Organization:  Spammers aren’t nice people.  If you unwittingly leave an Open Relay that is discovered you can expect hundreds of thousands of spam messages to be bounced off your servers.  Also, always make sure your Spam definitions are up to date on whatever type of prevention system you are using.

-A looping message:  A misconfigured Outlook Rule, Contact or Transport configuration is often the cause.   A quick look at the actual messages in the queue usually determines if this is the case.

-A Public Folder Replication Storm:  In an Organization with a large number of Public Folders, if a new Public Folder Store is brought up with replicas added then data is going to be sent via email.

-An unknown internal application sending mass amounts of mail:  Even in some of the best IT departments I have worked with, sometimes an eager developer can configure an application that starts flooding the Exchange Server.

There are many other causes, but these are the ones we see most frequently.

The bottom line is that the Exchange Server doesn’t have the resources to handle the load and the queues will grow.  I like to explain this in terms of a bell curve.  Let’s say an Exchange server has 10, 512kb messages being processed.  I would expect Exchange to process each message in 0.1 seconds or faster.  Now let’s say that your server has 8000 512kb messages being processed with more coming in.  In that scenario I would expect Exchange to slow to ~.5 seconds /message.  Why?  The more messages that are queued, the more resources Exchange has to use to keep track of them all.  Of course actual processing time will vary based on hundreds of variables such as hardware and message content.

There are several common bottlenecks that we see that contribute to the overall speed of SMTP transfer.  Those are Disk I/O, Memory and LDAP latency.

When you remember that Exchange is just an Application at its core it becomes easier to understand why Disk Latency has become #1 bottleneck in SMTP throughput.  Exchange writes/reads from a disk in almost every action that it takes during message delivery:

- Inbound mail is written to the “mailroot\vsi 1\queue” directory.

- Message Tracking Logs are written to.

- Transaction logs and databases are read and written to.

- The mysterious Information Store “Working Directory.”

Everything that Exchange does relies on the disk subsystem.  To check the health of the disks, simply use Performance Monitor to look at the following counters:  Object: Physical Disk, Counter:  Avg Disk sec/Read and sec/Write.  The value that you get there is how long it takes the OS to read or write a binary 1 or 0 to the drive.  This value should never be over 0.02 and if it is, you have a bottleneck.  How long would it take you to write a “1” on a piece of paper?

Memory only becomes a significant bottleneck when the server runs out and begins to page.  Memory allocation is generally thought of in these terms:

Each member of a flat Distribution List requires 1 KB of Inetinfo.exe RAM and more if nested. 

The Scalability Guide references that vanilla Exchange 2003 will use the following memory allocation for SMTP messages:

·         1,000 open messages=10 MB of InetInfo memory

·         1,000 open messages + 20,000 closed messages = 80 MB of InetInfo memory

·         1,000 open messages + 89,000 closed messages = 366 MB of InetInfo memory

The number of open messages is configurable though the MaxPendingCat registry key: 

https://msexchangeteam.com/archive/2005/04/04/403297.aspx

On a heavily used server, AD Connectivity can easily become a bottleneck.  Exchange makes dozens of LDAP queries to DC/GCs during mail routing so you need to make sure that those servers are not only responding, but responding quickly.  Again use Perfmon and look at:  Object: MSExchangeDsaccess Process, Counter: LDAP Read and LDAP search.  These values shouldn’t average over 20ms and you should not see consistent spikes over that.

Knowing your system bottlenecks is a good way to help speed up message flow, but once you are in a Mail Storm you have 2 primary goals:  1.  Find out what caused this sudden influx of messages and 2.  Get the mail delivered. 

So what can be done to recover from a Mail storm?  Here is a list of the procedures I have developed over the years:   

1.        Regardless of the Exchange version the very first thing you need to know is what is causing the mail storm.  The most effective way to do this is to simply look at the messages that are queued and look for commonality.

-          Are they to or from the same person?  If so, disable or disconnect that mailbox.  Side Note:  NEVER disconnect an SMTP or System Attendant Mailbox.

-          Are they Spam?  If so, configure your gateways to catch it.

-          Are the Public Folder replication messages?  Disable replication.

-          Do they have the same size or subject?  Just another way to track it down.

You have to stop the incoming messages.

2.        Your next goal is to free up as many resources on the effected servers as possible.  Old mail can always be replayed when the storm is over.  Remember, the fewer messages Exchange has queued up, the quicker it will play them.

Here is my tried and true action plan for Exchange 2003

Assuming you have identified what is causing the Mail Storm you can now deal with the messages that are queued.

A.       Free up resources.  Disable Anti-virus by setting all of the AV services to “disabled” and reboot.  That is the best way to clear it out of memory.  Since calling anti-virus is the last thing Exchange does before submitting a piece of mail, often Exchange 2003 has to wait for AV to hand it back.  Plus, it takes up considerable amounts of RAM and Disk usage.  If you are protected at the gateway and client levels the risk is usually acceptable.  However, every situation is different so use your best judgment.

B.      Give Exchange a fresh queue.  Any mail that comes Exchange 2003 off the wire (SMTP) is first written to file system.  Hopefully, the vast majority of the queued mail is in that directory.  Stop the SMTP Service and navigate to the Exchsrvr\mailroot\vsi 1 directory.  Rename the Queue folder to Queue.old and create a new Queue folder.  When you start the SMTP service, monitor and make sure it is keeping up with incoming messages.

C.      Delete unwanted mail from the Queue.old folder.  If you have identified the problem message, you can do a simple windows search within the messages of that folder to bulk select them.  Then either delete them or move them somewhere else.  You should be left with only the good production mail.

D.      Replay the old messages in smaller chunks.  I usually start with 100 messages and then adjust from there.  There are 2 methods for this.  First, you can simply cut and paste from Queue.old into the Pickup directory.  This is the fastest, but you lose any recipients that were on the BCC line.  Secondly, you can stop the SMTP service and paste messages into the new Queue folder.  When you start SMTP it will resend those messages preserving all of the headers.

It’s all about staying in control of the mail flow.  There is a lot of busy work, but you will make it through.

Developed and Written By David Michael and Mohammed Nadeem