The first question I ask when I get these specific cases is "Does the admin account have a mailbox"?

 In pretty much every case that I can remember, the response is "No".  And immediately, I know exactly where to look!

A little history…

 As you may know, because of the design of Exchange 2013 - the CAS role simply locates your mailbox, then proxies the request back to your mailbox server - an "anchor mailbox" is used.  This anchor mailbox is simply your Exchange GUID, and is used for a number of reasons in Exchange 2013 connectivity.  The most obvious, though, are:

• As mentioned above, so that an Exchange 2013 CAS knows what mailbox server to proxy a request to
• To prevent the 'Your administrator has made a change so you must restart Outlook' message that you get when a mailbox is moved to a different site.

You can see evidence of the anchor mailbox when you configure an Outlook profile for an Exchange 2013 mailbox:

Notice the ExchangeGuid is used as the 'Server' in the outlook profile, instead of an actual server name.

OK that's great and all, but what does that have to do with this status 500 that I'm getting?

 Well, let's think about it.  If the Exchange 2013 CAS needs to know where to proxy a request, but the request is coming from an account that has no mailbox, and thus no ExchangeGuid associated with it, how does Exchange know which mailbox server to proxy?

 In instances like this, Exchange 2013 uses the ExchangeGuid of system mailboxes to determine where to proxy the requests.   In this specific scenario, Exchange uses the system mailbox SystemMailbox{bb558c35-97f1-4cb9-8ff7-d53741dc928c}

Note: When pulling up the system mailbox(es) using Get-Mailbox you must specify the -Arbitration parameter or the command will not work.

So how do I fix this?

Typically, what I've found is that this is caused by either the database with the system mailbox is unavailable, or some attribute on the system mailbox account has become corrupt or is missing. 

Many times, this issue is accompanied by another - such as users unable to connect to Exchange, and the 500 error is discovered when the admin attempts to log in to ECP to try and figure out what is wrong.  In this case, Exchange Management Shell is your friend as you will need to determine whether the database is mounted via shell. 

In other instances, I've found that system mailbox, for some reason has incorrect information, such as missing email addresses.  You can either compare it to the other system mailboxes (Get-Mailbox -Arbitration) or compare to your lab environment (I hope you have a lab!)

Finally, I've seen some companies simply deleted the database that had the system mailboxes on it.  Or delete the AD account for the system mailbox.  Or both.  This isn't cool.  Exchange uses these mailboxes for many things, including OAB generation, so it is critical to keep these mailboxes healthy!  To fix this:

If the mailbox was deleted but the AD account still exists:

Run: Enable-Mailbox "SystemMailbox{bb558c35-97f1-4cb9-8ff7-d53741dc928c}" -Arbitration

You may also need to specify the database parameter to assign a database.

If the AD account was deleted:

First, run: Setup /PrepareAD

Then run the above Enable-Mailbox command.

Of course, the /PrepareAD switch does a lot more than just create system mailboxes, so proceed with caution.

Is there a workaround?

Yup!  Since we use the system mailbox for it's ExchangeGuid, you can simply just add a mailbox to your admin account, and that should get you up and running.  I should stress again though that this is not the only thing system mailboxes are used for, so it is important to plan to fix whatever is wrong with them.

Thanks to Chris Thevaos for assistance with investigating the root cause of this issue.

What is the Queue Length setting in IIS?

Benjamin Perkins has a short blog post briefly going over the queue length and how to change it (in IIS 7), however this post was written primarily for developers of web applications.  Exchange utilizes IIS extensively for many services, so it is important to remember that while some IIS changes may be great for custom web applications, they can also severely impact how Exchange functions.

So what exactly is the queue length setting in IIS and how does it work?  Imagine each IIS application pool as a bucket.  And requests from clients are placed into each corresponding bucket. As the request is added to the bucket, the CPU takes the request out of the bucket to process it.  Now, imagine a large number of requests - more than the CPU can handle at once hitting the bucket.  Or, maybe something else is holding up the CPU, so that it can't empty the bucket as quickly as it is filling up.  This is a very high level view of how the queue length works.  As the CPU is unable to keep up with incoming requests, the new requests are added to the queue.  The request queue limit is basically, the limit on the number of requests the bucket can hold before IIS starts denying new requests. 

To put it simply, the queue length is NOT the number of concurrent requests a particular app pool can handle.  It is only a precautionary measure to prevent the server from falling too far behind in processing requests.

So should I change it?

As Benjamin mentioned in his article, "in most cases the default configuration will work best, therefore only make modifications when you are certain of the need and impact."  This holds true with Exchange.  The default queue length is 1000 for each app pool in all versions of IIS so far.  And in most cases, 1000 is plenty. 

The maximum value we typically ever recommend setting the queue length to is 10,000.

In Exchange, we typically refer to raising the queue length on an app pool as a 'band aid fix.'  This is because we know, in most cases, raising this value is unnecessary - so why should it be necessary in this particular case?

Let's consider a real world scenario:  You start receive complaints that your company's ActiveSync devices are having intermittent issues synchronizing.  After some fancy investigative research, you've determined that a single CAS server is returning 503 errors very frequently to requests.  More investigation finds that you are exceeding the queue length (more information on how to detect this below).  The system appears to be healthy - CPU is averaging 30-60%, which is on par with your previous baseline performance data and no other application pools appear to be affected - so why not simply bump that queue length up to 5000 and let those devices get their sync on?

Well - the more important question that I ask when I'm investigating an issue for a customer is, why.  Why did we all of a sudden exceed the queue length? And why are we queueing this many requests? CPU appears to not be adversely affected, so the server doesn't appear busy in that aspect.  And most importantly, what changed?

Sure, band-aiding the issue by increasing the length may resolve the impact, but when you think later down the road - will increasing the number of queued requests that we allow for this one app pool potentially cause performance issues later, further complicating the end resolution?  It's definitely a possibility. 

So, yes, you can increase the queue length - but unless you have specific guidance from Microsoft, it should generally be used to mitigate impact while you continue investigating the actual root cause of the issue.   Additionally, it should be noted that recycling the app pool will also clear the queue with little impact, but then allow the server to build that queue back up so that we can monitor and investigate the actual cause - this is probably a better method of troubleshooting the issue than simply increasing the queue length in most cases.

As a side note - recycling the app pool is different from stopping and starting the app pool:

From the TechNet article: https://technet.microsoft.com/en-us/library/cc770764(v=WS.10).aspx

"Occasionally, you may have to immediately recycle an unhealthy worker process instead of waiting for the next configured recycle. Rather than abruptly stopping the worker process, which can cause service interruptions, you can use on-demand recycling. Overlapping recycling, the default, lets an unhealthy worker process become marked for recycling, but continues handling requests that this unhealthy process has already received. It does not accept new requests from HTTP.sys. When all existing requests are handled, the unhealthy worker process shuts down."

The same goes with lowering the setting.  I recently had a customer, whom, as a precaution, decided to set the limit to 120.  Yes, that is 880 less than default.  As you could imagine, when they ended up having performance issues, this intensified the effect - and, of course, added another layer of symptoms that had nothing to do with the root cause.

How do I know if I'm hitting the limit set by the Queue Length?

There are a few different methods that you can use to determine if you are hitting the limit. 

First, when the limit is reached, the server starts denying connections to that particular app pool by returning a 503, which translates to "Service Unavailable."  So the first question would be, are you getting 503 responses?  If you are getting 401 responses, then you probably aren't hitting the limit - as 401 translates to "Unauthorized," which indicates IIS received your request, has attempted to authorize you, and was unable to.

When the limit is reached however, the server is rejecting requests before they even reach IIS.  Which means you won't see these 503 errors in the IIS logs.  These 503's occur before the IIS logs, in the HTTPERR logs.

The default location for the HTTPERR logs is C:\Windows\System32\LogFiles\HTTPERR

HTTPERR logs are the log files for the HTTP.sys driver.  As mentioned in this TechNet article:

"When you create a Web site, IIS registers the site with HTTP.sys, which then receives any HTTP requests for the site. HTTP.sys functions like a forwarder, sending the Web requests it receives to the request queue for the user-mode process that runs the Web site or Web application. HTTP.sys also sends responses back to the client."

So, when hitting the request queue limit, the request is being denied at the HTTP.sys driver level, thus logging the failure in the HTTPERR logs.  When hitting the limit, you'll normally see a long list of 503's in these logs, such as:

014-03-05 20:41:52 10.0.0.24 56086 10.0.0.25 443 HTTP/1.1 POST /Microsoft-Server-ActiveSync/Proxy?User=jdoe&DeviceId=ApplXXXXXXXXXXX&DeviceType=iPhone&Cmd=Sync 503 2 QueueFull MSExchangeSyncAppPool

Notice the "503 2"?  That's a 503.2 (503 with a sub status of 2), which translates to "Service Unavailable: Concurrent request limit exceeded"

Note, you may also see 503.3, which translates to "Service Unavailable: ASP.NET queue full".  This is slightly different - see this TechNet article.

IIS literally gives you the answer with the reason QueueFull, but be aware that sometimes this can show N/A as well.

And, of course, the logs also indicate which app pool the rejected request was attempting to access, in this case being the MSExchangeSyncAppPool - or ActiveSync - matching up with our earlier scenario. 

Well - what if I want to monitor for this proactively? 

We've got you covered!  As you may know, PerfMon offers thousands of counters that help identify baselines - and point out potential issues.  Keep an eye on the following counter:

Http Service Request Queues \ CurrentQueueSize

There is a counter for each application pool (Exchange Web Services, ActiveSync, Default App Pool, etc).  Any value that sustains greater than 0 should be investigated as this is a potential queue length limit hit for that particular app pool.

NOTE: Regarding any performance data, spikes can be normal.  You should monitor this counter over time and look for sustained values above 0; if the value is consistently above 0, then there may be something to investigate, for example - has your environment exceeded the initial server scale? Or is a user/device mis-behaving?  Etc…

In Exchange 2013, baseline performance is captured by default every 1 minute, and includes this counter.

The default location is C:\Program Files\Microsoft\Exchange Server\V15\Logging\Diagnostics\DailyPerformanceLogs.

For previous versions of Exchange (Exchange 2007 or 2010), or if you want to customize the interval or location of files in Exchange 2013, you can use ExPerfWiz to create a data collector set for you.

Thanks to Sainath Vijayaraghavan for reviewing and suggesting content.

Environment:

• Exchange 2010 SP3 RU6 migrating users to Exchange 2013 CU6 (Note it may affect other versions)
• A large number of Outlook for Mac users

Problem:

1. User, who is only a Mac Outlook 2011 user, is migrated to Exchange 2013.
2. After which, when a calendar meeting invite is received, the meeting is added to the calendar as tentative, but no message is received in the Inbox for the user to Accept/Decline/Tentatively accept it.
3. The same behavior occurs whether or not OWA or Outlook for PC is used when receiving the calendar meeting invite.
4. The same behavior occurs with meeting invites sent from any source.
5. There were no commonalities between the affected mailboxes, other than they all used Mac Outlook 2011 prior to being migrated (Different target/destination DBs/Servers)

Workaround:

We discovered that recreating the user's mailbox resolved the issue.  This, of course, is rather destructive, time consuming, and causes major interruptions for the end user.

So first, I thought - OK maybe this person previously used Outlook for Windows, and setup a Delegate, configuring calendar meeting requests to be sent to delegates only:

But, the customer was able to configure a Windows profile and confirmed, this was not set.

The second thing I suspected was CalendarProcessing (Get-CalendarProcessing <User> | fl):

https://technet.microsoft.com/en-us/library/dd298137(v=exchg.150).aspx

However, after inspecting the results, it was found that the user had the same settings as everyone else in the company, including unaffected users.

I then had the customer grab the EventHistory table for the affected mailbox.  For more information on utilizing and accessing the EventHistory table for mailboxes and databases, a great read is here:

https://blogs.technet.com/b/exchange/archive/2013/06/24/adventures-in-querying-the-eventhistory-table.aspx

The EventHistory table essentially lets you see exactly what is changing (what events occur) within a mailbox or database.  In this case, we wanted to trace what was happening to a calendar meeting invite when received by the mailbox, and whether something was deleting it, moving it, etc.

After capturing the EventHistory table, we were able to locate the invite, and it's ParentEntryId.  According to the blog above, the ParentEntryId is the folder which the item is current in.  Great!

We went into MFCMapi (https://mfcmapi.codeplex.com/), and were able to use the ParentEntryid to figure out where the messages were going!

So, it turned out, the messages were going to…the Schedule folder.  Huh?

The Schedule folder is a folder within the mailbox, located on the root (so hidden from Outlook and other clients).  It is typically a folder used for Free/Busy information - but is not intended to have mail delivered to it.  Why were we delivering to this folder?

I then had the customer collect an Extra trace.  For those of you who are unaware, an Extra trace is a tool support uses to see what the code is doing at a very high level.  Think of it as - reading the comments in code - they give you an idea of what is happening, but not necessarily why. 

But in this case, the Extra gave me gold. When the message was dropped off to the mailbox by Store Driver I could see the following:

"overridden by receive-folder table: True"

In my lab, and with other calendar meeting invites sent to unaffected users in the Customer's environment, I always saw:

"overridden by receive-folder table: False"

In the mailbox, there is a table, called the Receive Folder Table, which is exactly that - a table of message classes, and the folders that they should be delivered to.  3rd party software developers can tap into this, so that custom  message types that a user doesn't need to see, can be delivered and picked up by their software (https://msdn.microsoft.com/en-us/library/office/cc815405.aspx).  You can see this table in MFCMapi:

And Bingo - there was the IPM.SCHEDULE.MEETING class association:

So, to remove the association in MFCMapi:

Then:

After doing this, the user was finally getting calendar meeting invites delivered to their Inbox.

Unfortunately, the customer had no more mailboxes in Exchange 2010 to try and reproduce the issue - and I was unable to reproduce the issue in a lab environment no matter how many times I tried.  A quick search found only one other reported incident, and they simply used a script to remove the association - No root cause was found in that case either.

So in the end - we have a solution, other than recreating the mailbox, however no actual root cause.  All I know is that it appears to only occur to Mac Outlook 2011 users when migrating from Exchange 2010 to Exchange 2013.  Hopefully this blog post will be useful to someone facing this issue in the future!

Thanks to Sainath Vijayaraghavan for reviewing this for me!