We have seen some issues lately where application pools have been crashing.  Usually this is a brief service interruption and sometimes the users do not notice there was an issue.  There have been a few cases recently where we are seeing the MSExchangeRPCProxyAppPool crashing.  Usually you will see the following events in the System log

 Log Name:      System
Source:        Microsoft-Windows-WAS
Date:          11/8/2013 9:16:32 AM
Event ID:      5011
Task Category: None
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      Mailbox.Contoso.com
Description:
A process serving application pool 'MSExchangeRpcProxyAppPool' suffered a fatal communication error with the Windows Process Activation Service. The process id was '4356'. The data field contains the error number.

And

Log Name:      System
Source:        Service Control Manager
Date:          11/8/2013 9:16:53 AM
Event ID:      7031
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      Mailbox.Contoso.com
Description:
The Microsoft Exchange RPC Client Access service terminated unexpectedly.  It has done this 1 time(s).  The following corrective action will be taken in 5000 milliseconds: Restart the service.

Microsoft determined that in most cases this was an issue and have provided a fix in Exchange 2013 CU3.  For most people, this resolves their issue.

However, there is another reason that this could be happening, so if you have applied Exchange 2013 CU3 and you are still hitting this issue, please read on.

In 2013 we move from a Public Folder database to a Public Folder Mailbox.  We provide step by step migration instructions here https://technet.microsoft.com/en-us/library/jj150486(v=exchg.150).aspx.

Sometimes when you delete things, AD doesn't always get the update that it should.  We have seen this in the past with connectors or user objects where we see reference to a deleted object.  That is what can be happening here.  In some of the cases that I have worked, the Legacy Public Folder is showing as a deleted object in AD, specifically the msExchHomePublicMDB attribute.  To check to see if this is populated, take the following steps

1. Open ADSIEdit

2. Connect to the Configuration container

3. Expand Services

4. Expand Microsoft Exchange

5. Expand your Organization name

6. Expand Administrative Groups

7. Highlight Databases

8. In the right hand pane you will see a list of databases. 

9. Right click on the database object and chose Properties

10. Scroll down to msExchHomePublicMDB

11. If you see reference to the old database delete the value

12. Check the rest of the databases to make sure that they are not populated in the same way.

Once this has been removed, you should stop seeing the System events noted above.

Thanks to Carl McDaniel, Corey Callaway and JT Thompson for their valuable assistance with this.

~JC

One thing that was done in service pack 3 for Exchange 2010 can cause an issue for companies who heavily use Outlook Anywhere.  These companies may start to see users calling saying that they either cannot connect or are losing connectivity to their mailboxes.   I plan to address this particular issue in this post.

In service pack 3, Exchange for the first time is going to utilize the ASP.NET runtime to process RPC over HTTP connections (Outlook Anywhere connections). ASP.NET runtime environment was architected according to design principles that place a premium on reliability and performance. The resultant ASP.NET process model comprises two system elements—an in-process connector living inside the Web server process and an external worker process. In addition, the ASP.NET runtime infrastructure is scalable enough to automatically exploit any selected processor on multi-processor hardware.   

ASP.NET has a connection limit of 25,000, by default.  In some organizations, this may not be high enough.  Once this limit is reached, users start getting dropped or they cannot connect at all.  Think about it this way, once your Exchange server reaches the RPC Request limit of 500, users are disconnected or cannot connect, making for a poor user experience.   This is a similar issue, just with a different component, ASP.NET.

There are a couple of ways of determining if you have hit this limit.  The first is to fire up Performance Monitor, go to the object ASP.NET Apps V2.0.50727, expand that out and choose Requests Executing and choose the counter _LM_W3SVC_1_ROOT_Rpc and under the RPC/HTTP Proxy object add the counter "Current Number of Incoming RPC over HTTP Connections".  Both these counters should be below 25,000.  If you are close, watch the counter.  If you see that the counter bumps to 25,001 and then drops to 24,994 you are exceeding this maximum.  Below is what this looks like for the "Current Incoming of Incoming RPC over HTTP Connections":

These 2 counters track the same exact way, so having only one or the other will be sufficient.  I had to choose "Scale the selected counter" by Right Clicking on the counter.

The second way to see if you are running into this issue is to look in the IIS logs.  You will see the following entries:

2013-03-21 23:06:14 10.0.0.1 RPC_OUT_DATA /rpc/rpcproxy.dll cas.contoso.com:6001 443 CONTOSO\user 10.0.0.10 MSRPC 503 3 0 15

2013-03-21 23:06:15 10.0.0.1 RPC_IN_DATA /rpc/rpcproxy.dll cas.contoso.com:6001 443 CONTOSO\user 10.0.0.10 MSRPC 503 3 0 46

These 2 items are clear indications that you are hitting this limit. 

The good news is there is a simple fix for this.  The following steps can be used to increase this limit:

1. Verify which app pool the /RPC virtual directory is using appcmd:
1.  cd %windir%\system32\inetsrv
2. appcmd list app /path
   /RPC this will show which app pool for example

APP "Default Web Site/Rpc" (applicationPool:DefaultAppPool)

2. Verify the .net Framework the app pool is using for example:
appcmd list apppool /name:DefaultAppPool

APPPOOL "DefaultAppPool" (MgdVersion:v2.0,MgdMode:Integrated,state:Started)

In the above Example we know the RPC\HTTP is using .net v2.0 managed framework (this is the default) and would use the
following instructions:

3. On the computer that is running the Client Access server role, locate the following file:

C:\WINDOWS\Microsoft.NET\Framework64\v2.0.50727\CONFIG\Machine.config

4. Open the Machine.config file.

5. In the <system.web> section, locate the following line:

<processModel
requestQueueLimit="25000" autoConfig="true" />

6. Change the value of requestQueueLimit property to 65535.

7. Open a command prompt and type IISReset /noforce **

**Please note that if you have other applications running on your Exchange 2010 SP3 server that utilize IIS, doing an IISReset /noforce will also restart those applications, as well as your Exchange web applications.**

These recommendations are noted in Knowledge Base Article 2832198

How do you proactively get ahead of this?  That's a good question.  It is pretty simple though.  I recommend that you start to look at the counters in Performance Monitor that are listed above.  These counters are available for you to monitor in Exchange 2010 SP2.  I would look at these counters over a span of a week or so to get an idea of what their daily average is.  If they are well under 25,000 there will be no need to make the above changes to the machine.confg file.  If you are close or going over the 25,000 limit that is imposed by ASP.NET you will need to make the changes to the machine.config file after applying SP3.

I would like to recognize Scott Oseychik, Bryan Talbert and Lou Prete for their assistance on this blog.  Their knowledge and insight into Exchange and IIS, respectively, have been invaluable.

We have seen some cases in CSS where recurring meetings have brought an entire Exchange Organization to its knees and leaving everyone thinking, “Why would someone do that?”.  It happens day in and day out and users are none the wiser because they have not been properly educated.  And that is where the crux of the problem lies, user education.  As Administrators, its not only our job to make sure that the environment runs smoothly, but also letting our user base know what is a best practice and what isn’t.

When it comes to calendaring there is some great Office documentation that is out there already, however, in an effort to point customers to documentation on what would be best practices for recurring meetings, there were some things that were missing.  As a Support Engineer, its easy for me to tell you what’s wrong, but it goes a long way if I have something that says it as well. 

One of the things that we see in CSS is when recurring meetings are modified.  The first couple of times are no problem, but there comes a time when a simple change turns into a critical situation.  Recurring meetings are special meetings and their history is very important, so it needs to be retained.  So, when we make a change to that meeting, the original gets copied to the modification.  We make another change and now the 1st modification and the original get copied to the 2nd modified meeting.  Make a 3rd change and we start the process over again.  At some point, there are too many copies attached to the message, so we go and clean it up and make it a nice neat embedded attachment.

Each time that meeting needs to be processed, because its been modified, store will take that meeting and expand out all the attachments and go through all the attendees before the modifications can be sent.  Guess what all that processing does to the Exchange server?  Uses up all its resources causing your users to complain of poor performance. 

Here are some of the ways that I have seen this manifested:

1. Excessive transaction log creation.  Using the steps from this blog we can take a look into the logs and get a feel for what is going on.  Its not a perfect world, but we really don’t have anything better to view these with.  What we are looking for is the last few lines and what they are telling us.  Look for time zone info or anything that looks like a calendar appointment.  The last case I worked with this, the bottom four lines were the same 2 individuals.  In 200 log files those 2 names were mentioned over 4 million times!  That was only the bottom 4 lines, there were many many more listings for the users.  When we went to open the calendar item on one of the users mailbox, their Outlook locked up and became unusable.  We had to go into the mailbox with MFCMapi to delete calendar item.  We had to do this on the other calendars where this meeting showed up. 

2. ESE 623 events “The version store for instance…has reached its maximum size of…It is likely that a long-running transaction is preventing cleanup of the version store and causing it to build up in size.”  These events tell us that there is a long running transaction that is causing the store to stop responding to requests.  These events are not exclusive to modified meetings.  The only way we can tell is by getting 3 consecutive store dumps about 3 minutes apart.  This will show us what we are waiting.  In one case I worked with this, the EE that was reading the dump came over and asked me why the user was sending out a meeting request to 500 users.  I started looking over their findings and started to see the same names over and over again.  That is when we put 2 and 2 together.  We got the originator on the phone and asked them about this meeting.  The meeting only had a handful of users, but we were processing the same attendees over and over again and there were definitely attachments.  That was when the user told me that they modify the meeting on a weekly basis.

Learning what user behavior went a long way in solving these cases.  My first questions was why are we modifying a recurring meeting so many times?  Doesn’t that defeat the purpose?  I think of a recurring meeting that happens for a specific period of time and during that time, it never changes, but that is me.  Some of the answers to these questions make sense, but are what leads us to our problem.

“I update the meeting agenda on a weekly basis.  Its just easier to do it here”

“I need to send out the weekly sales statistics so that everyone is prepared for the meeting”

“This is just a place holder for that block of time, the meeting place changes each week”

See, it makes their life easier.  They just don’t understand the consequences of their actions.  Here are some good best practices when it comes to recurring meetings:

1. A recurring meeting should have an end date.  That end date should be no further out then a year to two years.  Think about it, do you know where you are going to be in a year?  Will you still have the same job, the same company, the same boss even?  Setting up a never ending recurring meeting with your boss doesn’t ensure that you will have a job next year.

2. If you need to add an agenda to your meeting, send it in a separate email.  Simply copy and paste the attendee list into a new email and send that. 

3. If you need to send an attachment, send a link to where that data can be found.  Make sure it is a file share that everyone can access and this way any changes to past and present documents are located in the same place.  Now everyone can see the changes and we do not fill a mailbox with documents

4. Force the users to set end dates on their meetings.  This doesn’t prevent them from setting end dates that are 10-15-20 years out, but it makes them stop and think about it.  Hopefully most of the users will follow your recommendation (1-2 years).  You can set the following registry key DisableRecurNoEnd. 

For Outlook 2003: https://support.microsoft.com/kb/952144

For Outlook 2007: https://support.microsoft.com/kb/955449

For other great office tips, see the following link:

Outlook meeting requests: Essential do’s and don’ts:

https://office.microsoft.com/en-us/help/HA011276781033.aspx

~JC

In Exchange 2007, we don’t expose this in the UI anymore.  In order to set this, we have to run Powershell commands.  Even for a Powershell newb like me, that is not difficult to do.  Microsoft even makes it easy for you to find out how to do this on Technet.   The command we run is Set-ExchangeServer –Identity (Your Server name goes here) –StaticDomainControllers (Your DC name goes here). In the picture below, you will see that I set my CAS/Hub server to use win-19ichdu0fvs as my only DC.

As you can also see I am going to run the Get-ExchangeServer | FL to make sure that it was set properly.  What I expect to see is that I now have a statically set Domain Controller.

But wait, where is it?  Did the command run successfully?  I didn’t get an error when I ran it, so it looks like I ran it properly.  Let me run that again.  Ok still no error let me run Get-ExchangeServer | FL again.  And it still doesn’t show.  I must be doing something wrong.  Don’t worry your not, your just missing a switch that needs to be run in order for that to be populated.

Before we get into that switch, let’s talk about where this gets set actually.  Setting a static Domain Controller is not done in AD.  It is actually done in the registry of the Exchange Server.  Below is a snapshot of the regisrty before and after statically setting the Domain Controller.

Before:

After:

As you can see we add the UserDC1 subfolder to Default after setting the static Domain Controller.  Because this is not an AD setting, we need to call in to the Active Directory Topology server to pull this information.  The Exchange product group determined that there was too much of performance hit when we called into the Active Directory Topology service, which calls into the Remote Registry Service to warrant populating this information.  To offset that, they added the –status switch.  When you run Get-ExchangeServer –Identity (Your server name goes here) –Status | FL you will be able to see the static Domain Controller.

Lately, Exchange Support has fielded some calls on the symptoms that are seen when you set the –StaticDomainControllers.  The call usually starts out as, all my Domain Controllers are not reporting their information to Exchange properly.  Only one seems to be able to communicate with Exchange.  What the callers are looking at is the MSExchangeADAccess 2080 event.  Below is how the 2080 event will look when you set a static Domain Controller (server names have been blacked out to protect the innocent)

Event Type:    Information
Event Source:    MSExchange ADAccess
Event Category:    Topology
Event ID:    2080
Date:        3/18/2010
Time:        12:51:32 PM
User:        N/A
Computer:   XXXXXXXXXXXXXX

Description:

Process IISIPM38882B6C-3365-4687-BC84-D5D0A52F5F59 -AP "MSEXCHANGEOWAAPPPOOL (PID=5704). Exchange Active Directory Provider has discovered the following servers with the following characteristics:
(Server name | Roles | Enabled | Reachability | Synchronized | GC capable | PDC | SACL right | Critical Data | Netlogon | OS Version)
In-site:
StaticDC        CD- 1 6 6 0 0 1 1 6 1 
NotSet1    --- 1 0 0 0 0 0 0 0 0
NotSet2    --- 1 0 0 0 0 0 0 0 0
NotSet3    --- 1 0 0 0 0 0 0 0 0
NotSet4    --G 1 1 7 1 0 1 1 7 1 
NotSet5    --- 1 0 0 0 0 0 0 0 0

At first glance this looks pretty bad and the natural instinct is to call PSS.  There has to be something wrong with AD.  But if you begin to break this event down, we can see a whole different story.  In my troubleshooting, the first thing that struck me was how NotSet4 was reporting.  Why would it only show a G?  I know that G stands for Global Catalog server and in order for that to be a Global Catalog server, it would have to be a Domain Controller. Talking with the customer, I determined that the other servers were only Domain Controllers. So why doesn’t that show the D in the event?  At this point this would tell me this is a configuration setting and not something wrong with AD. 

The previous Engineers were running Get_ExchangeServer | FL and were getting thrown off the path of what the true issue was.  The customer was extremely confused because they thought it was a static Domain Controller as well, but the data we were getting back was not reflective of that.  Once we ran Get-ExchangeServer –Status | FL we could see that it indeed was a static Domain Controller.  To remove that we had to run Set-ExchangeServer –Identity (name of your Exchange Server goes here) –StaticDomainControllers $NULL.  After running that and waiting 15 minutes, we could see that all of our Domain Controllers were back to speaking to the Exchange server.

Our latest version of Exchange introduces a new character, the Exchange Trusted Subsystem.  The Exchange Trusted Subsystem (ETS) was created to support the new Role Based Access Control (BAC) model.  ETS is a highly-privileged Universal Security Group that has the Read and Write access on all Exchange related objects your Exchange Organization.  If you take a look at the Technet Article Exchange 2010 Deployment Permissions Reference you will see just what permissions are set as you run each Setup /prepare* command (go ahead, expand out the bullets) .  The point is, this is an extremely important account and should be treated with care.

So now we know what the ETS is all about, lets go back to the “Allow Inheritable Permissions”.  Security people are always trying to lock down their environment so that the right people have permissions to certain objects and others are locked out.  Understandable, as most of corporate theft doesn’t come from the outside, it comes from within the corporation itself.  There is always a fine line that when crossed creates one of the biggest headaches known to Support.  I ran into a case yesterday that many moons ago, an IT Admin removed the “Include inheritable permissions from this object’s parent” on all containers under the Organization in AD.  ETS got applied at the Organization level, but the lower level folders never got ETS. That is when the frustration begins.  The customer was trying to work with one of our BETA products and needed to make sure that the Arbitration Mailboxes and the FederatedEamil mailbox were provisioned properly.  In their installation they deleted their first databases and recreated them with distinct names.  So we need to set the DiscoverySearchMailbox as an Arbitration mailbox. Not a big deal as we have well documented steps to do this.

So we go and run the following command:

Enable-Mailbox "DiscoverySearchMailbox {D919BA05-46A6-415f-80AD-7E09334BB852}" –Arbitration and we get a cryptic response:

The user's Active Directory account must be logon-disabled for linked, shared, or resource mailbox.
    + CategoryInfo          : InvalidArgument: (Discovery Search Mailbox:ADUser) [Enable-Mailbox], RecipientTaskExcept
   ion
    + FullyQualifiedErrorId : BC7CCA19,Microsoft.Exchange.Management.RecipientTasks.EnableMailbox

Hmm the DiscoverySearchMailbox is a default account that should already be disabled.  We check and sure enough it is disabled.  We start looking at the arbitration mailboxes to see if they are working correctly and we run get-mailbox –arbitration and get nothing returned, we just get sent back the command prompt.  Well lets look what happens when we run this Enable-Mailbox in verbose mode:

C:\Windows\system32>Enable-Mailbox "DiscoverySearchMailbox {D919BA05-46A6-415f-80AD-7E09334BB852}" -Arbitration -De
bug -Verbose
VERBOSE: [14:47:19.205 GMT] Enable-Mailbox : Active Directory session settings for 'Enable-Mailbox' are:  View Entire
Forest: 'False', Default Scope: 'XXXXXXX.com', Configuration Domain Controller: 'XXXX.com',
Preferred Global Catalog: 'XXXXX.com', Preferred Domain Controllers: '{
XXXXX.com, XXXXX.com }'
VERBOSE: [14:47:19.208 GMT] Enable-Mailbox : Runspace context: Executing user: Admin
Accounts/Enterprise Admin, Executing user organization: , Current organization: , RBAC-enabled: Enabled.
VERBOSE: [14:47:19.208 GMT] Enable-Mailbox : Beginning processing.
VERBOSE: [14:47:19.211 GMT] Enable-Mailbox : Instantiating handler with index 0 for cmdlet extension agent "Query Base
DN Agent".
VERBOSE: [14:47:19.212 GMT] Enable-Mailbox : Instantiating handler with index 1 for cmdlet extension agent "Rus Agent".
VERBOSE: [14:47:19.213 GMT] Enable-Mailbox : Instantiating handler with index 2 for cmdlet extension agent "Mailbox
Resources Management Agent".
VERBOSE: [14:47:19.214 GMT] Enable-Mailbox : Instantiating handler with index 3 for cmdlet extension agent
"Provisioning Policy Agent".
VERBOSE: [14:47:19.215 GMT] Enable-Mailbox : Instantiating handler with index 4 for cmdlet extension agent "Admin Audit
Log Agent".
VERBOSE: [14:47:19.699 GMT] Enable-Mailbox : Current ScopeSet is: {Domain Read Scope: {, }, Domain Write Scope : {,
}, Configuration Scope: {, }, Server Configuration Scope : {, }, , Exclusive Scope: {, }}
VERBOSE: [14:47:19.701 GMT] Enable-Mailbox : Searching objects "DiscoverySearchMailbox
{D919BA05-46A6-415f-80AD-7E09334BB852}" of type "ADUser" under the root "$null".
VERBOSE: [14:47:19.734 GMT] Enable-Mailbox : Previous operation run on domain controller XXXXX.com.

Confirm
The user's Active Directory account must be logon-disabled for linked, shared, or resource mailbox.
[Y] Yes  [A] Yes to All  [H] Halt Command  [?] Help (default is "Y"): Y
The user's Active Directory account must be logon-disabled for linked, shared, or resource mailbox.
    + CategoryInfo          : InvalidArgument: (Discovery Search Mailbox:ADUser) [Enable-Mailbox], RecipientTaskExcept
   ion
    + FullyQualifiedErrorId : BC7CCA19,Microsoft.Exchange.Management.RecipientTasks.EnableMailbox

VERBOSE: [14:47:22.778 GMT] Enable-Mailbox : Ending processing.

Ok, lets see if we can create a new user.  We start seeing “Insufficient Rights to perform this action”.  Looking at the RBAC rights we are a member of the Organizational Management group.  That should be sufficient to complete the operation.  So we log in with the Root Forest account and we are still getting an Access Denied.  We add the Recipient Management Role to the user and still the same thing.  So we have all the rights, yet we cannot create a user.  There are 2 possible causes, there is a problem with the RBAC rights, which we know can’t be because we can verify we have the proper rights, but better yet, we actually can see the commands that are needed to be run.  You see if you don’t have the proper RBAC roles, you won’t even be able to run the command.  So in our case, we can run the command, so we must not have rights to AD.

That leads us to what could be our second issue, rights to AD. Our account is an Enterprise Admin, Schema Admin, Domain Admin, etc. You get the point, we have rights to do whatever we want in AD.  So we start at the top of the Organization and we see that the Exchange Trusted Subsystem is listed there with the proper AD rights.  What about our sub-folders.  We check the container where we are creating the user and low and behold, the ETS is not listed. Click the Advanced button and the “Allow inheritable permissions to propagate from the parent” was unchecked.  Checked that button and replicate AD and run Get-Mailbox –Arbitration and we see our 3 arbitration mailboxes. 

We like to call this “Defeated by Checkbox”.

Hope this helps ]]>