CommunicationErrorTransientException: The call to 'https://MRSPROXYHOST/EWS/mrsproxy.svc SERVER.domain.com (14.3.178.0 caps:05FFFF)' failed.
Error details: The remote endpoint no longer recognizes this sequence. This is most likely due to an abort on the remote endpoint. The value of wsrm:Identifier is not a known Sequence identifier. The reliable session was faulted.

$stats = Get-MoveRequestStatistics <user> -IncludeReport $firstMailboxLockedError = $stats.Report.Failures | where { $_.FailureType -eq 'SourceMailboxAlreadyBeingMovedTransientException' } | Select -First 1 $stats.Report.Failures | sort Timestamp -Descending | where { $_.Timestamp -lt $firstMailboxLockedError.Timestamp } | Select -First 1 | fl

Here are the scenarios we’ve identified so far:

• DNS Round robin load balancing when DNS TTL expires
• DNS Round robin load balancing when outbound HTTP Proxy servers are used
• SSL Session-ID persistence with a hardware load balancer

The reason that each one of these scenarios occur is because of the new “LCIE” or “Tab Process” model in Internet Explorer 8.  Essentially Internet Explorer 8 can create new “tab processes” for new browser tabs or browser windows that are opened.  It doesn’t do this for every tab or window but uses an internal algorithm to determine whether or not it should spin up a new process. 

For a detailed description of the new architecture, see the following two posts:
IE8 and Loosely-Coupled IE (LCIE)
Opening a New Tab may launch a New Process with Internet Explorer 8.0

Due to the new architecture certain things such as kept-alive connections and SSL sessions are not shared between Tab Processes.  This plays an important role in the scenarios we have found. 

In advance: Yes, I know it’s not called an “OWA Server” – I’m calling it that to make this work for Exchange 2003 or Exchange 2007.

The design of Forms Based Authentication in Outlook Web Access is another contributing factor.  Forms based authentication works by the OWA server storing a cookie in the client browser for authentication.  This cookie contains the encrypted credentials of the user that the OWA server can translate into basic authentication credentials for IIS with each new request.  However this cookie is encrypted by a key that is maintained (and rotated) in memory only on the OWA server that set the cookie.  For instance if a client logs into Server A and receives a cookie from Server A, it cannot use that same cookie to authenticate to Server B because Server B does not have they key to decrypt the credentials.  If you do try to present a cookie from Server A to Server B, your browser will be redirected to the login page on Server B to provide valid authentication.

That explanation given, here is a brief explanation of the behavior you may see in each scenario.

DNS Round robin load balancing when DNS TTL expires

If you use DNS round robin load balancing for Outlook Web Access, you may experience confusing client behavior when the DNS time to live values expires on the record.  The scenario usually manifests itself like this:

You are logged into OWA (with forms authentication) when the DNS TTL expires.  If you continue to work within the main OWA window, you should be able to continue to work within the main OWA window, you should be able to continue to work and not be logged out.  This is because IE uses it’s kept-alive connections that are already established to the original OWA server.  However if you open a number of email messages in new windows (double-click mail messages), a new tab process for iexplore.exe will be created.  This new tab process will have to create a new TCP session that, when resolved by Winsock, returns the IP address of a different OWA server in the DNS round robin.  When this new window opens, you will be redirected to the forms authentication login page.  If you provide your credentials to the “new” server, you will in-turn get logged out of the “main” OWA window that still has a persistent connection to the “old” server. And the next time you spin up a new tab process, the experience starts over again. 

So needless to say, it’s a confusing user experience and the only way to recover is to close all browser windows, restart IE and log back in to OWA.

DNS Round robin load balancing when outbound HTTP Proxy servers are used

If you use DNS round robin load balancing for Outlook Web Access, you may experience being logged out randomly with Outlook Web Access.  This is really due to the same behavior as above except instead of the name resolution happening on the client, it happens on the proxy server.  Since proxy sessions are not maintained across tab processes, new outbound connections will also be established from the proxy server.  Since many users may be accessing the same DNS name through the same proxy server, the TTL expiration may seem completely random.

SSL Session-ID persistence with a hardware load balancer

If you are using SSL session-id based persistence on your hardware load balancer, you may experience the same behavior as above except in a more severe form.  In this situation the new connections that are established by the new tab process are routed to a different OWA server by the load balancer.  This will happen every time a new tab process is spawned, not just at some arbitrary interval making this by far the worst of the experiences.

Workarounds and Solutions

We’ll start with the easy one: SSL Session-ID persistence.  Because of issues (just like this) where the SSL session-id changes, SSL session-id based persistence is not (and has never been) a supported persistence method for load balancing Outlook Web Access.  In fact, there aren’t really any Exchange Services for which we support or recommend SSL session-id based persistence (Edit: Obviously for services that don’t require persistence at all such as ActiveSync and certain Outlook Anywhere configurations, you can use SSL-ID based persistence to keep the SSL handshakes to a minimum).  The supported persistence methods for Outlook Web Access are Client-IP based or Cookie-based persistence.  In certain situations where many clients are coming from the same IP (maybe a proxy server), cookie-based persistence is a much better solution than client-ip based persistence to get the load spread evenly.  Be aware that Outlook Web Access is the only Exchange web service that can use cookie-based persistence.

For DNS round-robin the answer is a bit more complicated.  We’ve never really recommended DNS round robin for load balancing, but it technically is supported.  Typically we just outline the caveats like the reliance on the client-side DNS resolver cache and the fact that it gives you zero redundancy in the case of a service failure.  However with all this guidance, many folks still see this as the poor man’s load balancer and implement it anyway.  If you are experiencing this issue, I strongly recommend you move to a real load balancing solution whether this is Windows NLB, the load balancing built-in to ISA Server 2006+, or a hardware-based load balancer.  If you can’t or won’t, this is something you’ll have to live with for now.  You can work around this issue by setting the TabProcGrowth registry value for the IE browser to 1 (It’s mentioned in the second of the earlier linked IE blogs).  However, this isn’t feasible in most situations because you have little control over the client-base.

We are currently pursuing a design change request with the Internet Explorer team to see if we can make any of these experiences more consistent with the IE7 behavior.  However, since changes in this area can be very risky, we may have to live with this new behavior.  After all, it’s not a bug, it’s just an effect of the change in design for IE8.

Thanks to everyone for reading, and special thanks to John Towler for reporting this.

I want to call out one issue I was informed about that one customer had run into (but most folks likely won't).  There actually is an upper-limit to the maximum number of values (or actually overall size) of a multi-valued attribute in Active Directory.  For these site values that limit typically exists somewhere between 800-900 entries. 

The error returned is:

Set-ClientAccessServer : Active Directory operation failed on MICROSOF-D1CEB0.contoso.com. This error is not retriable. Additional information: The administrative limit for this request was exceeded. Active directory response: 00002024: SvcErr: DSID-02080490, problem 5008 (ADMIN_LIMIT_EXCEEDED)

In this case, the customer had multiple load-balanced CAS's in any given site.  He simply split up the assignment and put 400 Sites in one SCP and 400 Sites in the other.  It seems that this is likely the best workaround if you were to run into this.

On another note, I found out something really cool you could do with Powershell today and wanted to update my previous script post so that it is 100% Powershell.  I was under the impression that you could not call Win32 API's from Powershell; while that is technically accurate, you can call Win32 API's from C# and you can call C# from Powershell so guess what you get?  Dynamically compiled C# code to call the Win32 API embedded in and invoked from a Powershell script!

This is adapted from the following blog post: Powershell - PINVOKE or accessing WIN32 APIs

The main script is identical to the script I previously posted.  However, this time instead of calling the command-line EXE, I call my .NET static method that I defined previously.

First: A Compile-CSharp function to do the on-the-fly compilation of my C# source code

We create a new instance of the CSharp code provider to do the compilation.

$cp=new-objectMicrosoft.CSharp.CSharpCodeProvider

We then set our parameters for the compilation operation.

$cpar=New-ObjectSystem.CodeDom.Compiler.CompilerParameters

$cpar.GenerateInMemory =$true

$cpar.GenerateExecutable =$false

$cpar.OutputAssembly ="custom"

Then we compile the source.

$cr=$cp.CompileAssemblyFromSource($cpar, $code)

Second: Defining the C# code and calling the compilation function

$siteCostCode=@'

<C# code here>

@'

# Compile our C# code for doing the site cost calculations

Compile-CSharp$siteCostCode

Finally: Calling the managed static method defined in our C# Code

[int[]] $costArray=$null;

$siteCostStatus= [SiteCost.SiteCostLib]::QuerySitesByCost($clientsite.Name, $exchangeSites, [ref]$costArray);

NOTE: At this time the download is not available.  I am trying to locate the script and will repost it if/when I can find it.

In these 5 months, I've had plenty of time to see customers do some "interesting" things with the Client Access Server deployments.  Since many customers still ignore our prescriptive guidance, I'm just going to give you the gun to shoot yourself in the foot. 

Here are some things you can do so you can ensure you have fun-filled days, every day of the week, on the phone with Microsoft Exchange Support.

Deploy multiple Client Access Servers in the same AD Site with different configurations

Here are some common things that lead customers to do this:

1. Want different functions for Client Access Servers.  I.E. "This is my OWA / ActiveSync CAS, and this is my Autodiscover / EWS CAS", or "This is my Internal CAS, and this is my External CAS."
2. Want the Client Access role on the mailbox server as a "backup" in case the CAS goes down.
3. Have their "Test" or pre-production environment in the same AD-Site/Same Exchange Org as their Production environment.

Don't assume that just because you have deemed CAS A as Internal and CAS B as External that Exchange will know the difference.  I will admit there are (most times) certain configurations changes that can make configurations like this work, but they are extremely complicated and prone to error.  All Client Access Servers in the same site should be configured the same way.  I make mention of this on my previous blog post, CAS Load-Balancing Best Practices (Part 1).  You will run into problems if you try this. There is no reason to try and split services as mentioned in reason 1 above. This was never tested and is not recommended. In general CAS Services scale well together and you should not try to isolate any particular service.  For #2 and #3 above, just remember, that Exchange and Outlook can't read your mind. As soon as a server is deemed a CAS (in AD), it is fair game for Outlook clients, proxy targets from other sites, etc, etc.  It is very difficult to understand and implement all the different factors that are considered when a client or a CAS in another site is choosing a CAS to connect to.  This will cause you issues; so just don't do it.

Deploy your Client Access Servers in a DMZ or Perimeter network, but "pretend" it's not a DMZ

We've seen customers again and again try and skirt our support stance on this.  Just in case you didn't know:

Planning for Client Access Servers: https://technet.microsoft.com/en-us/library/bb232184(EXCHG.80).aspx

Installation of a Client Access server in a perimeter network is not supported. The Client Access server must be a member of an Active Directory directory service domain, and the Client Access server machine account must be a member of the Exchange Servers Active Directory security group. This security group has read and write access to all Exchange servers within your organization. Communications between the Client Access server and the Mailbox servers within the organization occurs over RPC. It is because of these requirements that installing a Client Access server in a perimeter network is not supported.

Don't pretend your DMZ/Perimeter network isn't a DMZ/Perimeter network

We've had numerous customers who want to argue about what is and is not a Perimeter network.  What was meant in the original documentation by "perimeter network" is any network that does not have unrestricted access to every Domain Controller and Exchange Server in the Organization.  We've had customer who have called it a "pocket-DMZ" meaning that it's not their main DMZ where there web servers are.  This DMZ sits off their internal network in a separate "pocket" where access to and from the internal network is restricted.  This is still a Perimeter network and falls into the above support policy.  If your CAS is NOT on your internal network, it's probably safe to assume that it's in a DMZ and likely not supported.

Not supported, means not supported

If you call into PSS and the support engineer you are working with finds that your Client Access Server is in a restricted or perimeter network, you are deemed unsupported.  This does not mean that the PSS engineer hangs up the phone and says "too bad" right off the bat.  What this does mean is that the Engineer will gather logs and attempt to better understand your issue.  If at any point during troubleshooting, the PSS Engineer feels your issue may be caused by, or complicated by, the Client Access Server being in the perimeter, the engineer may request that troubleshooting be suspended until the Client Access Server in question be moved into the internal network.  If the customer is not willing to do this, then the customer is at that point unsupported by Microsoft PSS.

It's not even a good idea...

Our guidance recommends that an ISA Server or another reverse proxy server be placed in the DMZ to handle requests for Internet Exchange Services such as OWA and ActiveSync.  ISA 2004 and later allows ISA to operate in a Workgroup (non-domain) configuration and still pre-authenticate requests to Active Directory.  This ensures that no unauthenticated traffic is ever passed to the Client Access Servers and also ensures that every URL and request is processed and scanned by ISA URL Filtering/Scanning/and IDS feature-set.  This allows for a single port to be opened up (443) to each Client Access Server in the internal network (to allow ISA Web publishing) and is the most secure configuration by far.

If an Exchange 2007 Client Access Server is deployed in the perimeter, a plethora of ports must be opened (both random and static) to Exchange mailbox servers, Domain Controllers, and other infrastructure roles such as DNS.  For an Exchange Mailbox Server, you must open Dynamic RPC ports 1024 and above, in addition to 135, 139, and 445 (standard Windows RPC and File sharing ports).  These ports essentially make your back firewall in the perimeter scenario into swiss cheese and is counter-productive to a secure environment.

All of the Above

A picture is worth a thousand words.  Names have been changed to protect the guilty.

This customer was doing all of the above.  The CAS was in a DMZ, though according to the customer "It's not really a DMZ...Look, we even labeled it something different on the visio!"  Additionally, they had Client Access Servers "designated" as "internal" or "external" in the same AD site. 

Guess what?  They ended up on the phone with us! (shocking I know). 

And guess what the problem was? ActiveSync was attempting to proxy from the CAS in Data Center A's DMZ to the CAS in Data Center B's DMZ, but this traffic was not allowed.  The customer intended for the proxy traffic to only go the the HUB/CAS combo servers in the Internal Network in data center B.  But it didn't (obviously) and the firewall/routing caused the request to be blocked and fail.

This is an illustration of the simplest of all crazy issues you can and will hit if you deploy this way, but hey - maybe you're weird and just love problems or perhaps you just love calling into PSS.  Either way - happy deployments!

I'm sure I'll be adding to this list as time goes on (or as I think about more things).  Please spread the word, help our customers, and don't let them get into this situation.  This goes out to the field, partners, MVP's, etc - It's your job to help our customers make the right call; together we can make a difference.

The DNS SRV Record method was introduced by an Outlook 2007 Post-RTM hotfix and is also included in Outlook 2007 SP1.:
940881 A new feature is available that enables Outlook 2007 to use DNS Service Location (SRV) records to locate the Exchange Autodiscover service
https://support.microsoft.com/default.aspx?scid=kb;EN-US;940881

This method is great except for its few drawbacks.  First, you have to make sure all of your clients have this fix or SP1. In some environments, control of the client is not possible so this may discount this option immediately.  Another consideration with the SRV record method is whether or not *client* environments support the DNS SRV records.  For instance, if your only internet access and/or name resolution is through a proxy (or ISA Firewall Client), you have no way of resolving SRV records.

 With these drawbacks in mind, I wanted to bring light to the original method of redirecting autodiscover clients, the HTTP redirect method.  The implementation of this method via IIS is discussed in by Jason Mayans at https://blogs.technet.com/jmayans/archive/2006/09/07/454716.aspx.

After reading that, you should have a good idea that load balancing and hosting this extra redirect IIS site can be a nightmare.  Luckily, you can do it ALL at ISA!  Well 04 and 06 at least, I don't care if ISA 2000 can do it or not.  You still need an extra public IP, but the ISA redirection works great and if you have an array of ISA servers, it's automatically load balanced!

First, bind that new public IP to ISA.  Then start by creating a new Web Publishing Rule.

The Rule action will actually be set to Deny (We'll configure the redirect later)

The rest of this stuff is mostly irrelevant since it's a Deny Rule.  I chose:

"Publish a single Web site or load balancer"
"Use non-secured connections to connect to the published Web server or server farm"
"this.does.not.matter" for the "Internal Site Name"

Set the path to the actual Autodiscover.xml file: /autodiscover/autodiscover.xml

For the public name, you can set it to either the first of your hosted SMTP domains or just set the "Accept Requests for:" to "Any".

You'll want to create a new HTTP web listener, call it something like Autodiscover Redirect Listener.

Choose "Do not require SSL secured connections with clients" (very important as this redirect must be HTTP).

Select the new IP you bound to ISA for the Autodiscover Redirect listener.

Choose "No Authentication" as these requests will always be anonymous.

Ensure your new listener is selected in the wizard.

For the Auth delegation, you can choose "No delegation, and client cannot authenticate directly"

The rule cannot require authentication, so ensure "All Users" is selected.

Complete the rule.

You'll need to bring up properties on the rule and proceed to the "Action" tab.  It should be set to deny.  Put a check in the "Redirect HTTP requests to this Web Page" and enter the full HTTPS URL (matching your certificate), a which you have autodiscover published.  Mine was https://mail.wingtiptoys.com/autodiscover/autodiscover.xml.

On the "Public Name" tab, you will need to add autodiscover.<each hosted smtp domain>

Finally in each hosted SMTP domain's public DNS zone, create an A record for Autodiscover and point it to this new IP on the redirect site.

You can test this by browsing to https://autodiscover.hosteddomain.com/autodiscover/autodiscover.xml in your browser. 

It should redirect you to the URL you specified in the rule.

On launch of Outlook 2007, you should end up with a one-time dialog asking the user to allow this URL to configure their Outlook profile if you've done this right.

Test-ServiceHealth | Select-Object -Expand ServicesNotRunning | Start-Service

There is also the old standby:
Get-Service *Exchange* | Start-Service

But that one can start unnecessary services (set to manual) like the Exchange Pop3 and Imap4 service.

Just thought it'd make and interesting post because not too many people know about the Test-ServiceHealth cmdlet.

For Part 1 of this post, go here: CAS Load-Balancing Best Practices (Part 1)

You need to understand CAS->CAS Proxying before you try to understand load-balancing proxy CAS's.  Please read the following technet documentation if you don't understand CAS->CAS Proxy before proceeding:

Understanding Proxying and Redirection

The first big question is whether or not load balancing is required in the second site. Originally it was thought that the CAS in the first site would do load balancing and redundancy for the individual CAS's at an application layer.  We later found out that this is 100% true for OWA and Exchange Web Services Proxy but untrue for Activesync and Unified messaging proxy.

So, to get full load-balancing and redundancy for all services, you must deploy load balancing in the proxy sites as well as pictured below:

The trickiest part to this is not conifguring the NLB, it is knowing how to configure all of the URL parameters for the destination CAS.

Our documentation topic on technet "Understanding Proxying and Redirection" covers this in some detail.  I'm leveraging our chart below for reference on how to set the individual parameters for the Virtual Directories.

Virtual directory	InternalURL setting
/OWA	https://computername/OWA
/OAB	https://NLBname/OAB
/UnifiedMessaging	https://NLBname/UnifiedMessaging
/Microsoft-Server-ActiveSync*	https://NLBname/Microsoft-Server-ActiveSync
/EWS**	https://NLBname/EWS

* NOTE: This does NOT work in RTM for ActiveSync due to some Keberos issues.  You must set the InternalUrl to https://computername/Microsoft-Server-Activesync and you get no fault tolerance if a CAS in the seond site goes down.  We corrected these Kerberos issues in SP1 so you should be able to set the InternalUrl to the NLBname at that time.

** NOTE: EWS does not use the InternalUrl for proxying core EWS traffic and instead uses the InternalNLBBypassUrl configured on the Virtual Directory (this should always be configured to the machine FQDN).  Proxying for the Availability service will use the InternalUrl but affinity is not required.

 Note in regard to Affinity settings
OWA and EWS are "stateful" applications.  Because of this, they must maintain the connection with the same CAS throughout a users session.  However, because OWA and EWS use application-level load-balancing and failover and do not route through your load-balancer in the remote site, Affinity on the load balancer should not be an issue.  EAS and UnifiedMessaging are stateless and can be handled by any CAS on subsequent requests so for them, the load balancing affinity can be set to "None" in the proxy site to ensure even distribution of requests.

Have fun deploying your CAS Proxy + NLB Solutions!

You can find the whitepaper here:
https://technet.microsoft.com/en-us/library/bb332063.aspx

You can find the Exchange team blog announcment here:
https://msexchangeteam.com/archive/2007/10/03/447176.aspx

The only thing it doesn't really cover right now is load-balancing, for that you can see my series on CAS load-balancing here: https://blogs.msdn.com/brad_hughes/archive/2007/09/10/cas-load-balancing-certificates-autodiscover-and-webservices.aspx