So what can you do? 

 In short, nothing. You will have to wait until OOS reaches General Availability. But hey, every cloud has a silver lining: it will give you time to properly plan your new Exchange 2016 environment!

Now, there is another part of the WAC/OOS story.  Namely, co-existence in a 2013 environment. First, let's recap how Exchange 2013 uses WAC.  WAC is configured at Organization level for 2013. If you've already got a WAC server in place (for Lync/SfB), the setup is nice and simple:

Set-OrganizationConfig -WACDiscoveryEndPoint https://Server1/hosting/discovery

 If only everything was that simple.

 Now, in Exchange 2016 things are a little different. We have two different places where we can configure our endpoint: at the organization level, and at the Mailbox server level. Where you configure the endpoint depends on the size of your organization and the location of your servers and users.

Organization - There are a couple of reasons why you might configure the Office Online Server Preview endpoint at the organization level:

• Single-server or single-location deployment You can configure the endpoint at the organization level if all of your Exchange 2016 Mailbox servers are in the same location and you don't plan on having geographically distributed Office Online Server Preview servers.
• Fallback for large deployments You can configure an endpoint at the organization level as a fallback if the endpoint configured on a Mailbox server isn't available. If an Office Web Apps server isn't available, the client will try to connect to the endpoint configured at the organization level.

Mailbox server - If you want to distribute client requests between two or more Office Online Server Preview servers, if you want to geographically distribute Office Online Server Preview servers, or if you have Exchange 2013 in your organization, configure the endpoints at the Exchange 2016 Mailbox server level. When you configure an endpoint at the server level, mailboxes located on that server will send requests to the configured Office Online Server Preview server.

Before we move on, take in the following:

Exchange 2013 uses Office Web Apps Server and does not support Office Online Server.

Exchange 2016 uses Offline Online Server and does not support Office Web Apps Server.

Exchange 2013 was an Organization level configuration.

Let's think about this.. If I have configured WAC for Exchange 2013 and I want to introduce Exchange 2016 into my environment, I am going to have a problem. Exchange 2016 does not work with the settings configured at Organization level because they are pointing to the WAC server, which is not supported.  I cannot change the Organization-level settings to OOS because Exchange 2013 servers do not support it.

So we're stuck, right?  Well, not quite.  Remember we can now configure our endpoint at server level in Exchange 2016! So that's what we do:

Set-MailboxServer ShinyNew2016Server -WacDiscoveryEndpoint "https://oos.internal.contoso.com/hosting/discovery"

Easy, right?  Well, yes, but we need to remember one small point about how Exchange 2016 works. It was mentioned above, but let's say it again here and highlight the important bit. "You can configure an endpoint at the organization level as a fallback if the endpoint configured on a Mailbox server isn't available. If an Office Web Apps server isn't available, the client will try to connect to the endpoint configured at the organization level."

Do you spot the potential issue here? What if my OOS server goes down?  Or what about when we need to patch it? That's right, we would fall back to the Organization-level setting, which has been configured with a WAC endpoint for our Exchange 2013 deployment. Not supported, remember?

Hmm.. So what's the Microsoft guidance for this situation then?

You will need to make sure that you configure your endpoint at server level on your Exchange 2016 servers, and that the endpoints are highly available. By this, we mean you would need to deploy at least two OOS servers in a farm configuration to make sure that, should one go down, you will still have an OOS endpoint available.  This will make fall-back to the WAC server configured at Organization level no longer a possibility and keep everything working in a supported configuration.

This post is about using AAD Connect to synchronise your On-Premises Active Directory accounts to Azure AD when you have a resource forest topology, with Exchange in one forest (the resource) and identity (the user account) in another forest.  By the end of it, we will have all our accounts sync'd to Azure AD, and we will be able to onboard a mailbox to Office 365, and log on to it with SSO. Pretty sweet.

Some things are outside the scope of this article, so I will not be telling you: 

• How to configure ADFS for SSO
• How to set up your Office 365 tenant
• How to configure your on-premises Exchange environment
• How to run the Hybrid Configuration Wizard

There are some excellent resources already out there for all of the above, so get Binging if you're not quite there. I may add some bits another time.

So, with that said, you should have the following all working already: 

• Exchange (I am using 2013, of course) deployed in a resource forest topology, with users all working properly.
• Fully functional ADFS environment, with an ADFS proxy (I am using WAP, of course)
• Fully configured Office 365 tenant, with domains verified, SSO enabled etc. It will have an empty directory, of course - we're going to populate that badboy in a while.

 Right, shall we get on with it?

Oh, before we do - you are doing all this in a lab first, aren't you? :) Use test OU's and test accounts! Be Safe!

Firstly, download AAD Connect from here and copy it to your sync server. In my lab I used my DC. This server should be in the "accounts" forest, where the users identity is.

Fire up the MSI and you will be greeted with the welcome screen.  Sign your life away and accept those T&C's..

You should now see the below screen.  As we do not have a single Windows Server Active Directory Forest, click Customize...

At the "Install Required Components" page, select a desired option if needed and/or click Next...

....and we're off...

You should see some icons pop up on your desktop. Sit tight until it runs through. Once done, you will get the below screen.  Select ADFS and click Next...

Next up, we connect to Azure AD, so add those credentials and click Next...

Once that goes through, we need to add some directories. As we are in a resource forest topology, we need to add both forests. We add the local "accounts" forest first. The order does make a difference.

...add the second one and then click Next...

Next up we need to decide how we are going to identify our users. We are in a resource forest, so for our identity and the disabled user account in the resource forest to be "merged" together in the metaverse, we need to match them up. For this we use the ObjectSID and msExchangeMasterAccountSID attributes.  I am using ObjectGUID as my Source Anchor. In certain environments you may choose to use something else. One common one is an EmployeeID, or staff number. We want something that will never, ever change. So, select as desired and click Next...

If you want, you can sync based on group membership. This is handy for a POC. Alternatively, you can edit the containers later on. Decide what you want to do and click Next...

Next up, lets choose our optional features.  How cool are some of these? Password Writeback! Awesome!  Anyway, we want to select "Exchange hybrid deployment" because, well, that is what we are doing, right? Click Next...

Right, now for some AD FS Configuration. Add your AD FS Server here and click OK...

You will see the below.  Click Next...

Once done, you will see the below screen.  Add your credentials and click Next...

Note: You will need to enable Remote Management on the AD FS server for this to go through. It should tell you! Make sure those pesky firewalls are not in your way, too!

Nearly there! Are you excited?

Select your Azure AD domain and click Next...

Woo hoo, we are ready to configure! I know you are excited now...

Note: Notice I have unticked "Start synchronization process as soon as the configuration completes". This disables the task in Task Scheduler. We have some more configuration to do so we don't want to sync just yet. I also enable Staging mode. This keeps us nice and safe. Safety first, remember boys and girls.

Click Install...

...and we're off again..

WOO HOO!! Now click Verify... A little verification never hurt anybody, did it?

Ooooh yes, we are rolling now!

Phew, we got through it! A few more steps and you will be in cloud heaven.

In the Start menu, you will see Synchronisation Service. Open that up. We need to edit which OU's we are going to Sync. You can add all sorts of filters here if you want to get creative. Remember, KISS

Click Connectors, and double-click one of your On-Premises directories

Then on the left, select Configure Directory Partitions, and then on the right, click Containers. You will be prompted for credentials. Use your On-Premises account here.

Next, select which OU you want to Sync and click OK>OK.

Now do the same for the other On-Premises forest!

Right we are soooooo close...

Lets run a sync and see what we get?  Remember, nothing is going up to Azure AD yet.

On each of the forests, run a Full Synchronisation.  Select the forest, Click Run on the right, and run a Full Synchronisation.

Now lets search the metaverse. I love saying metaverse.

Oh yes, look at all those lovely users just waiting to be sync'd to the cloud. We know our user matching worked as we only see one account here. Remember we are syncing two forests here! As they say in Spanish... Muy Bueno!

Right, last two things...  Open AAD Connect up again and get rid of the Staging Mode.  I don't think this needs a picture so you are not getting one.

Next, go in to Task Scheduler and enable the task there. Then run that badboy.  You don't need a pic for that either. 

Let it do its thing, and then log into your Office 365 portal...  Boom, there are your users. Can I get an Amen?

Nice! I love it when a plan comes together!

Now, go license them and move a mailbox up there (configuration of your on-premises Exchange environment is out of the scope of this article. But, in short, make sure your DNS records are right and your MRS Endpoints are enabled and your certs are all good). 

Go to your corporate OWA login page, which should redirect to your AD FS Sign-In page..

Are your fingers crossed?

Oh, where has our mailbox gone?  That's right, it's in the cloud...

Lets hit the portal instead...

Of course it will redirect us to our AD FS Server as we have configured our domain in Office 365 as a federated domain.

Back here again...

Fingers crossed...

.......

...................

..........................

....success!  We are in.

I hope this helps you get up to the cloud! Shout below if you have any questions :)

Adrian Moore

Senior Premier Field Engineer

Some bedtime reading:

Exchange Server 2013 Hybrid Deployments

Hybrid deployment best practices

Integrating your on-premises identities with Azure Active Directory

Understanding Users and Contacts in Azure Active Directory Sync

Understanding Key AD FS Concepts

Thanks to Pawel Partyka, Kris Wilson, Timothy Heeney and Andreas Kjellman for helping me out!

Until the PAL tool is upgraded by my colleague and author of the tool, ClintH, I am providing the file here for everyone to add manually.  If you already have PAL installed, you just need to copy the Exchange2013.xml file to C:\Program Files\PAL\PAL and you should then see it as per below when you run PAL:

Have fun :)

Adrian Moore

Senior Premier Field Engineer

EDIT: Clint has now updated the PAL tool which includes the Exchange 2013 definitions, so please update your version to 2.7.4 instead.

Active Directory Location

In RTM we created the Health Mailboxes in Contoso/Users.  I would love to you show you a pretty picture here, but I don't have an RTM lab and I don't feel like building a new forest just for one screenshot.  Close your eyes and it should come to you.

In CU1 we created a dedicated home for these objects, as per below:

Amount of Health Mailboxes and Naming Convention

Up to Exchange 2013 CU6, we created one Health Mailbox per mailbox database copy and one per CAS.  The naming convention was not particularly admin-friendly, being that it was the GUID - either of the CAS or of the database.  It would look something like this:

CU6 introduced some tasty changes to both the naming convention and the amount of Health Mailboxes that are created.  We now create a Health Mailbox for every mailbox database hosted on a Mailbox server (Active or Passive) and 10 Health Mailboxes for every CAS role!  Yes, there will be lots of these Health Mailboxes around.  As I mentioned, we also changed the naming convention to make it a bit more admin-friendly, as we are nice like that.

Database Health mailbox is now HealthMailbox+Servername+DatabaseName:

CAS Health Mailboxes are now HealthMailbox+ServerName+001-010 (remember, 10 per CAS role):

When you view these through the ADUC console, we still show the GUID unless you add the Display Name column.  But of course you are using the EMS, so no bother!

Common Issues

So now we know a bit more about the Health Mailboxes, lets look at two common issues.

1. "Corrupt" Health Mailboxes.  We often hear this term, and it seems to come from the error message which is sometimes shown when you run Get-Mailbox -Monitoring.  "/HealthMailboxXXXXXXXXX has been corrupted, and it's in an inconsistent state. The following validation errors happened: Database is mandatory on UserMailbox". So what does this mean, and is the mailbox really "corrupted"?  Well, no, not really.  What has actually happened is that this Health Mailbox has the database it corresponds to deleted.  It is therefore "orphaned" and will throw up this error.  This can often happen when admins install a new server, which gets a database created by default, and this is then removed and the clean-up piece doesn't happen properly.  The AD account is still there, but the mailbox is gone and the database attribute is empty.  These can (and should) be safely removed from Active Directory. It may be prudent to restart the Health Manager service on the affected server, too, just in case any probes are referencing them.

2. Account Lockouts. How on earth can a "system" mailbox account get locked out?  Well, as I said, for all intents and purposes, you can look at these Health Mailboxes as normal mailboxes with a corresponding AD account.  They have passwords which are periodically reset.  The password is a random 128 character secure string, so if you have any kind of domain password policy which could affect that, then it's possible to cause issues when the passwords are reset on the Health mailbox accounts.  It is best practice to make sure /Monitoring Mailboxes is not included in ANY domain password policies (including lockouts).  You can view password change/failure activity from the following log:

....\Exchange Server\V15\Logging\Monitoring\Monitoring\MSExchangeHMWorker\ActiveMonitoringTraceLogs

I hope this helps demystify the Health Mailbox a little bit!

Adrian Moore

Senior Premier Field Engineer

One of the often-cited design principles in Exchange 2013 deployments is the notion that architects should “scale out, not scale up”. So what does this mean, and why are we pushing this? First, lets look at what we mean when we use these terms.

Scaling Up

“Scaling Up” is where we add more resources to a server.  Let’s say our fictional company, Tailspin Toys, is half way through the design of their shiny new Exchange 2013 environment. They have 5,000 mailboxes and the calculator (you are using the calculator, aren’t you?) tells us we can house 1250 mailboxes per server, giving us 4 servers in our DAG. In this example, each server has 96GB RAM and 10 cores. Tailspin then acquires Contoso, and we get a requirement that we need to migrate in another 5,000 users (for the sake of this example, we will assume that Contoso have the exact same Message Profile as Tailspin, so our numbers are the same). In the past, we would have to add more resources to the server to take on the extra pressure this load would add: more RAM, more disk and more cores. Logic would dictate that this is a perfectly reasonable course of action to take, and, to some extent, it is. However, as we shall see further on, this is certainly far from the best course of action with Exchange 2013.

Scaling Out                 

“Scaling Out” is where we add more servers. So, with our example above, rather than increasing the cores to, say, 16 and the RAM to 146GB, we simply add more servers of the same spec. The math’s is nice and easy here; we add another 4 nodes to our DAG to make it an 8 node DAG.

The Cons

We hear a few common criticisms against this principle.  Now, it’s clear that this will cost you more in licenses. There will be 4 more Exchange 2013 Enterprise licenses needed and 4 more Windows Server licenses (assuming you are on physical servers and not virtual boxes on Windows Server 2012 R2 Datacenter Edition). 

However, the second point is one we hear a lot and it’s a common misconception about larger DAG sizes. To take advantage of the benefits of a larger DAG, it does not mean more copies of your database. Our Preferred Architecture is for you to have four copies of your database. Two in each datacenter and one set as a lagged copy.  It seems the thinking goes that if I have four servers with four copies, then having eight means I need eight copies for them all to be highly available. This is completely wrong, and even with a DAG at it’s maximum capacity of sixteen nodes, we still have four copies of the database.  The most important thing about making them highly-available (in addition to the other benefits of a larger DAG) is the way the database copies are spread out across the nodes. Let me repeat that, it is database-copy placement that counts.

The final common complaint against this is also about money. It will cost you another X dollars to buy more servers when it would only cost Y dollars to add more resources. Yes, it will cost a bit more. However, I think it’s fair to say that, for most companies, Email is a mission critical service, with lots of LOB applications relying on it to keep the show on the road and thousands of users relying on it for mail and calendaring. Quite frankly, it’s worth spending money on.

The Pros

To quote Ross Smith IV, “failure happens”. Your servers will die at some point, and when they do, there are really two things we don’t want to happen: firstly, we don’t want to bring your DAG down by losing quorum. So, the more nodes you can lose, the better. Yes, with 2012 R2 we have Dynamic Quorum, but that really only comes into it’s own when nodes drop in a sequential fashion and the cluster has time to jiggle things around. Second, in the event of a *over, we absolutely do not want our other nodes to have so much load put on them that keel over and die. The more nodes you have to spread the load, the better.

The other benefit is increased CAFÉ servers. Scaling Out means you add another 4 CAFÉ servers to your CAS array. The more nodes you have to spread the CAS load, in peacetime and in battle conditions, the better.

So, lets look at a few scenarios where our hardware fails and see the impact on load during node failure. Below we can see a four, eight and sixteen node DAG and how the DB’s are balanced during a single server failure, a quadruple server failure and, finally, a site failure.

4 Node DAG, 4 HA Copies (One lag)

Failing Server 1 gives you 14 active on Server 2 and Server 3 and Server 4 stay with their normal load. Hardly seems fair.

8 Node DAG, 4 HA Copies (One lag)

Failing Server 1 gives you a small increase in load on servers in DC1

Failing Servers 1 and 2 gives you, again, a smaller increase in load - spread over DC1 and DC2:

16 node DAG, 4 HA Copies (One lag)

Failing Server 1 gives you 8 DB’s per server in DC1:

So a much, much better distribution of the load based on server failures. 

Failing 4 servers in DC1 gives you this spread of DB’s:

Failing over the entire WAN gives you:

Real World

All of this information is good, and hopefully you will take it all on board before starting out with your Exchange 2013 design and deployment.  But sometimes, as Premier Field Engineers, we are assigned to a customer who has already completed the design phase and has a POC in. One of the best things about being a Dedicated Support Engineer is that you get to aid them in their design, supplying the kind of information that will help them avoid problem scenarios before they spend all their cash. 

One recent customer had a POC in with a 4 node DAG, spec’d to house around 5000 users. The company has 160,000 mailboxes, and their thinking was that they would use a DAG as a unit of scale (which it is), and every 5000 users they would put in a new DAG.  For those of you not blessed with a brain for maths, that means 32 DAG’s in their environment. So, what was their thinking and why did we change it?  Well, their thinking was that if “something” happened which would take the configuration of the DAG out, the outage would be limited to 5000 users, as opposed to, say, 10,000 (8 node DAG) or 20,000 (16 node DAG).  They weren’t sure what that event may be, but better safe than sorry.  Think about this for a minute.  We are actually making the DAG “weaker”, because we are reducing the amount of servers that can fail before the DAG keels over. Less servers means less redundancy. On top of that, we are creating a huge overhead from a management perspective, because we now have to manage 32 DAG’s instead of 16 or even 8.  Imagine a site switchover scenario. Would you rather complete the steps 32 times or 8 times?  We supplied the customer with enough information that they felt comfortable in increasing the number of nodes in the DAG.

Another recent example comes from an internal discussion with another PFE, also dedicated to their customer. Their customer wanted to use a server with 60 cores and 192GB RAM. This was a classic case of “Scaling Up”. They had plugged into the calculator the below:

The environment was a 25,000 seat estate, and they wanted to use 16 servers. Management had said that they had no budget to buy more servers, so they wanted to populate all 4 sockets. You are now all well versed in what is “wrong” with this approach.  Technically, there is nothing “wrong” with this design, per se.  But, as we have discussed, this is not the right approach. This is going against all of our guidance and thinking.  Moreover, customers need to understand that Exchange is not designed to run on this type of hardware. We do no testing on servers with this many cores. I cannot share what servers we use in the worlds’ largest Exchange deployment, Office 365, but I can tell you that we practice what we preach and have no servers with anywhere near that amount of cores.  This trailblazing customer would be doing something that we have never done ourselves, so we cannot advise them what will happen. We know that we start to see strange behaviour with .NET memory allocation when the amount of cores starts climbing, and the Product Group have been involved in CritSits (Severity A cases) where the Root Cause has been put down to the high number of cores in the server. Far better to increase the amount of servers - as we have seen - populating only 2 sockets (or even 4 sockets with fewer cores), and work within our comfort zone.  No one should want to be bobbing up and down in unchartered waters alone when it comes to a mission critical application like your Email system.

Conclusion

We know things will fail.  When we design an Exchange environment to deal with these failures, we need to make sure we get the best out of the design. As you can see, the more nodes in your DAG, the more nodes you can lose while a) still maintaining quorum and b) keeping the load lighter on the nodes that pick up the slack.  On top of that, you get the added bonus of extra CAFÉ servers in the array.  They say size isn’t everything, but when it comes to DAG’s I would respectfully disagree!

Adrian Moore

Senior Premier Field Engineer

 References

Database availability groups

Exchange 2013 Server Role Requirements Calculator

The Preferred Architecture

Datacenter Switchovers

Concerning Trends Discovered During Several Critical Escalations