1. We re-organized the content into Guides.
2. We released a Preview of a new feature called Compliance Manager[2].

Guides

• Compliance Guides include ISO, SOC, FedRAMP and other audit reports, bridge letters and materials related to independent, third-party audits of Microsoft’s cloud services such as Microsoft Azure, Microsoft Office 365, Microsoft Dynamics 365, and others.
• Trust and Transparency Guides include deep dive whitepapers that provide details on how we design and operate our cloud services, FAQs, end-of-year security assessments, and penetration test results.
• Azure Blueprints provide turn-key compliance solutions and support tailored to the needs of industry verticals that accelerate cloud adoption and usage for customers with regulated or restricted data. Available blueprints include U.S. Department of Defense, FedRAMP, Healthcare, NIST, PCI-DSS, and U.K. G-Cloud.

Compliance Manager

• Enables you to conduct risk assessment of Microsoft cloud services: Combines the detailed information provided by Microsoft to auditors and regulators as part of various third-party audits of Microsoft ‘s cloud services against various standards (such as International Organization for Standardization 27001:2013 and ISO 27018:2014) and information that Microsoft compiles internally for its compliance with regulations (such as the EU General Data Protection Regulation or GDPR) with your own self-assessment of your organization’s compliance with these standards and regulations.
• Provides you recommended actions and detailed guidance to improve your data protection capabilities that can help you meet regulatory requirements.
• Simplifies your compliance workflow and enables you to assign, track, and record compliance and assessment-related activities, which can help your organization cross team barriers to achieve your organization’s compliance goals. It also provides a secure repository for you to upload and manage evidence and other artifacts related to your compliance activities, so that you can produce richly detailed reports in Microsoft Excel that document the compliance activities performed by Microsoft and your organization, which can be provided to auditors, regulators, and other compliance stakeholders.

[embed]https://www.youtube.com/watch?v=-ScjtTIOnQs[/embed]

Controlling Access to Compliance Manager

• Compliance
• Privacy
• Security
• Risk assessments
• Auditing
• Data protection
• IT administration

• Global administrator
• Exchange administrator
• Skype for Business administrator
• SharePoint administrator
• Compliance administrator
• Security administrator
• And others

Compliance Manager Data

Compliance Manager Feedback

• We use your customer data only to provide the services we have agreed upon, and for purposes that are compatible with providing those services.
• We do not share your data with our advertiser-supported services, nor do we mine it for marketing or advertising.
• If you leave our services, we take the necessary steps to ensure your continued ownership of your data.

• Secure
• Private
• Compliant
• Transparent

• Permissions   Enables you to assign permissions such as Compliance Administrator, eDiscovery Manager, and others to people in your organization so they can perform tasks in the Security & Compliance Center.
• Security policies   Enables you to create and apply device management policies using Office 365 Mobile Device Management and to set up Data Loss Prevention (DLP) policies for your organization.
• Data management   Enables you to import email or SharePoint from other systems into Office 365, configure archive mailboxes, and set retention policies for email and other content within your organization.
• Search & investigation   Provides content search, audit log and eDiscovery case management tools to quickly drill into activity across Exchange Online mailboxes, groups and public folders; SharePoint Online, and OneDrive for Business.
• Reports   Enables you to quickly access reports for SharePoint Online, OneDrive for Business, Exchange Online, and Azure AD.
• Service assurance   Provides information about how Microsoft maintains security, privacy and compliance with global standards for Office 365, Azure, Microsoft Dynamics CRM Online, Microsoft Intune, and other cloud services. Also includes access to third-party ISO, SOC and other audit reports, as well as Audited Controls, which provides details about the various controls that have been tested and verified by third-party auditors of Office 365.

Service Assurance

Audited Controls

• Control ID (as assigned by the particular mapped standard)
• Test status (whether or not the control passed testing)
• Test date (when the control was last tested)
• Tested by (typically third-party auditors or Microsoft)
• Control implementation details (how the control is implemented)
• Testing performed to evaluate control effectiveness (how the control was tested)

I call this post Keeping up to date with what’s happening with Set-MailboxDatabase II because it builds upon Tony’s post about the changes to the Set-MailboxDatabase cmdlet in Exchange 2013.

The first change Tony mentions is the description for the AutoDagExcludeFromMonitoring parameter.  The purpose of this parameter is to exclude a database from Managed Availability’s ServerOneCopyMonitor.  We consider the ServerOneCopyMonitor, which executes only on DAG members and not standalone Mailbox servers, to be our most important redundancy protection mechanism from an HA perspective.  As Tony also mentions, it is the latest incarnation of the database redundancy checks we introduced and I blogged about in May 2010. We think it is very important that all customers monitor for situations in which database redundancy is compromised and immediately take action to restore database redundancy to avoid catastrophic data loss.  What started as an external script was later shipping in the box and configured automatically on all Mailbox servers (which is why the Task Scheduler service is a requirement for installing the Mailbox role in Exchange 2010).  And in Exchange 2013, the checks are now done in code as part of Managed Availability.  As a result, there are no scheduled tasks configured by Exchange, and the Task Scheduler service is not required to install the Mailbox role in Exchange 2013.

The ServerOneCopyMonitor checks databases once a minute on all DAG members.  It specifically is looking for databases that where:

• The database copy is (Healthy || Mounted) &&
• ServerComponentState is NOT Offline &&
• The database copy is NOT Activation Blocked &&
• The server is NOT exceeding MaxActive &&
• The copy queue length < AutoDatabaseMountDial &&
• The server is NOT Activation Disabled

30 consecutive failures are considered an escalating condition by Managed Availability.  In this event, the ServerOneCopyMonitor is notified and transitions into an Unhealthy state.

And that is the way it will work in Exchange Server 2013 Cumulative Update 6 (CU6).  But in CU5 and earlier, that’s not quite how it works.  As Tony reported, about 15 minutes after setting AutoDagExcludeFromMonitoring to True, it mysteriously reverts back to False, and the ServerOneCopyMonitor generates 4113 alerts about databases that should have been excluded.  And that brings us back to those small changes that can elude even those closest to the process.  As it turns out, the resetting of AutoDagExcludeFromMonitoring from True wasn’t a huge mystery or major code bug.  Rather, it was a function of a datacenter (Exchange Online) process that made its way into the enterprise (on-premises) product.  Can you think of a feature in Exchange 2013 that is mysterious and makes changes to your servers behind the scenes and under the covers?  That’s right…it was Managed Availability again.  As it turns out, Exchange 2013 on-premises ships with an Exchange Online responder called EnableDatabaseMonitoringResponder that is part of the DataProtection health set used in Exchange Online.  I bet you can guess from the name of the responder and the context of this blog as to what that responder does.

As you probably guessed, that responder runs every 15 minutes, and if it finds any databases excluded from monitoring, it re-includes them by changing the value of AutoDagExcludeFromMonitoring from True to False. The primary purpose of the AutoDagExcludeFromMonitoring parameter is to allow administrators to exclude a non-replicated database in a DAG from single copy alerting. But in Exchange Online, we don’t have any non-replicated databases in our DAGs. All databases in all Exchange Online DAGs are replicated. As such, databases should not be excluded from monitoring, and AutoDagExcludeFromMonitoring should always be set to False. To ensure that we catch cases in Exchange Online where AutoDagExcludeFromMonitoring was somehow set to True, we created the EnableDatabaseMonitoringResponder to check the value of AutoDagExcludeFromMonitoring and change it back to False if it finds it set to True. And because the EnableDatabaseMonitoringResponder was included in Exchange Server, this behavior is happening in on-premises environments, too.

As I mentioned before, we have fixed this in a later build, and it will be included in CU6.  In the meantime, if you have set the value of AutoDagExcludeFromMonitoring to True because you want to exclude a database, you should be able to override the EnableDatabaseMonitoringResponder by using a server or global override; for example:

Add-GlobalMonitoringOverride -Identity DataProtection\EnableDatabaseMonitoringResponder -ItemType Responder -PropertyName “Enabled” -PropertyValue “0”

For the above override, also add either –Duration or –Build to specify a time length or have the override apply to a specific build of Exchange 2013.

After this override takes effect, AutoDagExcludeFromMonitoring should remain set to True.

The second change Tony mentions is the MaintenanceSchedule parameter.  I actually opened a bug to remove this parameter from Exchange 2013 way back in September 2012.  No, that is not a typo.  That was one of three mailbox database parameters that we deprecated and discussed removing from Exchange 2013:

1. MaintenanceSchedule – because we no longer have scheduled maintenance
2. PublicFolderDatabase  - because we no longer have public folder databases
3. QuotaNotificationSchedule – because we no longer schedule quota notifications; we check at logon time now

We went through several rounds of discussions about this, and decided that instead of removing the parameters, they should be deprecated. The primary reason for this was to avoid any interoperability issues in mixed environments that contained Exchange 2013 and legacy (e.g., Exchange 2010 and/or Exchange 2007) servers and databases. For example, the PublicFolderDatabase  parameter is used by Exchange admins to point Exchange 2013 users to legacy public folder databases in single-forest coexistence deployments.

We also wanted to be particularly sensitive about any custom scripts that might be out there in use by admins that were leveraging these parameters. If we left the parameters in, but returned a Not_Supported error, that would break these scripts.  If we took the parameters out altogether, that would break these scripts.  Of course, with functionality being removed, these scripts need to be fixed anyway, but we wanted the experience to be as frictionless as possible.  So we opted for a third path – deprecate the parameters in content and in the product.

So for the MaintenanceSchedule parameter, we documented in TechNet that “This parameter has been deprecated in Exchange 2013 and it no longer does anything.”  However, as Tony and fellow MVP Jaap Wesselius discovered, that last part is not true.  Sort of.  It depends on how you look at it.  If you look at it from a pure parameter perspective, then TechNet is right.  The parameter no longer does anything.  But if you look at it from cmdlet perspective, Set-MailboxDatabase -MaintenanceSchedule does do something.  It will change the value of the database’s MaintenanceSchedule property; but like the parameter of the same name, the property does nothing on Exchange 2013 databases because scheduled maintenance no longer exists in Exchange 2013. But to avoid any further customer confusion, I changed the description of this parameter in TechNet to read as follows:

“This parameter has been deprecated in Exchange 2013. While it can be used to change the MaintenanceSchedule property of a database, that property is ignored in Exchange 2013 because scheduled maintenance no longer exists.”

That change will go live with the next content refresh to TechNet.

As Tony says at the end of his post, “All of this goes to prove that you have to keep your wits about you to keep up to date with changes that are made to a major product like Exchange 2013. It makes the job interesting – or demanding.”  I choose to think of it as interesting. 

In this post, I’ll focus on two important changes in Windows Server 2012 R2 that increase DAG resilience, namely:

• Dynamic Quorum
• Dynamic Witness

Of course, Dynamic Quorum is not new in Windows Server 2012 R2; it was introduced in Windows Server 2012.  But since it is enabled by default in Windows Server 2012 R2, it’s worth discussing here, as well (and everything I write about Dynamic Quorum applies to DAGs running on Windows Server 2012).

I’ll also talk about a new way to create DAGs on Windows Server 2012 R2: a DAG without a cluster administrative access point.

In addition to the behavior changes, some WFC terminology has changed in Windows Server 2012 R2. For example, in Windows Server 2012 R2, the parameters for Set-ClusterQuorum no longer use the same quorum mode terminology used in previous versions of WFC, such as NodeMajority, NodeAndFileShareMajority or NodeAndDiskMajority. Instead, simplified cmdlet parameters are used, such as NoWitness, FileShareWitness, and DiskWitness. In the Validate Quorum Configuration report, simple terminology is used, such as Witness Type and Witness Resource. Where applicable, this new terminology is integrated into this post, and will be integrated into the Exchange 2013 High Availability and Site Resilience documentation on TechNet.

Dynamic Quorum

In Windows Server 2008 R2, quorum majority is fixed, and based on the initial cluster configuration. Unless an administrator manually removes a node’s vote, a node always has a vote, and the number of votes needed to maintain a majority is always the same.

In Windows Server 2012 and Windows Server 2012 R2, quorum majority is determined by the set of nodes that are active members of the cluster at a given time. This behavior is called Dynamic Quorum (DQ), and it is enabled for all clusters by default.

Unless manually disabled by an administrator, DQ is therefore enabled on all DAGs that are deployed on Windows Server 2012 or Windows Server 2012 R2. With DQ, the cluster dynamically manages the assignment of votes to nodes, based on the state of each node, and dynamically recalculates quorum requirements. It’s pretty simple, actually:

• When a node shuts down, crashes, or loses connectivity with the cluster, it loses its quorum vote
• When a node rejoins the cluster, it regains its quorum vote

If the cluster maintains quorum after a shutdown or failure, and whenever a node rejoins the cluster, the number of votes required to maintain quorum will be recalculated, based on this change in vote count.

DQ’s ability to dynamically manage votes is different from the ability an administrator has to manually remove a vote from a node. If an administrator removes a node’s vote by setting it’s NodeWeight property to a value of 0, then DQ does not dynamically give back the vote.

The idea behind DQ is that, by adjusting the assignment of quorum votes and dynamically increasing or decreasing the number of quorum votes required to keep running, the cluster can sustain sequential node shutdowns (or failures) all the way down to a single node (referred to as a “last man standing”).

As long as quorum is maintained after a shutdown or a failure, DQ can recalculate quorum requirements, and the cluster will reduce the number of votes needed to maintain quorum. And that is a fundamental requirement for DQ: quorum must be maintained after a shutdown or failure. If quorum is lost, DQ does nothing for you. DQ does not allow a cluster to sustain a simultaneous failure of the majority of voting members.

Determining Assigned and Current Vote Counts

Even though a node is Up, it might not have a vote (for reasons other than an administrator removing the vote).. For example, when a cluster is down to the last two active nodes, DQ will remove the vote from one of the nodes. One node will have a vote, and the other node will not. It is important that in this configuration the administrator be aware of which node has the vote. In this scenario, although an administrator can safely shut down the node that does not have a vote, if they shut down the node with a vote, there are no votes left in the cluster, and the remaining node (without a vote) terminates service.

You can see which nodes have a vote using Get-ClusterNode. For example:

Get-ClusterNode <Name> | ft name, dynamicweight, state

The output you’ll see is similar to the following:

A DynamicWeight value of 1 indicates the node has a vote, and a value of 0 indicates the node does not have a vote (because DQ took it away).

You can also view this information in Failover Cluster Manager, provided your DAG has an administrative access point (see the section below).

If your DAG does not have an administrative access point, then you can use Get-ClusterNode as described above, or run the cluster validation tests using Test-Cluster to view voting information. For example:

Test-Cluster -Node EX4 -Include Cluster

Test-Cluster will create a Validate Quorum Configuration report that will display the assigned and current vote count, as shown below.

Best Practices for Dynamic Quorum

The WFC team’s best practice recommendation is to keep DQ enabled for your clusters because it generally increases the availability of the cluster, and it allows the cluster to continue running in failure scenarios that are not possible if DQ is disabled.

The Exchange team’s best practice recommendation is to keep DQ enabled for your DAGs. Exchange is not DQ-aware, but DQ can increase the availability of a DAG in failure scenarios where a Windows Server 2008 R2 DAG would have lost quorum. It’s worth noting that all of our internal testing on Windows Server 2012 and Windows Server 2012 R2 is performed with DQ enabled, and DQ is enabled for all DAGs in Office 365 that run on those platforms, as well.

That being said, customers should not factor DQ into their capacity planning and sizing plans for Exchange Server 2013. For example, if you are planning to deploy an 8-member DAG, you would not size each server to handle the entire 8-server workload in case it became the last man standing. Continue to following the published guidance and use the Server Role Requirements Calculator for sizing your DAGs.

Note that while we recommend DQ remain enabled for all DAGs, we have not tested it with DAGs that use applications that support the third party replication API built into Exchange 2013. If you are using third-party replication for your DAG instead of the built-in continuous replication, consult with your replication vendor to determine if DQ should remain enabled for your DAG.

Dynamic Witness

All DAGs must have a valid server configured for the DAG’s WitnessServer property. However, the witness server and cluster File Share Witness resource are configured for use within the DAG’s cluster only when the DAG contains an even number of members and is using the FileShareWitness (previously Node and File Share Majority) quorum model. If the DAG contains an odd number of members, the DAG server management tasks will execute subtasks that configure the DAG to use a NoWitness (previously Node Majority) quorum model.

In previous versions of WFC, when a cluster is configured to use the FileShareWitness quorum model, it will use the configured witness server as an extra vote to maintain quorum when the cluster is one vote away from losing quorum. It is at that point where a file on the witness server called witness.log would be locked by one of the DAG members, and the witness would count as a vote for quorum calculation purposes.

Prior to Windows Server 2012 R2, if when needed for quorum the File Share Witness cluster resource is in a Failed state, the cluster service will try to restart it once in an attempt to bring the resource online so that the witness.log file on the witness server can be locked. If the cluster can bring the resource online, a lock can be established. If the cluster cannot bring the resource online, the witness server and share will not be available for use, and a lock cannot be established. In this event, quorum would be lost and service within the DAG would terminate (e.g., databases would dismount). If when needed for quorum the File Share Witness resource is in an Offline state (a state which can be achieved only via administrator action), the cluster service will not try to bring the resource online, and quorum will be lost and service within the DAG will terminate.

In Windows Server 2012 R2, a cluster configured to use DQ (which is all clusters by default), will also use a feature called Dynamic Witness (DW). In DW, the witness vote is dynamically adjusted based on the number of current votes. Like DQ, the logic used here is pretty simple:

• If there are an odd number of votes, the witness does not have a vote.
• If there are an even number of votes, the witness has a vote.

The witness vote is also dynamically adjusted based on the state of the witness resource. If the File Share Witness resource is Offline or Failed, the cluster sets the witness vote to 0.

This is an important change over previous versions of WFC. With DW, the cluster decides whether to use the witness vote based on the number of votes that are currently available in the cluster.

Check the Witness Vote

You can see if the witness server currently has a vote by using Get-Cluster. Windows Server 2012 R2 includes the new WitnessDynamicWeight cluster common property that you can use to view the witness vote. For example:

(Get-Cluster).WitnessDynamicWeight

or

Get-Cluster –Name EX4 | FT Name, WitnessDynamicWeight –auto

0 means the witness does not have a vote, and 1 means the witness has a vote.

Best Practices for Dynamic Witness

Like DQ, the WFC team’s best practice recommendation is to keep DW enabled for your clusters. It significantly reduces the risk that the cluster will lose quorum because of witness or witness resource failure.

The Exchange team’s best practice recommendation is to keep DW enabled for your DAGs. Exchange is not DW-aware, but DW can increase the availability of a DAG in failure scenarios where a Windows Server 2008 R2 DAG would have lost quorum due to witness failure. All of our internal testing on Windows Server 2012 R2 is performed with DW enabled, and DW is enabled for DAGs in Office 365 that run on Windows Server 2012 R2.

DAG Without a Cluster Administrative Access Point

All DAGs running Windows Server 2008 R2 or Windows Server 2012 require at least one IP address on every subnet included in the MAPI network. The IP address(es) assigned to the DAG are used by the DAG’s cluster. The name you assign to the DAG becomes the cluster network name (also known as the cluster administrative access point, or AAP), which enables name resolution and connectivity to the cluster using the cluster’s IP address (or more precisely, connectivity to the cluster member that currently owns the cluster core resource group) using the cluster name.

Windows Server 2012 R2 enables you to create a failover cluster without an AAP. A Windows failover cluster without an AAP has the following characteristics:

• There are no IP addresses assigned to the cluster, and therefore no IP Address resources in the cluster core resource group.
• There is no name assigned to the cluster, and therefore no Network Name resources in the cluster core resource group.
• Because there is no name or IP address assigned to the cluster, there is no DNS entry for the cluster, and the cluster is not resolvable on the network.
• A cluster name object (CNO) is not used, and therefore not created in Active Directory.
• The cluster cannot be managed using Failover Cluster Manager. It must be managed using Windows PowerShell, and the PowerShell cmdlets must be run against individual nodes.

Clusters without AAPs can be created with New-Cluster by setting the AdministrativeAccessPoint property to a value of None.

Exchange Server 2013 SP1 supports creating a DAG without a cluster AAP as a new optional configuration. A DAG without an AAP has the following characteristics:

• There are no IP addresses assigned to the DAG. Because the IP address properties are required properties for DAG objects, they are populated with a placeholder/pseudo IPv4 address value of 255.255.255.255.
• The DAG has a name, and that name appears in the cluster properties of each DAG member, but it is not used by the cluster for any purpose, it is not registered in DNS, and it is not resolvable on the network.

Creating a DAG without an AAP reduces the complexity of your DAG and simplifies DAG management. In addition, it also reduces the attack surface by removing a network access point to the node that owns the default cluster resource group, by removing the cluster/DAG name from DNS, and by eliminating a CNO.

When running Exchange 2013 SP1 on Windows Server 2012 R2, you can create a DAG without an AAP using the Exchange Admin Center (EAC) or the Shell. The process in each of these tools is very different when it comes to the DAG IP address (e.g., the DatabaseAvailabilityGroupIPAddresses parameter):

• In the EAC, you give the DAG an IP address of 255.255.255.255 while creating the DAG
• In the Shell, you create the DAG using New-DatabaseAvailabilityGroup and use a specific syntax for the DatabaseAvailabilityGroupIPAddresses parameter. For example:
  
  New-DatabaseAvailabilityGroup –Name D2 –WitnessServer ex3.e15demos.com -DatabaseAvailabilityGroupIPAddresses ([System.Net.IPAddress])::None

When you create a DAG using either of the above methods, Exchange will automatically create a cluster for the DAG without an AAP.

DEMO: Creating a DAG without an Administrative Access Point

I created some simple videos to demonstrate the two different ways to create a DAG without an AAP. These videos have no audio, they are meant to be watched in HD, and they have been time compressed:

• In the first video, I create a DAG without an AAP using the EAC.
• In the second video, I create a DAG without an AAP using the Shell.

My demo topology is very simple, and consists of four servers:

• DC1 – domain controller/global catalog server running Windows Server 2012
• EX4 – Exchange Server 2013 SP1 CAS/Mailbox running Windows Server 2012 R2
• EX5 – Exchange Server 2013 SP1 CAS/Mailbox running Windows Server 2012 R2
• EX3 – witness server running Windows Server 2012

Converting an Existing DAG

In WFC, you can configure administrative access point settings only when you create a cluster. Thus, a DAG with one or more members cannot have its configuration changed from having an AAP to not having an AAP, or vice versa.  There is no supported way to do this because WFC does not support it. So don’t even try it.  But, I know some of you will anyway, so…

If you have a DAG with members without an AAP, and you use Set-DatabaseAvailabilityGroup to try to assign it an IP address or configure it to use DHCP, you will get a warning and an error message similar to the following:

WARNING: An unexpected error has occurred and a Watson dump is being generated: Some or all identity references could not be translated. 
Some or all identity references could not be translated.
    + CategoryInfo          : NotSpecified: (:) [Set-DatabaseAvailabilityGroup], IdentityNotMappedException
    + FullyQualifiedErrorId : System.Security.Principal.IdentityNotMappedException,Microsoft.Exchange.Management.SystemConfigurationTasks.
   SetDatabaseAvailabilityGroup
    + PSComputerName        : ex4.e15demos.com

In this event, the properties of the DAG will change, but the properties of and IP addresses assigned to the cluster will not(and a CNO will not be created). Attempting such a procedure can break your DAG. In this case, you must immediately run Set-DatabaseAvailabilityGroup and reconfigure the DAG with the 255.255.255.255 IP address.

If you try to change the IP address settings in EAC (for example, by removing 255.255.255.255 and adding a static IP address), you’ll receive a similar error message:

Similarly, if you have a DAG with members and with an AAP and you try to use the Shell to remove the AAP, you will get a warning message similar to the following:

WARNING: No static address matched networks 'Cluster Network 1'. Specified static addresses: '255.255.255.255'

In addition, the cluster core resource group will fail, as the Set-DatabaseAvailabilityGroup does remove the IP Address resource, which causes the Network Name resource to go into a Failed state. In this state, the Network Name resource cannot be brought online or deleted. In this event, you must run Set-DatabaseAvailabilityGroup and re-assign the DAG’s previous IP address(es). This will recreate the IP address resource in the cluster, and allow the Network Name resource to come back online.

In any event, WFC does not support changing this configuration option, and neither does Exchange. If you have a DAG that has one or more members, you must first remove all of them (which in turn destroys/removes the existing cluster). Then, you can change the DAG properties (e.g., use Set-DatabaseAvailabilityGroup to change the IP address settings for the DAG). When you add members to the DAG, the DAG and cluster will use the new property settings. Note, though, that this process will result in downtime.

Keep in mind that creating a DAG without an AAP is completely optional. Although a DAG does not require a cluster AAP, you may be using third-party applications that are levering the AAP of your DAG’s cluster (for example, backup, management, and monitoring applications). When Exchange Server 2013 SP1 is running on Windows Server 2012 R2, you can decide for yourself, on a DAG-by-DAG basis, whether or not a DAG should have an AAP:

• Creating a DAG with one or more static IP addresses, or using DHCP for the DAG’s IP address(es), creates a DAG with an AAP.
• Creating a DAG using the process described in this post and demonstrated in the linked videos creates a DAG without an AAP.

For more information about creating a DAG without a cluster AAP, see Creating DAGs and Create a Database Availability Group.

Windows Server 2012 R2 and DAGs

Windows Server 2012 R2 is an excellent platform for DAGs and for all Exchange Server 2013 SP1 servers. It streamlines deployment, because it already includes most of the Exchange pre-requisites (the only thing you need to download separately to install Exchange 2013 is the UCMA package), and it enables you to build DAGs that can take advantage of the improvements in WFC.

In previous versions of Exchange, monitoring critical system components often involved using an external application such as Microsoft System Center 2012 Operations Manager to collect data, and to provide recovery action for problems detected as a result of analyzing the collected data. Exchange 2010 and previous versions included health manifests and correlation engines in the form of management packs. These components enabled Operations Manager to make a determination as to whether a particular component was healthy or unhealthy. In addition, Operations Manager also used the diagnostic cmdlet infrastructure built into Exchange 2010 to run synthetic transactions against various aspects of the system.

Exchange 2013 takes a new approach to monitoring and preserving the end user experience natively using a feature called Managed Availability that provides built-in monitoring and recovery actions.

Overview

Managed availability, also known as Active Monitoring or Local Active Monitoring, is the integration of built-in monitoring and recovery actions with the Exchange high availability platform. It's designed to detect and recover from problems as soon as they occur and are discovered by the system. Unlike previous external monitoring solutions and techniques for Exchange, managed availability doesn't try to identify or communicate the root cause of an issue. It's instead focused on recovery aspects that address three key areas of the user experience:

• Availability   Can users access the service?
• Latency   How is the experience for users?
• Errors   Are users able to accomplish what they want?

Managed availability is an internal process that runs on every Exchange 2013 server. It polls and analyzes hundreds of health metrics every second. If something is found to be wrong, most of the time it will be fixed automatically. But there will always be issues that managed availability won’t be able to fix on its own. In those cases, managed availability will escalate the issue to an administrator by means of event logging.

For more information about this new feature, see the newly published topic Managed Availability.

Health Sets

From a reporting perspective, managed availability has two views of health, one internal and one external. The internal view uses health sets. Each component in Exchange 2013 (for example, Outlook Web App, Exchange ActiveSync, the Information Store service, content indexing, transport services, etc.) is monitored by managed availability using probes, monitors, and responders. A group of probes, monitors and responders for a given component is called a health set. A health set is a group of probes, monitors, and responders that determine if that component is healthy. The current state of a health set (e.g., whether it is healthy or unhealthy) is determined by using the state of the health set’s monitors. If all of a health set’s monitors are healthy, then the health set is in a healthy state. If any monitor is not in a healthy state, then the health set state will be determined by its least healthy monitor.

For detailed steps to view server health or health sets state, see the newly published topic Manage Health Sets and Server Health.  For information on troubleshooting health sets, see this topic.

Health Groups

The external view of managed availability is composed of health groups. Health groups are exposed to System Center Operations Manager 2007 R2 and System Center Operations Manager 2012.

There are four primary health groups:

• Customer Touch Points Components that affect real-time user interactions, such as protocols, or the Information Store
• Service Components Components without direct, real-time user interactions, such as the Microsoft Exchange Mailbox Replication service, or the offline address book generation process (OABGen)
• Server Components The physical resources of the server, such as disk space, memory and networking
• Dependency Availability The server’s ability to access necessary dependencies, such as Active Directory, DNS, etc.

When the Exchange 2013 Management Pack is installed, System Center Operations Manager (SCOM) acts as a health portal for viewing information related to the Exchange environment. The SCOM dashboard includes three views of Exchange server health:

1. Active Alerts Escalation Responders write events to the Windows event log that are consumed by the monitor within SCOM. These appear as alerts in the Active Alerts view.
2. Organization Health A rollup summary of the overall health of the Exchange organization health is displayed in this view. These rollups include displaying health for individual database availability groups, and health within specific Active Directory sites.
3. Server Health Related health sets are combined into health groups and summarized in this view.

Overrides

Overrides provide an administrator with the ability to configure some aspects of the managed availability probes, monitors, and responders. Overrides can be used to fine tune some of the thresholds used by managed availability. They can also be used to enable emergency actions for unexpected events that may require configuration settings that are different from the out-of-box defaults.

Overrides can be created and applied to a single server (this is known as a server override), or they can be applied to a group of servers (this is known as a global override). Server override configuration data is stored in the Windows registry on the server on which the override is applied. Global override configuration data is stored in Active Directory.

Overrides can be configured to last indefinitely, or they can be configured for a specific duration. In addition, global overrides can be configured to apply to all servers, or only servers running a specific version of Exchange.

For detailed steps to view or configure server or global overrides, see Configure Managed Availability Overrides.

When you configure an override, it will not take effect immediately. The Microsoft Exchange Health Manager service checks for updated configuration data every 10 minutes. In addition, global overrides will be dependent on Active Directory replication latency.

Below are some examples of adding and removing global and server overrides:

Example 1 - Make Information Store maintenance assistant alerts non-urgent for 60 days:

Add-GlobalMonitoringOverride -Identity Store\MaintenanceAssistantEscalate -ItemType Responder -PropertyName NotificationServiceClass -PropertyValue 1 -Duration 60.00:00:00

Example 2 - Change the maintenance assistant monitor to look for 32 hours of failures for 30 days:

Add-GlobalMonitoringOverride -Identity Store\DirectoryServiceAndStoreMaintenanceAssistantMonitor -ItemType Monitor -PropertyName MonitoringIntervalSeconds -PropertyValue 115200 -Duration 30.00:00:00

Example 3 - Remove the maintenance assistant monitor override added in Example 2:

Remove-GlobalMonitoringOverride -Identity Store\DirectoryServiceAndStoreMaintenanceAssistantMonitor -ItemType Monitor -PropertyName MonitoringIntervalSeconds

Example 4 - Remove the Information Store maintenance assistant alerts non-urgent override added in Example 1:

Remove-GlobalMonitoringOverride -Identity Store\MaintenanceAssistantEscalate -ItemType Responder -PropertyName NotificationServiceClass

Example 5 - Apply the database repeatedly mounting threshold override (change to 60 minutes) for a period of 60 days:

Add-GlobalMonitoringOverride -Identity Store\DatabaseRepeatedMountsMonitor -ItemType Monitor -PropertyName MonitoringIntervalSeconds -PropertyValue 3600 -Duration 60.00:00:00

Example 6 - Remove the database repeatedly mounting threshold override added in Example 5:

Remove-GlobalMonitoringOverride -Identity Store\DatabaseRepeatedMountsMonitor -ItemType Monitor -PropertyName MonitoringIntervalSeconds

Example 7 - Change the database dismounted alert from HA to Store for a period of 7 days:

Add-GlobalMonitoringOverride -Identity Store\DatabaseAvailabilityEscalate -ItemType Responder -PropertyName ExtensionAttributes.Microsoft.Mapi.MapiExceptionMdbOffline -PropertyValue Store -Duration 7.00:00:00

Example 8 - Disable VersionBucketsAllocated monitor for a period of 60 days:

Add-GlobalMonitoringOverride -Identity Store\VersionBucketsAllocatedMonitor -ItemType Monitor -PropertyName Enabled -PropertyValue 0 -Duration 60.00:00:00

Example 9 - Update logs threshold in DatabaseSize monitor for a period of 60 days:

Add-GlobalMonitoringOverride -Identity MailboxSpace\DatabaseSizeMonitor -ItemType Monitor -PropertyName ExtensionAttributes.DatabaseLogsThreshold -PropertyValue 100GB -Duration 60.00:00:00

Example 10 - Applying a server override to disable quarantine monitor across all database copies for a period of 7 days:

(get-mailboxDatabase <DB Name>).servers | %{Add-ServerMonitoringOverride -Server $_.name -Identity "Store\MailboxQuarantinedMonitor\<DB Name>" -ItemType Monitor -PropertyName Enabled -PropertyValue 0 -Duration:7.00:00:00 -Confirm:$false;}

Management Tasks and Cmdlets

There are three primary operational tasks that administrators will typically perform with respect to managed availability:

• Extracting or viewing system health
• Viewing health sets, and details about probes, monitors and responders
• Managing overrides

The two primary management tools for managed availability are the Windows Event Log and the Shell. Managed availability logs a large amount of information in the Exchange ActiveMonitoring and ManagedAvailability crimson channel event logs, such as:

• Probe, monitor, and responder definitions, which are logged in the respective *Definition event logs.
• Probe, monitor, and responder results, which are logged in the respective *Results event logs.
• Details about responder recovery actions, including when the recovery action is started, and it is considered complete (whether successful or not), which are logged in the RecoveryActionResults event log.

There are 12 cmdlets used for managed availability, which are described in the following table.

Information workers need to communicate and collaborate, and organizations are continuously looking for more efficient and economical ways to facilitate this. The de facto method is email. In just a short time, email has become a necessary component of business. For many businesses, email is mission-critical. At least one study has found that the overwhelming majority of business workers consider email more necessary than the telephone, and that the loss of email would present a greater hardship than the loss of the phone service.¹

The success of many companies is tied to their information technology infrastructure and how it is used. In business today it is critical for information to get to the right people at the right time so that informed decisions can be made quickly. Corporate information can generally be put into one of two categories: that which is used to support the purpose of the business, such as sales and marketing data, and that which is used to support the business’ infrastructure, such as payroll and accounting data.

Today, a significant percentage of corporate information flows through corporate email systems, and over the Internet. It is generally accepted that in order to do this, messaging and collaboration systems must be:

• Reliable—the system must continuously work as expected, without loss of service or failure of core functionality.
• Secure—the system must support secure messaging features, such as encryption of transmitted data, verification and authentication of messages and message origins, and privacy features that protect confidential content. In addition, the system must be secured against tampering, unauthorized access, unauthorized use of services and other maliciousness.
• Interoperable—the system must interoperate with all other email systems on the Internet and inside an organization.
• Accessible—end-users should be able to access messaging and collaboration services regardless of their location, platform, or available bandwidth.
• Standards-based—in order to interoperate and be accessible from any location and any platform, the system must use public protocols for communication and transmission of data.
• Economical—the system must be inexpensive and cost-effective, with the lowest total cost of ownership (TCO) possible.

But how did email get its start, and where did it come from?

A Brief History of BBN

You may not have heard of a company called Bolt Beranek and Newman (BBN) or a BBN engineer named Ray Tomlinson. Nor might you be aware of their impact on business messaging and collaboration. In 1948, two MIT professors, Richard Bolt and Leo Beranek started a small consulting firm in the field of acoustics. A year later, they hired one of Bolt’s former students—Robert Newmann—and founded BBN. Between 1948 and 1997, BBN was a small, independent R&D company. They worked on various ways to improve sound and sound quality, reduce noise pollution from aircraft, and establish standards for noise metrics.

GTE purchased BBN in 1997, and in 2000, GTE merged with Bell Atlantic to create a new company—Verizon. Certainly Verizon has had an impact on business messaging with its cell phone, wireless and broadband services. But the real impact on business messaging; that which brought business messaging to the masses, happened more than 30 years ago.

Although they began their work in the field of acoustics, they later expanded their research into the most sophisticated areas of computer science. In 1960, BBN developed a priority-interrupt system for a Digital Equipment Corporation (DEC) computer called the PDP-1. For the first time, a computer could multi-task: processes could consume computing resources based on priority, as scheduled by the system’s kernel. After demonstrating the first time-sharing computer system in 1962, BBN designed the world’s first computer modem (the Data-Dial) in 1963.

Throughout the next several years, BBN worked to create new computer languages, refine calculations and formulas for predicting structural and noise vibrations, and even created the Grafacon—a tablet-based device that digitized graphical data for input into a computer. The Grafacon is literally the first ancestor of the Tablet PC, and other pen-based computing devices.

Building ARPANET

Taken individually, these accomplishments have had major impacts on the human race, but they pale in comparison to work BBN began in 1968, when the government awarded BBN a contract to create the Advanced Research Projects Agency Network (ARPANET), a network that would connect military and educational institutions using packet-switching protocols. ARPANET would later undergo several name changes, but ultimately settled on the one we all use today—the Internet.

In less than a year, they had established a network with four sites, with one node at each site. Not that it was a reliable network, however. The first transmission of data was successfully from a node in Stanford to a node in UCLA on August 29, 1969. However, when that same data was sent to a third node, it crashed the third node. Once the network was stabilized, additional nodes were added, and the network was primarily used for transferring files and for logging on to remote systems using technology called Telnet that is still in use today.

One of the scientists at BBN who worked on the ARPANET project was Ray Tomlinson. Tomlinson and his team worked on a project called CYPNET. CYPNET research and development concentrated on a primitive file transfer protocol that was the forbearer to today’s File Transfer Protocol (FTP). One of Ray’s other projects was an equally primitive method of sending messages called SNDMSG and its companion RDMAIL. These components worked together to allow users on the same machine to send messages to each other. A user creates a text file which was then delivered to one or more mailboxes on the same machine. On this system, a mailbox was simply a text file to which text was appended. The name of the mailbox was nothing more than the name of the text file. In addition, data could be added, but not deleted or changed. Of course, the bigger limitation was that all users had to use the same computer; SNDMSG could not send messages to a user on a different computer.

Tomlinson changed all of this with a series of ideas. First, he realized that the file sharing technology in CYPNET very closely met the functionality needed to transfer messages between computers. It dawned on him that just as computers could exchange files, they could also exchange messages. Combining a protocol that can transfer files with one that can transfer messages was described by Tomlinson as a “no-brainer” and as a “neat idea.”² Once he realized this, he modified the file sharing code from CYPNET so that it could append to a file. This was a simple and quick change. After that, Tomlinson made another quick decision that would impact every Exchange administrator and user, and every Internet email user that was to come.

The problem was location; or put slightly different, although email from one user to another user on the same computer had been used for some time, network email—email between users on different computers—did not exist. If the user did not have a local mailbox, and instead had their mailbox located on another computer, you could not send them a message. A mechanism would be needed so that a message could be sent to a remote user at a remote computer. As it turned out, the key word here was at or in computer symbols, @. According to Tomlinson, after less than a minute of thought on the problem, he chose the @ symbol to separate the username (also called the mailbox name) from the computer name (also called host name) because it enabled him to indicate that the user was at some other host and not local. Effectively, Tomlinson created the format of a user’s email address by using a mailbox name, an @ symbol, and the name of the host containing the mailbox.

After making some minor code modifications in late 1971, Tomlinson did some tests and sent the world’s first email between two different computers that sat side-by-side.³ Legend has that the first email ever sent had something like “QWERTYUIOP” or possibly “TESTING 1 2 3 4” in its body, however, nothing was ever saved or recorded for history or posterity, and Tomlinson himself no longer remembers exactly what it said. Nonetheless, history was made that would forever change human communication. Tomlinson had created networked email—messaging for the masses.

OK, maybe not yet for the masses. Network email had no features other than the ability to send a text message to another user on another computer. There were no attachments, no rich-text, no spell checking, no file or folders, and no ability to reply to or forward messages. There was also no electronic address book or directory. Instead, email addresses were listed on printed paper directories and distributed to email users.

Email at the Speed of Light

By 1972, the first email management program had been written and distributed. RD (no fancy marketing names here), by programmer Larry Roberts at ARPANET, enabled users to list incoming messages, store them in a file-system like environment, and forward and reply to messages. RD would later inspire others to create email management programs (now generically referred to as email clients), including some that are still in use today, such as Eudora.

Within two years of Tomlinson’s first email to himself, 75% of all traffic on ARPANET was email. By 1978, ARPANET was providing email services to a large number of government institutions and universities, as well as a few private businesses, such as BBN and DEC. Although it wasn’t called it at the time, the first unsolicited commercial email (UCE)—also known as spam—was sent by a DEC employee on May 3, 1978, in violation of the acceptable use policy for ARPANET in effect at the time. It is not truly known if this spam was accidental or intentional, but one thing is clear: the baby Internet was already a miniature model of things to come—email is hugely popular among those with access to it, and the system is wide-open for abuse.

Between 1971 and 1976, Request for Comments proposals (RFCs) for standards-based mailbox and messaging protocols had been published. The very first RFC for a mailbox protocol—published on July 20, 1971—was RFC 196, "Mail Box Protocol," by Richard W. Watson. Later came RFC 561, "Standardizing Network Mail Headers", and RFC 680, "Message Transmission Protocol.” These were later rendered obsolete by RFC 733, “Standard for the Format of ARPA Network Text Messages.”

In November 1975, Jon Postel wrote RFC 706, "On the Junk Mail Problem.” In it, Postel noted that ARPANET had no mechanism for that allowed a host to selectively refuse messages. Each host that accepted email had to accept every message it was sent. There was no way to filter out email; it was simply all or nothing (a spammer’s paradise if spammers had existed back then). On September 1, 1980, Postel and fellow researcher S. Sluizer published RFC 772, "Mail Transfer Protocol." This in turn led to a series of RFCs which eventually led to RFCs 821 and 822, which initially defined one of the Internet email protocols in use today—the Simple Message Transfer Protocol (SMTP). Those RFCs were superseded by RFCs 2821 and 2822, respectively, which were then obsoleted by RFC 5321 in October 2008.  By the way, if you’re going to administer Exchange, you should read these RFCs, which you can do at http://www.ietf.org.

Conclusion

So that’s where email came from.  Email has come a long way since Ray Tomlinson sent the world’s first message between two different computers in late 1971. Now more than forty years later, billions of email messages are sent around the globe every day.  And some of them are even worth reading!

¹ April 22, 2013: META Group, Inc. (Senior Executive Panel)

² http://www.rit.edu

³ http://openmap.bbn.com/~tomlinso/ray/ka10.html

Although these hotfixes are generally recommended for all customers, we recommend that you evaluate each fix to determine whether it applies to your environment. If you determine a server in your organization is affected by the problem(s) that a fix addresses, install the fix on each cluster node by using the procedures that are described in Knowledge Base article 174799: Patching Windows Server Failover Clusters.

The recommended hotfixes include two important updates:

• 2848344 – This is an updated roll-up hotfix that replaces 2838669.  This fix is most important for any Hyper-V deployment, and critical for any deployment using DPM for backup.
• 2838043 – This is a fix for NetName, that addresses a password synchronization issue.  This has been a top CSS call generator, where Repair was used in the past.

The hotfixes we recommend for Windows Server 2012 failover clusters are as follows:

For more information, see http://support.microsoft.com/kb/2784261.

Witness Server Warning Message When Using Certain Database Availability Group Tasks

I first wrote about this issue back in June 2011.  This is where the system displays an incorrect warning message when you are using a non-Exchange server as your witness server, even when you have configured things correctly.  This issue was eventually fixed in Exchange 2010 Service Pack 2 RU5, but it didn’t make it’s way into Exchange 2013 RTM.  Fortunately, the fix did make its way into CU1.

Self-recovery Behaviors

Exchange 2013 continues the innovation introduced in Exchange 2010 by including functionality that allows the system to self-recover from failures that affect resiliency or redundancy. In addition to the Exchange 2010 self-recovery behaviors, Exchange 2013 RTM includes additional behaviors for long I/O times, excessive memory consumption by the Microsoft Exchange Replication service (MSExchangeRepl.exe), and severe cases where threads can't be scheduled. For example, every 30 seconds, the Exchange Replication service heartbeats the crimson channel, as it is a required component for normal operations. If this heartbeat fails, an indication that the crimson channel is inaccessible for some reason, the Exchange Replication service self-recovers the server by forcibly rebooting the server, thereby triggering a server failover.

In addition to the behaviors in Exchange 2013 RTM, CU1 includes new behaviors:

• Bus Resets – Event 129 is logged in the System event log when a bus reset occurs.  Bus resets, particularly the rare but possible bus reset storm, can often result in storage issues, such as hung IO. When these events occur, it typically requires administrator intervention to resolve the issue. To obviate the need for administrator intervention, CU1 includes new functionality that triggers a forcible reboot of the server when event 129 is detected in the System event log.
• Replication service endpoints not responding – Exchange periodically verifies that the TCPListener component in the Exchange Replication service is responding to connection requests by periodically heart beating the local instance of the Exchange Replication service.  If the TCPListener does not respond, the system will automatically self-recover by forcibly rebooting the server.

Autoreseed

Automatic reseed, or AutoReseed, is a feature that's the replacement for what is normally administrator-driven action in response to a disk failure, database corruption event, or other issue that necessitates a reseed of a database copy. When properly configured, AutoReseed is designed to automatically restore database redundancy after a disk failure by using spare disks that have been provisioned on the system.

CU1 includes numerous fixes to AutoReseed, including fixes for issues around AutoReseed not detecting spare disks correctly and AutoReseed not using detected spare disks.  In addition, the following enhancements have been made to AutoReseed:

• GetCopyStatus now has a new field 'ExchangeVolumeMountPoint', which shows the mount point of the database volume under C:\ExchangeVolumes (or a custom folder if you are not using the default setting of C:\ExchangeVolumes).  This is useful information to know because in a configuration that uses multiple disks per volume, the LogicalDisk performance counters show up as the first mount point (which would be the one under C:\ExchangeVolumes) instead of as the database path like they used to in Exchange 2010 with a single disk per volume.
• We now have better internal tracking around mount paths and the ExchangeVolume path.
• The limits for AutoReseed have been increased from 4 databases per volume in Exchange 2013 RTM to 8 databases per volume in CU1.
• AutoReseed properties have been added to Active Directory that allow you to enable and disable automatic reseeding and the DiskReclaimer function (which formats Exchange volumes). The two new properties are:
  • AutoDagAutoReseedEnabled - Setting AutoDagAutoReseedEnabled to false turns off AutoReseed (including automatic resume, sparing, and in-place reseeds).
  • AutoDagDiskReclaimerEnabled - Setting AutoDagDiskReclaimerEnabled to false turns off the DiskReclaimer, which formats exchange volumes. The default setting is true, and it only tries to format volumes mounted under C:\ExchangeVolumes.
• The unused AutoDagFailedVolumesRootFolderPath property was also removed from the DAG object.

As a result of these and other changes, the workflow for AutoReseed in CU1 has changed.  The primary input condition for the AutoReseed workflow is still a database copy that is in an Failed and Suspended (F&S) state for 15 consecutive minutes.  When that condition is detected, the following AutoReseed workflow is initiated:

1. The system will first try to resume the database copy up to 3 times, with 5 minute sleeps in between each try.  Sometimes, after an F&S database copy is resumed, the copy remains in a Failed state. This can happen for a variety of reasons, so this first step is designed to handle all such cases; AutoReseed will automatically suspend a database copy that has been Failed for 10 consecutive minutes to keep the workflow running. If the suspend and resume actions don’t result in a healthy database copy, the workflow continues.
2. Next, AutoReseed will perform a variety of pre-requisite checks. For example, it will verify that a spare disk is available, that the database and its log files are configured on the same volume, and in the appropriate locations that match the required naming conventions. In a configuration that uses multiple databases per volume, AutoReseed will also verify that all database copies on the volume are in an F&S state.
3. Next, AutoReseed will attempt to assign a spare volume up to 5 times, with 1 hour sleeps in between each try.
4. Once a spare has been assigned, AutoReseed will perform an InPlaceSeed operation using the SafeDeleteExistingFiles seeding switch.  If one or more database files exists, AutoReseed will wait for 2 days before in-place reseeding (based on the LastWriteTime of the database file).  This provides an administrator with an opportunity to preserve data, if needed.  AutoReseed will attempt a seeding operation up to 5 times, with 1 hour sleeps in between each try.

Once all retries are exhausted, the workflow stops.  If, after 3 days, the database copy is still F&S, the workflow state is reset and it starts again from Step 1.  This reset/resume behavior is useful (and intentional) since it can take a few days to replace a failed disk, controller, etc..

Update-MailboxDatabaseCopy

The Update-MailboxDatabaseCopy cmdlet includes some new parameters in CU1 that are designed to aid with automation of seeding operations.  These parameters include:

• BeginSeed – This is useful for scripting reseeds, because with this parameter, the task asynchronously starts the seeding operation and then exits the cmdlet.
• MaximumSeedsInParallel – This is used with the Server parameter to specify the maximum number of parallel seeding operations that should occur across the specified server during a full server reseed operation. The default value is 10.
• SafeDeleteExistingFiles – This is used to perform a seeding operation with a single copy redundancy pre-check prior to the seed. Because this parameter includes the redundancy safety check, it requires a lower level of permissions than the DeleteExistingFiles parameter, enabling a limited permission administrator to perform the seeding operation.
• Server – This is used as part of a full server reseed operation to reseed all database copies in a Failed and Suspended state. It can be used with the MaximumSeedsInParallel parameter to start reseeds of database copies in parallel across the specified server in batches of up to the value of the MaximumSeedsInParallel parameter copies at a time.

Set-DatabaseAvailabilityGroup

The Set-DatabaseAvailabilityGroup cmdlet includes a new parameter named SkipDagValidation.  It is used to bypass the validation of the DAG's quorum model and the health check on the DAG's witness during certain DAG configuration operations.  While this parameter has some usefulness for us in Exchange Online (and that is why it was introduced), and while it is enabled for on-premises use, it won’t be of much use to on-premises environments. I’m only pointing it out because, as I said, it is enabled for on-premises use.

Managed Availability: Get-ServerHealth and Get-HealthReport

The Get-ServerHealth and Get-HealthReport cmdlets are used to get and process raw health set data from Managed Availability, the new monitoring and recovery framework used by the various components within Exchange. Get-ServerHealth can be used to view the various health sets and their current status.  In Exchange 2013 RTM, the Get-HealthReport cmdlet consumed results from Get-ServerHealth to produce a summary rollup of health.  But the way in which it was implemented made it very slow and inefficient.

With CU1, instead of piping Get-ServerHealth to Get-HealthReport, Get-HealthReport is now capable of reporting the consolidated results on its own, and it now takes an Identity parameter that enables you to specify a server instead of InputObject/InputEntries.  Get-HealthReport also includes a new HealthSet parameter, which is used to return the health state for a group of monitors. However, to use a rollup group, a list of names must be pipelined to Get-HealthReport. Unfortunately, Get-HealthReport -Identity does not support an array of names, so our recommended way to do this is to simply get the list of DAG members and pipe that to Get-HealthReport. For example to display a rollup summary of transport health on members of a DAG, you would run:

(Get-DatabaseAvailabilityGroup DAG1).Servers | Get-HealthReport -RollupGroup -HealthSet HubTransport

There are a couple of changes for Get-ServerHealth in CU1; namely, two parameters have also been added:

• HaImpactingOnly – This is used to display only monitors that have HaImpacting set.
• HealthSet – This is used to return the health state of a group of monitors.

Best Copy and Server Selection Changes

Best Copy and Server Selection (BCSS) is the algorithm used by Active Manager in Exchange 2013 to select the best database copy to activate in response to a failover or a target-less switchover. In CU1, a change was made so that the Primary Active Manager (PAM) now keeps track of the number of active databases per server, so that during BCSS it can honor the value of MaximumActiveDatabases, if configured.  The server holding the PAM role now keeps an in-memory state that tracks the number of active databases per server. When the PAM role moves or when the Exchange Replication service is restarted on the PAM, this information is rebuilt from the cluster database.

This change allows Active Manager to exclude servers that are already hosting the maximum amount of active databases when determining potential candidates for activation.  Prior to this change, Active Manager would not evaluate whether a potential server candidate for activation was already at its configured active database limit. Thus, if such a server were selected for activation, the activation process would fail during the mount attempt, and a new server would have to be selected (if available). This scenario is now avoided as a result of this change.

Other CU1 Changes

Of course there are other changes in CU1 besides the above, so be sure to read the Release Notes and other appropriate documentation when everything is released.

Although Exchange 2013 continues to use DAGs and Windows Failover Clustering for Mailbox server role high availability and site resilience, site resilience is not the same in Exchange 2013. Site resilience is much better in Exchange 2013 because it has been operationally simplified. The underlying architectural changes that were made in Exchange 2013 have significant impact on site resilience configurations and their recovery aspects.

Challenges with Exchange 2010 Site Resilience

Exchange 2010 undoubtedly made achieving site resilience for the messaging service and data easier than any previous version of Exchange. By combining the native site resilience features in Exchange 2010 with proper planning, you were able to activate a second datacenter to serve a failed datacenter's clients. The process you perform to do this is referred to as a datacenter switchover. This is a well-documented and generally well-understood process, although it takes time to perform, and it requires human intervention in order to begin the process.

Anyone who has performed a datacenter switchover in Exchange 2010 will tell you that they are operationally complex. This is in part because in Exchange 2010, recovery of mailbox data (DAG) and client access (namespace) are tied together. This leads to other challenges for Exchange 2010 in certain scenarios:

• If you lose all or a significant portion of your Client Access servers, or the VIP for the array, or the top of rack (TOR), or if you lose a significant portion of your DAG, you were in a situation where you needed to do a datacenter switchover.
• You could deploy a DAG across two datacenters and host the witness in a third datacenter and enable failover for the Mailbox role for either datacenter. But you didn’t get failover for the messaging service, because the namespace still needed to be switched over for the non-Mailbox roles.

But all of that aside, the biggest challenge with Exchange 2010 is that the namespace is a single point of failure. In Exchange 2010, the most significant single point of failure in the messaging system is the FQDN that you give to users because it tells the user where to go. Changing the IP address for that FQDN is not that easy because you have to change DNS and deal with DNS latency, which in some parts of the world is very bad. And you have name caches in browsers which are typically around 30 minutes or more that also have to be dealt with.

Exchange 2013 Addresses the Challenges

Significant changes have been made in Exchange 2013 that address the challenges with Exchange 2010 site resilience head on. With the namespace simplification, consolidation of server roles, removal of AD-sited-ness, separation of CAS array and DAG recovery, and load balancing changes, Exchange 2013 provides new site resilience options, such as the ability to use a single global namespace. In addition, for customers with more than two locations in which to deploy messaging service components, Exchange 2013 also provides the ability to configure the messaging service for automatic failover in response to failures that required manual intervention in Exchange 2010.

Specifically, site resilience has been operationally simplified in Exchange 2013. In addition, in Exchange 2013, the namespace does not need to move with the DAG. Exchange leverages fault tolerance built into the namespace through multiple IP addresses, load balancing (and if need be, the ability to take servers in and out of service). One of the most significant changes we made in Exchange 2013 was to leverage the clients’ ability to get more than one place to go. Assuming the client has the ability to use more than one place to go (which almost all HTTP clients do, and since almost all of the client access protocols in Exchange 2013 are HTTP based (Outlook, Outlook Anywhere, EAS, EWS, OWA, EAC, RPS, etc.), all supported HTTP clients have the ability to use multiple IP addresses), thereby providing failover on the client side. You can configure DNS to hand multiple IP addresses to a client during name resolution. The client asks for mail.contoso.com and gets back two IP addresses, or four IP addresses, for example. However many IP addresses the client gets back will be used reliably by the client. This makes the client a lot better off because if one of the IP addresses fails, the client has one or more others to try to connect to. If a client tries one and it fails, it waits around 20 seconds and then tries the next one in the list. Thus, if you lose the VIP for the CAS array, and you have a second VIP for a second CAS array, recovery for the clients happens automatically, and in about 21 seconds.

Modern HTTP clients (operating systems and Web browsers that are ten years old or less) simply work with this redundancy automatically. The HTTP stack can accept multiple IP addresses for an FQDN, and if the first IP it tries fails hard (e.g., cannot connect), it will try the next IP in the list. In a soft failure (connect lost after session established, perhaps due to an intermittent failure in the service where, for example, a device is dropping packets and needs to be taken out of service), the user might need to refresh their browser.

Operationally Simplified Site Resilience

So what does it mean that site resilience has been operationally simplified in Exchange 2013? Going back to the failure scenarios discussed above for Exchange 2010, if you encounter those scenarios in Exchange 2013, depending on your site resilience configuration, you might not need to perform a datacenter switchover. With the proper configuration, failover will happen at the client level and clients will be automatically redirected to a second datacenter that has operating Client Access servers, and those operating Client Access servers will proxy the communication back to the user’s Mailbox server, which remains unaffected by the outage (because you don’t do a switchover). Instead of working to recover service, the service recovers itself and you can focus on fixing the core issue (e.g., replacing the failed load balancer). Any administrator will tell you that the stress involved with replacing a failed piece of equipment that isn’t blocking service is much lower than the stress involved in restoring service and data access via a datacenter switchover.

So by comparison, in Exchange 2010, if you lose the load balancer in your primary datacenter and you don’t have another one in that site, you had to do a datacenter switchover. In Exchange 2013, if you lose the load balancer in your primary site, you simply turn it off (or maybe turn off the VIP) and repair/replace it.

Clients that aren’t already using the VIP in the secondary datacenter will automatically failover to the secondary VIP without any change of namespace, and without any change in DNS. Not only does that mean you no longer have to perform a switchover, but it also means that all of the time normally associated with a datacenter switchover recovery is not spent either. In Exchange 2010, you had to deal with DNS latency (hence, the recommendation to set the TTL to 5 min, and the introduction of the Failback URL). In Exchange 2013, you don’t need to do that because you get fast failover (~21 seconds) of the namespace between VIPs (datacenters).

Since you can failover the namespace between datacenters now, all that is needed to achieve a datacenter failover is a mechanism for failover of the Mailbox role across datacenters. To get automatic failover for the DAG, you simply architect a solution where the DAG is evenly split between two datacenters, and then place the witness server in a third location so that it can be arbitrated by DAG members in either datacenter, regardless of the state of the network between the datacenters that contain the DAG members. The key is that third location is isolated from network failures that affect the first and/or second location (the locations containing the DAG members).

In this scenario, the administrator’s efforts are geared toward simply fixing the problem, and not spent restoring service. You simply fix the thing that failed; all the while service has been running and data integrity has been maintained. The urgency and stress level you feel when fixing a broken device is nothing like the urgency and stress you feel when you’re working to restore service. It’s better for the end user, and less stressful for the admin.

You can allow failover to occur without having to perform switchbacks (sometimes mistakenly referred to as failbacks). If you lose CAS in your primary datacenter and that results in a 20 second interruption for clients, you might not even care about failing back. At this point, your primary concern would be fixing the core issue (e.g., replacing the failed load balancer). Once that is back online and functioning, some clients will start using it, while others might remain operational through the second datacenter.

Exchange 2013 also provides functionality that enables administrators to deal with intermittent failures. An intermittent failure is where, for example, the initial TCP connection can be made, but nothing happens afterwards. An intermittent failure requires some sort of extra administrative action to be taken because it might be the result of a replacement device being put into service. While this repair process is happening, the device might be powered on and accepting some requests, but not really ready to service clients until the necessary configuration steps are performed. In this scenario, the administrator can perform a namespace switchover by simply removing the VIP for the device being replaced from DNS. Then during that service period, no clients will be trying to connect to it. Once the replacement process has completed, the administrator can add the VIP back to DNS and clients will eventually start using it.

Conclusion

Microsoft Exchange Server 2013 continues to innovate in the areas of storage, high availability, and site resilience, with its plethora of new, innovative features, such as:

• Multiple databases per volume
• Autoreseed
• Automatic recovery from storage failures
• Lagged copy enhancements
• Managed Availability
• Best Copy and Server Selection enhancements
• Maintenance Mode
• Automatic DAG network configuration
• Operationally simplified site resilience
• Separation of Mailbox and Client Access recovery
• Leveraging client-side DNS behaviors, such as IP failover for the namespace

It’s important to understand that major architectural changes had to take place in Exchange 2013 in order to enable these features. That means that while the Exchange 2010 design guidelines can apply to an Exchange 2013 organization, the additional enhanced Exchange 2013 design guidelines cannot be applied to Exchange 2010. All of the goodness above around new behaviors and design options applies to Exchange 2013 only.