Hi

Have you noticed the Office 365 directory sync database just getting larger and larger… If so try the following to clear the sync runs

1. Open the MIISClient.exe and click the Operations tab.

2. From the Actions menu, select "Clear Runs"

3. On the Clear Runs dialog

a. Uncheck the option "save runs before clearing them"

b. Click to select "Clear runs before: " radio button

c. Select a date that will clear runs starting with the oldest, not trying to clear more than about a week at a time.

IMPORTANT: This is pretty important as this operation will cause the transaction log file to grow temporarily. When trying to clear too many runs all at once, the transaction log file could grow a lot, possibly running out of disk space on the drive hosting the file.

4. Wait for it to finish the clear operation

a. This will be apparent when the list of runs has been modified to remove the runs just cleared.

5. Repeat as necessary

Written by Daniel Kenyon-Smith

HCW Issues:-

I have seen issues with the HCW failing on Get-FederationInformation, this can be caused by various things such as:-

·         The customer did not publish the autodiscover endpoints correctly to allow O365 to make federated autodiscover lookups

·         There is pre-authentication set on the TMG/ISA or UAG. By connecting to https://autodiscover.company.com/autodiscover/autodiscover.xml for example as shown in Figure 1 - Autodiscover pre-auth authenticating - UAG and Figure 2 - Autodiscover pre-auth authenticating - TMG, you can see that UAG/TMG is prompting for pre-auth instead of the auth being directly on exchange, as shown in Figure 3 - No pre-authentication

Figure 1 - Autodiscover pre-auth authenticating - UAG

Figure 2 - Autodiscover pre-auth authenticating - TMG

Figure 3 - No pre-authentication

·         You can also run the command in powershell with the –verbose switch to get more detail, such as get-federationinformation……-verbose

·         If you are using TMG you can follow this article for configuring TMG with hybrid setups

·         You can also check autodiscover and other O365 endpoints (such as AD FS) have been published and are reachable from the internet using the ExRCA

Another issue I have seen is with certificates and as you run through the HCW, under Manage Hybrid Configuration – Mail Flow Security, you are asked to select the certificate that will be used for TLS mail flow between on-prem on and the cloud. If you are using a wildcard cert such as *.company.com then you will need to ensure you are running at least Exchange 2010 SP2 RU1. If you are not running SP2 RU1 then you most probably see an empty box when you get to the certificate page as shown in Figure 4 - HCW certificate blank below.

Figure 4 - HCW certificate blank

Of course you need to also ensure the certificate has been installed onto the exchange hybrid server(s) via the EMC or powershell, you can check using get-exchangecertificate from powershell.

Hybrid steps

Below are the detailed steps that run behind the scenes when you start the hybrid wizard:-

Create Federation Delegation and Organizational Relationships

Creates a new Delegation Federated Trust to 'Microsoft Federation Gateway'

Creates new 'On Premises to Exchange Online Organization Relationship'

Creates new 'Exchange Online to on premises Organization Relationship'

Enables MRSProxy on the Exchange 2010 Hybrid Servers

Configure the 'On Premises to Exchange Online Organization Relationship' to set:-

MailboxMoveEnabled 'True'

FreeBusyAccessEnabled 'True'

FreeBusyAccessLevel 'LimitedDetails' –

ArchiveAccessEnabled 'True'

MailTipsAccessEnabled 'True'

MailTipsAccessLevel 'All'

DeliveryReportEnabled  'True'

TargetOwaURL 'http://outlook.com/owa/<company.com>

Configure the 'Exchange Online to on premises Organization Relationship' to set:-

FreeBusyAccessEnabled 'True'

FreeBusyAccessLevel 'LimitedDetails'

MailTipsAccessEnabled 'True'

MailTipsAccessLevel 'All'

DeliveryReportEnabled 'True'

Create Send and Receive Connectors

HCW creates a new On Premise Send Connector -Name 'Outbound to Office 365' and below is the full output from the send connector created by the HCW:-

AddressSpaces                : {smtp:TenantName.mail.onmicrosoft.com;1}

AuthenticationCredential     :

Comment                      :

ConnectedDomains             : {}

ConnectionInactivityTimeOut  : 00:10:00

DNSRoutingEnabled            : True

DomainSecureEnabled          : False

Enabled                      : True

ErrorPolicies                : DowngradeAuthFailures

ForceHELO                    : False

Fqdn                         : Company.com

HomeMTA                      : Microsoft MTA

HomeMtaServerId              : ServerName

Identity                     : Outbound to Office 365

IgnoreSTARTTLS               : False

IsScopedConnector            : False

IsSmtpConnector              : True

LinkedReceiveConnector       :

MaxMessageSize               : 10 MB (You can increase from 10MB which is the default in 2007 and 2010 to 25MB which is set in the service)

Name                         : Outbound to Office 365

Port                         : 25

ProtocolLoggingLevel         : None

RequireOorg                  : False

RequireTLS                   : True

SmartHostAuthMechanism       : None

SmartHosts                   : {}

SmartHostsString             :

SmtpMaxMessagesPerConnection : 20

SourceIPAddress              : 0.0.0.0

SourceRoutingGroup           : Exchange Routing Group (DWBGZMFD01QNBJR)

SourceTransportServers       : {ServerName}

TlsAuthLevel                 : DomainValidation

TlsDomain                    : outlook.com

UseExternalDNSServersEnabled : False

HCW creates a new On Premise Receive Connector 'Inbound from Office 365' on each of the Hybrid HT servers and below is the full output from the receive connector created by the HCW:-

AuthMechanism                           : Tls, Integrated, BasicAuth, BasicAuthRequireTLS

Banner                                  :

BinaryMimeEnabled                       : True

Bindings                                : {LocalIP:25}

ChunkingEnabled                         : True

DefaultDomain                           :

DeliveryStatusNotificationEnabled       : True

EightBitMimeEnabled                     : True

BareLinefeedRejectionEnabled            : False

DomainSecureEnabled                     : False

EnhancedStatusCodesEnabled              : True

LongAddressesEnabled                    : False

OrarEnabled                             : False

SuppressXAnonymousTls                   : False

AdvertiseClientSettings                 : False

Fqdn                                    : Company.com

Comment                                 :

Enabled                                 : True

ConnectionTimeout                       : 00:10:00

ConnectionInactivityTimeout             : 00:05:00

MessageRateLimit                        : unlimited

MessageRateSource                       : IPAddress

MaxInboundConnection                    : 5000

MaxInboundConnectionPerSource           : 20

MaxInboundConnectionPercentagePerSource : 2

MaxHeaderSize                           : 64 KB (65,536 bytes)

MaxHopCount                             : 60

MaxLocalHopCount                        : 12

MaxLogonFailures                        : 3

MaxMessageSize                          : 28 MB (29,360,128 bytes)

MaxProtocolErrors                       : 5

MaxRecipientsPerMessage                 : 200

PermissionGroups                        : AnonymousUsers

PipeliningEnabled                       : True

ProtocolLoggingLevel                    : None

RemoteIPRanges                          : {RemoteIP Ranges}

RequireEHLODomain                       : False

RequireTLS                              : True

EnableAuthGSSAPI                        : False

ExtendedProtectionPolicy                : None

LiveCredentialEnabled                   : False

TlsDomainCapabilities                   : {outlook.com:AcceptOorgProtocol}

Server                                  : LocalServerName

SizeEnabled                             : Enabled

TarpitInterval                          : 00:00:05

MaxAcknowledgementDelay                 : 00:00:30

AdminDisplayName                        :

ExchangeVersion                         : 0.1 (8.0.535.0)

Name                                    : Inbound from Office 365

DistinguishedName                       : CN=Inbound from Office 365,CN=SMTP Receive Connectors,CN=Protocols,etc…

Identity                                : LocalServerName\Inbound from Office 365

Guid                                    : 9feef51e-1bd9-4aa4-9202-0614a1fcc0dd

ObjectCategory                          : company.com/Configuration/Schema/ms-Exch-Smtp-Receive-Connector

ObjectClass                             : {top, msExchSmtpReceiveConnector}

OriginatingServer                       : ServerName

IsValid                                 : True

There are also some connectors created in Forefront Online Protection for Exchange (FOPE). The connector created in FOPE will be called ‘Hybrid Mail Flow Inbound Connector’ and will have the following settings (assuming you are routing mail back on-premise):-

Description: The Hybrid Mail Flow inbound connector was created when hybrid mail flow was configured. This connector cannot be modified.
Sender Domains: *.*
Sender IP Addresses: 
Transport Layer Security (TLS) Settings: Forced TLS, and certificate matches specified domain

The recipient certificate matches: mail.company.com

Filtering Settings: Add these IP addresses to the safelist and only accept mail from these IP addresses for the domains specified above
IP Reputation Filtering: Disabled
Spam Filtering: Disabled
Policy Rules: Disabled

The other connector created in FOPE will be called ‘Hybrid Mail Flow Outbound Connector’ and will have the following settings:-

Description: The Hybrid Mail Flow outbound connector was created when hybrid mail flow was configured. This connector cannot be modified.
Recipient Domains: mail.company.com,*.*
Message Delivery Settings:

Fully Qualified Domain Name: mail.company.com

Transport Layer Security (TLS) Settings: Forced TLS, and certificate matches specified domain

The recipient certificate matches: mail.company.com

Create Remote Domains

Create new Remote Domain 'Hybrid Domain – company.com' set with:-

TrustedMailInbound 'True'

Create new Remote Domain 'Hybrid Domain - TenantName.mail.onmicrosoft.com' set with:-

 TrustedMailOutbound 'True'

TargetDeliveryDomain 'True'

AllowedOOFType 'InternalLegacy'

AutoReplyEnabled 'True'

AutoForwardEnabled 'True'

DeliveryReportEnabled 'True'

DisplaySenderName 'True'

NDREnabled 'True'

TNEFEnabled 'True'

Create new Remote Domain 'Hybrid Domain - mail.company.com '

DomainName 'mail.company.com'

TrustedMailInbound 'True

Setup Hybrid Mailflow

Set Hybrid Mailflow to:-

SecureMailEnabled 'True'

CentralizedTransportEnabled 'True'

OnPremisesFQDN 'mail.company.com '

CertificateSubject 'mail.company.com '

InboundIPs <>

OutboundDomains <>

 Set Address Policies

Update Default Recipient Policy to add <alias>@TenantName.mail.microsoft.com

Apply the updated Default Recipient Policy immediately

Written by Daniel Kenyon-Smith

We used the basic setup of 2 AD FS proxy servers in the DMZ and 2 internal federation servers to give us high availability. We had TMG sat in front of the AD FS proxy boxes, an example of the rule is shown below:-

Create a TMG Firewall Policy for AD FS

1. Log in to TMG as Domain Admin and launch Forefront TMG Management from the Start menu

2. Select Firewall Policy and select Publish Web Sites

3. Web publishing rule name: AD FS and click Next

4. Select Allow and click Next

5. Select Publish a single Web site or load balancer and click Next

6. Select Use SSL to connect to the published Web server or server farm and click Next

7. Internal site name: <your-federation-service-name

8. Check the checkbox to Use a computer name or IP address to connect to the published server

9. Computer name or IP address: <your-adfs-proxy> and click Next

10. Path: /* and click Next

11. Accept requests for: This domain name (type below):

12. Public name: <your-federation-service-name>

13. Path: /*

14. Click Next

15. Click New

16. Web listener name: AD FS Listener and click Next

17. Select Require SSL secured connections with clients and click Next

18. Check the checkbox next to External and click Next

19. Click Select Certificate, select your SSL cert certificate, click Select, and click Next

20. Select how clients will provide credentials to Forefront TMG: No Authentication and click Next and Next and Finish

21. Click Next, select No delegation, but client may authenticate directly, and click Next

22. Make sure All Users is displayed and click Next and Finish

23. Right-click the AD FS policy and select Configure HTTP

24. On the General tab, uncheck the checkbox for Verify normalization and click OK

25. Right-click the AD FS policy and select Properties

26. Select the Link Translation tab, uncheck the checkbox for Apply link translation to this rule, and click OK

27. Click Apply at the top of the TMG console

Follow this link for ISA - http://blog.c7solutions.com/2011/06/publishing-adfs-through-isa-or-tmg.html and it’s also worth noting that Forefront UAG works only with the WS-Federation Passive protocol (e.g. browser access, such as the portal and OWA) not the active (e.g. Exchange ActiveSync, Outlook 2007, Outlook 2010, IMAP, POP, SMTP and Exchange Web Services) or MEX (e.g. Office subscriptions and Lync) see here for more details.

We also just used the Windows Internal Database (WID) as we don’t require more than 5 federation servers in the farm. The WID uses a primary/secondary model to replicate the database to all of the nodes in the federation farm. Because only one server is configured as the primary server, configuration changes can only be made on that server. The federation services on the secondary nodes are not able to modify the database. In this configuration, there are two features that cannot be implemented because they rely on database changes across all nodes. These two features are Token Replay Detection and SAML Artifact Resolution (for more details see here).

We followed the TechNet article ‘Limiting Access to Office 365 Services Based on the Location of the Client’ to create our own custom rule to lock down access to O365. The rule we created was ‘Allow all external access to Office 365, except Browser based access’ this is the rule we created:-

exists([Type == "http://schemas.microsoft.com/2012/01/requestcontext/claims/x-ms-proxy"]) &&

NOT exists([Type == "http://schemas.microsoft.com/2012/01/requestcontext/claims/x-ms-forwarded-client-ip",

Value=~"\b1\.2\.3\.4\b"]) &&

exists([Type == "http://schemas.microsoft.com/2012/01/requestcontext/claims/x-ms-endpoint-absolute-path", Value == "/adfs/ls/"])

=> issue(Type = "http://schemas.microsoft.com/authorization/claims/deny", Value = "True");

If we break down the rule we can see what it’s doing and it’s a little confusing as it uses double negatives! When the Value = True in the claims deny (last line in the rule) and the expression is set to NOT exist it allows that expression through!

So looking at our rule we are allowing ‘x-ms-forwarded-client-ip’ with a value IP address of 1.2.3.4 which is the public IP address from our internet facing TMG (so the NAT’d IP from internal to external).

We are not allowing ‘x-ms-proxy’ or ‘x-ms-endpoint-absolute-path’ the absolute path is what’s blocking the browser access. So when we try to access OWA for example from a none managed device (e.g. not tunneled through the corporate network using the internet proxy servers and appearing to come from 1.2.3.4) but instead coming from another IP, such as home broadband, we get access denied as below.

Access Denied

There was a problem accessing the site. Try to browse to the site again.

If the problem persists, contact the administrator of this site and provide the reference number to identify the problem.

You are not authorized to access this site. Contact your administrator for more information.

Reference number: 11111111-1111-1111-1111-111111111111

You can then lock down POP and IMAP using the following link if required

Written by Daniel Kenyon-Smith

Flood mitigation on Threat Management Gateway (TMG) - http://community.office365.com/en-us/w/exchange/office-365-move-mailbox-fails-with-transient-exception.aspx

Written by Daniel Kenyon-Smith

Whilst working with a customer here in the UK on their O365 project, they asked about using Forefront Online Protection for Exchange (FOPE) (the version that comes with O365) how to setup rules to block credit cards details. There are various rues you can configure, Basic or RegEx, more details can be found here.

We setup up the following rules also using this site:-

The credit card format is like this 1234-1234-1234-1234, so we set up rues to cover various scenarios, such as:-

Credit Card Format	FOPE Rule
1234-1234-1234-1234	\d\d\d\d\-\d\d\d\d\-\d\d\d\d\-\d\d\d\d
1234 1234 1234 1234	\d\d\d\d\s\d\d\d\d\s\d\d\d\d\s\d\d\d\d
1234123412341234	\d\d\d\d\d\d\d\d\d\d\d\d\d\d\d\d

The top 2 rules for example can be added into a single rule, as shown below and in the screenshot

\d\d\d\d\s\d\d\d\d\s\d\d\d\d\s\d\d\d\d|\d\d\d\d\-\d\d\d\d\-\d\d\d\d\-\d\d\d\d

To configure and create new rules, follow the steps below:-

1. Login to FOPE admin center - https://Admin.messaging.microsoft.com (also as a side note, this is the site to access quarantine messages - https://Admin.messaging.microsoft.com, but also note that global admins can’t manage quarantine by default - http://support.microsoft.com/kb/2587698)

2. Select Administration

3. Policy Rules

4. New Policy Rule, on the right hand side under Tasks

5. Then configure the rule accordingly, so in this example we using body to match the rule

Since we are in the UK we are not worried about social security numbers, but we did add in American Express (Amex) formats too, as show below:-

Credit Card Format	FOPE Rule
1234 123456 12345	\d\d\d\d\s\d\d\d\d\d\d\s\d\d\d\d\d
1234-123456-12345	\d\d\d\d\-\d\d\d\d\d\d\-\d\d\d\d\d
123412345612345	\d\d\d\d\d\d\d\d\d\d\d\d\d\d\d

Written by Daniel Kenyon-Smith

Basically I like to write in Word rather than Live Writer. Live Writer is great but I spend a lot of time writing documentation in Microsoft Word 2010 and so that is my preferred writing tool. Most of my blog posts start off life in Word 2010 and then I have to transfer them over to Live Writer to get the content uploaded to my Blog. The problem is that the formatting is often lost during the transfer and I have to spend time trying to make it look how I wanted in the first place… surely there must be a way to get Word 2010 to write to my TechNet blog in the first place? Surely?? J

So… after some searching and some trial and error I have finally managed to come up with a solution to connect Word 2010 with the TechNet Blog platform…

Connecting Word 2010 to TechNet Blog Platform

1. Open Microsoft Word 2010
2. Click File -> New
3. Pick Blog Post from the Available Templates
4. Word should prompt you to Register a Blog Account at this stage
5. Select Register Now
6. In the Choose your blog provider drop down, select Other and click Next
7. Select MetaWebLog in the API type
8. Enter your TechNet blog URL with /metablog.ashx tagged on the end – in my case this ends up being http://blogs.technet.com/b/neiljohn/metablog.ashx
9. Enter your username and password

1. Click OK
2. Your TechNet Blog account is now registered in Word 2010

Hopefully someone else out there will find this useful J

Well.. I am pleased to say that Ross Smith IV has written a fantastic article that explains what is actually going on inside these mystical processes and what they actually do for us…

http://blogs.technet.com/b/exchange/archive/2011/12/14/database-maintenance-in-exchange-2010.aspx

So, I have decided to write a post today about all of the others posts instead.

Firstly here is a link to download SP2 if that's all you need – I still recommend that you skim through the release notes, even if you are progressing through your own test plan since it may save you some time.

• http://www.microsoft.com/download/en/details.aspx?id=28190
  

Released: Exchange Server 2010 SP2

The Exchange Team Blog (this is owned by the Microsoft Exchange Server Product Group if you were not aware) announced the release of SP2 on Dec 5th at 6.38pm with the following post…

• http://blogs.technet.com/b/exchange/archive/2011/12/05/released-exchange-server-2010-sp2.aspx

Some interesting things to note…

1. It requires a schema update (just like every other Exchange SP since 2007!)
2. It provides the following new features…
   

• Hybrid Configuration Wizard
• Address Book Policies
• Cross-Site Silent Redirection for Outlook Web App
• Mini Version of Outlook Web App
• Mailbox Replication Service
• Mailbox Auto-Mapping
• Multi-Valued Custom Attributes
• Litigation Hold

Exchange Server 2010 SP2 Release Notes

Before you consider deploying any new service pack for Exchange I strongly urge you to take a look at the release notes…

• http://technet.microsoft.com/en-us/library/hh529928.aspx

For this release the following areas may require attention – all are discussed in the release notes.

• Installation Prerequisites for Client Access Servers has changed
• Possible issue with RBAC on non SP2 servers after upgrading one server to SP2 in the organisation
• Outlook Web App redirection issue after upgrading to SP2
• Mailbox Replication Proxy
• Hybrid Configuration Wizard error with domains starting with a numeric value

It is worth noting here that Exchange Service packs are inclusive of all previous service packs and update rollups.  This means that Exchange Server 2010 SP2 includes the fixes and changes up to and including Update Rollup 6 for Exchange Server 2010 Service Pack 1.

Exchange Server 2010 SP2 Internet Content

This is a subset of the non-TechNet material I recommend reading to get up to speed on Exchange Server 2010 SP2

• Greg Taylor : Featuring GAL Segmentation (This is a TechEd recording on ABP’s)
• Greg Taylor: Exchange Server 2010 SP2 and Support for Hosted Exchange
• Henrik Walther: Changes to the MRSProxy Service Configuration
• Tony Redmond – Installing Exchange 2010 SP2
  

Other Interesting Stuff

I spotted that we released a new version of the Exchange Server User Monitor yesterday also – I don't know if this is required if you deploy SP2 but I thought I would include it here just in case

• Exchange Server User Monitor 14.2.247.5 (EXMON)

Hopefully this provides enough information in one place to save you searching the rest of the internet for your Exchange Server SP2 news

• http://blogs.technet.com/b/exchange/archive/2011/11/17/windows-disk-timeouts-and-exchange-server-2010.aspx

Whilst writing that article and especially researching how ESE timeouts have evolved, I began thinking about the history of Exchange availability and how the product has evolved over my time working with it…

In the beginning…

Back when I began in enterprise infrastructure (1997) the name of the game was all about server uptime, not necessarily service uptime.  What was interesting back then was that our servers were monitored for availability but we didn't have the technology to record service available very well.  We didn't really have many options to improve our service availability with Exchange 5.5 – I used to pack out my servers with resilient components such as redundant network cards, power supplies and raid controllers to ensure that they would keep running for as long as possible, but if the server did go down our only option was to bring it back up as quickly as possible.  My first concern during an outage was always the health of my 30GB EDB files – I knew that if one of these was damaged or the storage was unrecoverable then I was in for a minimum 10 hour restore from tape, I also knew that my manager would be pretty unimpressed  when I delivered this news, so I would go to elaborate lengths to try and avoid it.

Towards the end of my time with Exchange 5.5 we were encouraged (by Microsoft) to look at clustering technology.  I duly did this in my test lab (with real physical servers and a SCSI “Y” cable!) and I have to admit that I was pretty impressed initially.  This clustering I would be able to tolerate an entire server failure and be back online in a few minutes.  This was a huge step forward. 

It was only when I started to write my business case to justify my new hardware requirements, I realised that over the previous two years all of my serious outages had come from storage issues (Raid controller firmware, cache failure, multiple HDD failure).  I was hoping to use previous outage details to justify my request for new Exchange clusters, but the reality was that we had only had 1 real server failure (main board) and I was able to replace that fairly quickly with one from the test lab and it was back online in 2 hours.  All of the serious service outages had come from storage related failures where i had to revert to tape restoration.  I realised right at this point that clustering based on a shared storage model probably wasn't going to give me what i needed.

The next level…

A few years later I was going through the design phase for Exchange 2003.  My primary goal for the new platform was to improve our service levels, which I was informed by our service delivery manager were fairly steady at 99% for messaging.  I was still very aware at this point that to do this I needed to improve my restore times from tape.  Some progress had been made with the switch to DLT drives, but my EDB files were growing at an incredible rate and it only took 18 months before my restore times were back at 10 hours per server again.  I re-visited clustering during this timeframe but came to the same conclusion as before that it was just adding complexity and wasn't going to materially effect my service availability.  Instead I chose some fancy SAN technology which allowed me to use snapshots to recover my EDB files in minutes rather than hours.  This technology also allowed me to mirror my data off-site.  Management were duly impressed with my solution, until they realised how much it was going to cost!.  Still, they eventually implemented the design and service levels did improve somewhat (although not as much as I had hoped).

What I had totally neglected during this design was that my new storage technology was a lot more complicated than my old raid controllers were.  I quickly discovered that it was very easy to break things on a monumental scale.  One wrong command given to my clever new storage technology would be sufficient to stop the entire messaging service.  This realization very quickly lead me to adopt the “if it isn't broke, don't fix it” approach to maintaining service.  This brought its own issues though and I was faced with an annual update task to bring the messaging servers back inline with our enterprise standards, this was always a disaster and would entail almost a days worth of downtime while we tried to get the magical combination of OS hotfixes, drivers, firmware and SAN revision right.  Then, once we had a shiny rack of twinkly green lights we would leave it alone for another year…

Clustering that works?

Exchange 2007 seemed to address all of my previous concerns.  I was now working for Microsoft so I was getting to see lots of customers, all struggling with the same basic issue I had previously experienced as a customer, i.e how do we recover service quickly in the event of a database problem.  Exchange 2007 seemed to solve this elegantly with the introduction of CCR clusters.  I loved the simplicity of this solution – just copy all of the changes from the active server to the passive server and then fail over in the event of a problem.  It meant that even in a serious failure scenario we could just bring up our passive copy and be back online in minutes!  The solution was out-of-box and so there were no supportability issues either!  Customers loved CCR and I thought it was the best Exchange feature ever developed.

Over time though I started to see issues with the CCR model.  The most frustrating was that we had to fail the whole server over to the passive node.  This meant that if we had a isolated storage failure on the active node we had a difficult decision to make… leave the users on the failed database offline until a maintenance window was available, or interrupt service for everyone on the server and fail its workload over to the passive node.  Not a great position to be in and often it would depend “who” was on the failed database and how important they were, rather than anything more scientific!

Service Availability?

Exchange 2010 took the CCR model and addressed many of the issues reported by customers.  Now we could fail over individual databases between server nodes.  This was a huge step forward and meant that many customers were now actually hitting 99.9% availability.  Given that best practices were followed it took an unusual situation to take Exchange 2010 offline for any significant period of time.

Over the past couple of years I have noticed some interesting service outages for my customers though.  Given that the server experiences a clean failure of a storage component the database hosted on that storage will simply move to an alternate copy and service will be resumed.  However, if the storage device does not fail cleanly and instead just begins responding slowly or intermittently the database will not simply fail over.  This is quite common with JBOD or where the storage controller suffers an unusual failure, such as overheating or memory corruption.

Self Sacrifice

Now we return to the present day (well, earlier this year).  I was in a storage design meeting with a customer and their core storage vendors.  I began talking about changes in Exchange 2010 SP1 and how Exchange would force BSOD (bugcheck) a server if we didn't hear back from a LUN in 4 minutes.  At this point I could hear a sharp intake of breath from pretty much everyone involved.  The feeling in the room was that forcing a server to crash reboot was insane!

I began to question this behaviour myself and started to think back to some of the failures I had seen and how or if this behaviour would have helped.  I have to admit that forcing a server to blue screen does seem pretty extreme, however upon reflection I came to the conclusion that I quite liked this behaviour.  Given that we have multiple independent copies of our database in a DAG then would I rather a workload remained on a server with a storage I/O problem (4 minutes is a long time to get a response back from your storage!) or that it was moved to another copy?  Well, obviously I want it to be moved and actually I'm not even sure i want to wait 4 minutes!

For me the decision is pretty obvious.  If I have multiple independent copies of my databases, I want Exchange to switch over intelligently if it detects a problem with the currently active copy.  It may seem counterintuitive to crash reboot a server hosting an active service to improve service availability, but the alternative is to leave the service running on a wounded server until a human being comes along and does the same thing.  Frequently in my support days I would arrive in the datacentre to find a hung server with a black screen.  I would try remote RDP access, maybe trigger a remote reboot via RPC but all too frequently once a server gets into this state you need to press and hold the big red button on the front to get it to come back up and begin troubleshooting the event logs.  All of the time the server is in this hung state the service is unavailable.  By triggering a bugcheck someone still needs to troubleshoot the root cause, but at least the service is only interrupted for a few minutes rather than a few hours…

• http://blogs.technet.com/b/exchange/archive/2011/11/20/recommended-windows-hotfix-for-database-availability-groups-running-windows-server-2008-r2.aspx

A summary of the issue…

“This hotfix is strongly recommended for all database availability groups that are stretched across multiple datacentres. For DAGs that are not stretched across multiple datacentres, this hotfix is good to have, as well. The article describes a race condition and cluster database deadlock issue that can occur when a Windows Failover cluster encounters a transient communication failure. There is a race condition within the reconnection logic of cluster nodes that manifests itself when the cluster has communication failures. When this occurs, it will cause the cluster database to hang, resulting in quorum loss in the failover cluster.”

If you haven't already picked this up I strongly recommend taking a look and assessing if you need it on your production clusters.