Поделиться через

Exchange 2010 – File Share Witness oddities…

  • Статья

In Exchange 2010 when a Database Availability Group (DAG) it utilized, and there is an even number of DAG members, the underlying cluster is implemented utilizing the quorum type Node and File Share Majority. The settings utilized for the File Share Witness are defined on the DAG when the logical DAG object is created and are either set by the administrator or automatically defined.

To verify the quorum type you can use either cluster.exe or cluster powershell extensions (Preferred)

Cluster.exe <cluster> /quorum (Windows 2008 & Windows 2008 R2)

Cluster.exe cluster.domain.com /quorum

Witness Resource Name Path Type

--------------------- --------------------------------------------- --------

File Share Witness (\\HT-1.DOMAIN.COM\DAG.DOMAIN.COM) Majority

Get-Cluster <cluster> | Get-ClusterQuorum | FL (Windows 2008 R2 Only)

Cluster : DAG
QuorumResource : File Share Witness (\\HT-1.DOMAIN.COM\DAG.DOMAIN.COM)
QuorumType : NodeAndFileShareMajority

In Failover Cluster Manager, the resources can be viewed by looking at the Cluster Core Resources.

image

It may become necessary to change the server hosting the file share witness. In Exchange 2010 this is not done utilizing Failover Cluster Manager, but rather utilizing the set-databaseavailabilitygroup commandlet. It is after the witness server is successfully updated that the oddity occurs. Here’s an example:

Currently the DAG utilizes the witness server HT-1. Using the set-databaseavailabilitygroup command the witness server is changed to HT-2. (set-databaseavailabilitygroupserver –witnessServer HT-2) The command returns without error. When running the previous cluster commands the following output is noted:

Cluster.exe cluster.domain.com /quorum (Windows 2008 and Windows 2008 R2)

Witness Resource Name Path Type

--------------------- --------------------------------------------- --------

File Share Witness (\\HT-1.DOMAIN.COM\DAG.DOMAIN.COM) Majority

Get-Cluster <cluster> | Get-ClusterQuorum | FL (Windows 2008 R2 Only)

Cluster : DAG
QuorumResource : File Share Witness (\\HT-1.DOMAIN.COM\DAG.DOMAIN.COM)
QuorumType : NodeAndFileShareMajority

Also in Failover Cluster Manager the following is noted in the cluster core resources group.

image

After looking at this output the administrator could be lead to believe that the witness server did not successfully update. After all both cluster.exe and powershell both show the File Share Witness (\\HT-1.DOMAIN.COM\DAG.DOMAIN.COM). It is only in Failover Cluster Manager, if the windows is fully expanded, that you can see both (\\HT-1.DOMAIN.COM\DAG.DOMAIN.COM) and (\\HT-2.DOMAIN.COM\DAG.DOMAIN.COM). This leads administrators to believe that two file share witness servers are currently in use.

Thankfully both of these perceived conditions are false. The command was both successful in changing the witness server and only one file share witness is in use.

Each cluster resource has a display name and a set of public and private properties. Unfortunately when using set-databaseavailabilitygroup to change the witness server, the File Share Witness resource private property for where the witness is stored is updated but the public property display name, which contains the previous witness server, is not. Let’s take a look at this further.

Using cluster.exe or powershell I can review the private properties of the File Share Witness resource. (Command output truncated to show relevant values only.)

Cluster.exe <cluster> res <resource> /priv <or> /prop (Windows 2008 & Windows 2008 R2)

Cluster.exe cluster.domain.com res “File Share Witness (\\HT-1.domain.com\DAG.domain.com)" /prop

Listing properties for 'File Share Witness (\\HT-1.domain.COM\DAG.domain.COM)':

T Resource Name Value

-- -------------------- ------------------------------ -----------------------

SR File Share Witness (\\HT-1.domain.COM\DAG.domain.COM) Name File Share Witness (\\HT-1.domain.COM\DAG.domain.COM)

Cluster.exe cluster.domain.com res “File Share Witness (\\HT-1.domain.com\DAG.domain.com)" /priv

Listing private properties for 'File Share Witness (\\HT-1.domain.COM\DAG.domain.COM)':

T Resource Name Value

-- -------------------- ------------------------------ -----------------------

S File Share Witness (\\HT-1.domain.COM\DAG.domain.COM) SharePath \\HT-1.domain.com\DAG.domain.com

Get-ClusterResource –Cluster <cluster> –Name <ResourceName> | fl (Windows 2008 R2 Only – Public Properties)

Name : File Share Witness (\\HT-1.domain.COM\DAG.domain.COM)
State : Online
OwnerGroup : Cluster Group
ResourceType : File Share Witness

Get-ClusterResource –Cluster <cluster> –Name <ResourceName> | Get-ClusterParameter fl (Windows 2008 R2 Only – Private Properties)

Name : SharePath
IsReadOnly : False
ParameterType : String
Value : \\HT-1.domain.com\DAG.domain.com

At this time a set-databaseavailability group is issued to change the witness server. After the command completes successfully, the previous commands are run. (Command output truncated to show relevant values only.)

Cluster.exe cluster.domain.com res “File Share Witness (\\HT-1.domain.com\DAG.domain.com)" /prop

Listing properties for 'File Share Witness (\\HT-1.domain.COM\DAG.domain.COM)':

T Resource Name Value

-- -------------------- ------------------------------ -----------------------

SR File Share Witness (\\HT-1.domain.COM\DAG.domain.COM) Name File Share Witness (\\HT-1.domain.COM\DAG.domain.COM)

Cluster.exe cluster.domain.com res “File Share Witness (\\HT-1.domain.com\DAG.domain.com)" /priv

Listing private properties for 'File Share Witness (\\HT-1.domain.COM\DAG.domain.COM)':

T Resource Name Value

-- -------------------- ------------------------------ -----------------------

S File Share Witness (\\HT-1.domain.COM\DAG.domain.COM) SharePath \\HT-2.domain.com\DAG.domain.com

(Note: The SharePath in the previous output reflects the new witness server as expected)

Get-ClusterResource –Cluster <cluster> –Name <ResourceName> | fl (Windows 2008 R2 Only – Public Properties)

Name : File Share Witness (\\HT-1.domain.COM\DAG.domain.COM)
State : Online
OwnerGroup : Cluster Group
ResourceType : File Share Witness

Get-ClusterResource –Cluster <cluster> –Name <ResourceName> | Get-ClusterParameter fl (Windows 2008 R2 Only – Private Properties)

Name : SharePath
IsReadOnly : False
ParameterType : String
Value : \\HT-2.domain.com\DAG.domain.com

(Note: The SharePath in the previous output reflects the new witness server as expected)

As you can see the set-databaseavailability group command did complete it’s task successfully by updating the SharePath attribute of the quorum resource to utilize the correct witness server.

Comments

  • Anonymous
    January 01, 2003
    @Jack:

    I guess I can only speak from experience and the multitude of successful installations that i see (including those that actually service Office 365 for example).

    We're always here to help - and you can contact me directly through my blog should you desire - if there is some constructive assistance we can provide.

    TIMMCMIC

  • Anonymous
    January 01, 2003
    @Ayyappan... It should be a shared folder. TIMMCMIC

  • Anonymous
    January 01, 2003
    @JasonK: Excellent - I'm glad you found the command you needed. TIm

  • Anonymous
    January 01, 2003
    @Jason: My apologies for the delayed reply - I was offline. You do not pre-stage a secondary witness for witness failures like this.  In the event the server hosting the witness is offline / down you can simply run the set-databaseavailabilitygroup command and move the witness to a server that is still running. TIMMCMIC

  • Anonymous
    January 01, 2003
    @Jack:

    I'm sorry that you feel that way. There's an abundance of evidence to the contrary though...

    Maybe you should consider a support case to help work through some of the issues that you have been experiencing.

    TIMMCMIC

  • Anonymous
    January 01, 2003
    The comment has been removed

  • Anonymous
    January 01, 2003
    @Graham: I recommend configurating hte cluster for node majority and then running set-databaseavailabilitygroup which will convert the DAG back to the correct quroum type. TIMMCMIC

  • Anonymous
    January 01, 2003
    @Rick: Sorry for the delay in getting back to you.  I believe what you experienced is correct.  If the nodes + witness are in the primary and the WAN is lost for any reason the other nodes in the DR datacenter would loose quorum and therefore not be able to mount databases. You could place a witness in a third site - but I'm not sure what this would fix for you.  If the witness was in the third site and the primary site was legitimately down then the third site witness would provide the necessary vote for quorum.  Although in the situation you described if the primary was down then you would go through a site level activation leveraging the alternate file share witness. The third site witness also has several caveats to it which makes it not reliable in some installations.   TIMMCMIC

  • Anonymous
    January 01, 2003
    @V-9Prmo

    https://technet.microsoft.com/en-us/library/dd298065(v=exchg.150).aspx

    The requirements for the witness server are as follows:

    •The witness server can't be a member of the DAG.


    •The witness server must be in the same Active Directory forest as the DAG.


    •The witness server must be running Windows Server 2012, Windows Server 2008 R2, Windows Server 2008, Windows Server 2003 R2, or Windows Server 2003.


    •A single server can serve as a witness for multiple DAGs. However, each DAG requires its own witness directory.

    ========================

    An odd number of nodes does not have a file share witness configured in cluster.

    ========================

    An odd number of nodes does not have a file share witness configured in cluster, and will not have one automatically configured if the vote count becomes even because a node went down.

  • Anonymous
    January 01, 2003
    @Pierre

    No problem.

    TIMMCMIC

  • Anonymous
    January 01, 2003
    @Pierre:

    If your security team is that concerned about the witness.log on a VM that you own in your tenant provisioned in the cloud then you should probably open a case for an official answer.

    It is not supported to modify the contents of the directory, and you should not be looking for or considering any plan of deleting the file etc in response to a security team request.

    In the grand scheme of things this should be benign - if they are really that worried they should start with ensuring the host is secure rather than worrying about what's contained in the witness.log.

    TIMMCMIC

  • Anonymous
    January 01, 2003
    @Pierre:

    Sure...so you will not be able to read the contents of the witness.log file. (You should probably also tell your security guys there's not much to worry about).

    When the witness is necessary to maintain quorum - the cluster will log into the file certain information regarding the cluster quorum state - including the paxos information from the cluster.

    It is strictly used for cluster quorum determination.

    TIMMCMIC

  • Anonymous
    August 22, 2010
    Can the name displayed by "Failover Cluster Manager / Cluster Core Resources" be manually updated without issue?

  • Anonymous
    February 28, 2011
    Tim, We have a DAG with 7 nodes and the cluster is currently in a Node Majority configuration. We are migrating off of our current witness server and decommissioning it, but we need to change the witness server first so there are no references to the decommissioned server. We understand that in an odd numbered cluster the witness server really isn't in use, but we want to maintain the witness server settings just in case we remove a server from the DAG, which would then activate the witness server. Are there any other necessary steps that we need to take before changing the witness server configuration using the recommended set-databaseavailabilitygroupserver –witnessServer command? Thanks for any help you can provide!

  • Anonymous
    February 28, 2011
    Nevermind, we ran some tests in our test lab and figured it out. We were able to run the Set-DatabaseAvailabilityGroup -WitnessServer command without a problem while the cluster was in a Node Majority configuration. We then removed one of the members of the DAG and verified that the cluster changed to a Node and File Share Quorum. It worked beautifully! Thanks!

  • Anonymous
    February 28, 2011
    Excellent - that was the exact solution that I would have provided. TIMMCMIC

  • Anonymous
    April 19, 2011
    Jason, we have a DAG at now with 2 members and we're using NLB for CAS/HUB so i have my FSW in one of my CAS/HUB server, my question is what will happen if this server fail? can i have a FSW in another server in case the first one fail?

  • Anonymous
    May 11, 2011
    The comment has been removed

  • Anonymous
    June 12, 2013
    Tim, Is the file share witness not a shared folder? I see in the actual location that it is not a shared folder.

  • Anonymous
    July 23, 2013
    Hello all, Can some on tell me how to change the witness server with the Set-Databaseavailability? I tried to change the name but when i use Set-DatabaseAvailabilityGroup -Identity DAG01 -WitnessServer commserve01.corp.loc -WitnessDirector y d:FSW_DAG01 the FSW name stays the same. Greats, Menno

  • Anonymous
    August 01, 2013
    We have an Exchange 2010 (Ent Server 2008 R2) DAG running 16 nodes with two physical data centers containing 8 Exchange servers each. One physical site maintains all of our "1st preference" DAG members and we keep the primary File Share Witness at this same data center on a public folder server. We found recently during a network outage that our servers would not automatically fail over to the other physical site as the FSW was apparently "locked" and could not be accessed by the servers at the other site over the network. To provide some redundancy, is there a way to place the FSW in a third physical location (accessible by both data centers) such that it would provide FSW functionality but not allow one site that disappears from the network to lock it? We were able to manually force the FSW over to the other side after some work but would like this to be automated if possible.   Thanks for any suggestions

  • Anonymous
    May 19, 2014
    This DAG seems extremely flaky and problematic and poorly designed. It is such a shame as it seems like a good idea in theory.

  • Anonymous
    May 19, 2014
    A support case with who? Microsoft? I don't have a time to be calling microsoft every time the DAG decides to come up with a new problem.

    I would be interested in some of that evidence because i find that microsoft likes to convince itself that it creates better software than it actually does.

  • Anonymous
    May 19, 2014
    I appreciate your efforts and without people like you making blogs, I don't know how a lot of these MS products would ever function properly. As a lot of these sort of "fixes" are not in the microsoft training nor available in official documentation. I have experienced similar problems with SCCM, WDS/MDT, DFS and so on..

  • Anonymous
    June 23, 2014
    The comment has been removed

  • Anonymous
    February 02, 2015
    Hi Tim,

    Can we have FSW on a Mailbox server which is a part of a DAG?

    What impact will it have if I have 11 member servers in a DAG?

    Since its an odd number of nodes It wont need an FSW but what if one node goes down, what will be the possible impact?

  • Anonymous
    May 04, 2015
    can someone tell me the witness.log content ? we are not able to read it with Notepad.
    the security department ask because we are considering to put witness in azure for Exchange 2013
    regards,

  • Anonymous
    May 04, 2015
    can someone tell me the witness.log content ? we are not able to read it with Notepad.
    the security department ask because we are considering to put witness in azure for Exchange 2013
    regards,

  • Anonymous
    May 04, 2015
    i m so glad and grateful you answer... thanks a lot...

    1. i know, you are right... but security guys want more explanations about witness.log content before put it in cloud :D
    so lets say it cluster state : such as nodes up, node up, time lock etc

    2. can you just confirme it s right : (my final question dont worry)
    1 DAG : 2 nodes + 1 fsw.
    2 nodes ups = fsw no needed so shared directory can be empty.?
    1 node fail = fsw is need for quorum purpose so witness.log is created and locked by the locking node. ?

    My question is : only when needed the witness.log is created and locked otherwise informations are outdated and the witness.log can be deleted ??

    regards

  • Anonymous
    May 04, 2015
    i m so glad and grateful you answer... thanks a lot...

    1. i know, you are right... but security guys want more explanations about witness.log content before put it in cloud :D
    so lets say it cluster state : such as nodes up, node up, time lock etc

    2. can you just confirme it s right : (my final question dont worry)
    1 DAG : 2 nodes + 1 fsw.
    2 nodes ups = fsw no needed so shared directory can be empty.?
    1 node fail = fsw is need for quorum purpose so witness.log is created and locked by the locking node. ?

    My question is : only when needed the witness.log is created and locked otherwise informations are outdated and the witness.log can be deleted ??

    regards

  • Anonymous
    May 04, 2015
    in fact you are totally right !
    thanks a lot