This browser is no longer supported.
Upgrade to Microsoft Edge to take advantage of the latest features, security updates, and technical support.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(272, 49%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJA%3C/text%3E%3C/svg%3E)
We've had Windows clusters configured on various versions of Windows Server for a few years to support SQL Availability Groups and things have generally worked well. Recently, I've seen quite a few clusters stop when the secondary server (for SQL) reboots. I'll give details of the most recent one that happened over the weekend.
There are 2 Windows Server 2016 servers running SQL 2016 and a file share witness on Windows Server 2019. The SQL AG is asynchronous with manual failover. I never want the secondary to take over unless I do it manually. The primary server (Node 2) and FSW are at 1 site, the secondary (Node 1) is at another.
An issue with a VMware host caused the secondary server to reboot on Saturday night. Although the cluster should still have had quorum, the cluster service on the primary stopped. I don't have much experience with troubleshooting clusters, but used Get-ClusterLog to generate a detailed log. In there, I found this:
00001910.00000954::2023/01/14-23:26:37.287 INFO [RES] File Share Witness <File Share Witness>: Read 88 bytes from the witness file share.
00001910.00000954::2023/01/14-23:26:37.288 INFO [RES] File Share Witness <File Share Witness>: Releasing temporary lock on witness file share.
00000bdc.00000718::2023/01/14-23:26:37.288 INFO [QUORUM] Node 2: received request for quorum witness info. Replying with paxos tag 120:120:30042
00000bdc.00001b24::2023/01/14-23:26:37.289 ERR Quorum witness has better epoch than local node, this node must have been on the losing side of arbitration!
00000bdc.00001b24::2023/01/14-23:26:37.289 ERR [GUM] Node 2: failed to update epoch after one of the nodes went down. error 995
00000bdc.00001b24::2023/01/14-23:26:37.289 ERR mscs::GumAgent::ChangeEpoch: (995)' because of 'Quorum witness has better epoch than local node ABORT!'
00000bdc.00001b24::2023/01/14-23:26:37.289 ERR Changing Epoch failed (status = 1359)
00000bdc.00001b24::2023/01/14-23:26:37.289 INFO [NETFT] Cluster Service preterminate succeeded.
Event logs on the primary server include event ID 1073 with this message:
The Cluster service was halted to prevent an inconsistency within the failover cluster. The error code was '1359'.
It looks like it thens stops the cluster service, which took the SQL AG down as well. The secondary server rebooted quickly and everything came back within a minute, but I'm concerned that this is happening more frequently.
When patching secondary servers, they're quite open on the automated schedule, so we do nothing other than relying on the primary server and FSW keeping quorum. For the primary servers, we fail everything over to the secondary before patching in a maintenance window, then fail back to the primary at the end of the window. Several of the secondary servers rebooting for patching has caused the same issue recently.
Today I've manually shut down the secondary server and left it off for 10 minutes. The cluster behaved as expected, the primary continued with no interruptions. I've not been able to investigate while a cluster is in a failed state and can't replicate it, so not sure how to proceed.
Without going through the internals of how the cluster service works and the relevance of the paxos tag, the reboot of your secondary VMWare server (taking down Node 1 in the process) caused the cluster to lose quorum, despite still having 2 out of 3 votes (Node 2 and the FSW). This is by design. It's frustrating to have that false sense of security despite having 2 out of 3 votes. But if you understand the internals, it really is by design.
The way that the cluster determines quorum is by having majority of votes. The votes are decided by the voting members, if they are still available. Note that dynamic quorum and dynamic witness don't work in a 3-voting member (2-node + witness) scenario. It only works when you have more than 3 voting members.
Depending on how far away those voting members are from each other, there could be delays on updating the paxos tag. The paxos tag is a way of guaranteeing the consistency of the cluster configuration across all the cluster nodes in order to prevent split brain. The paxos tag gets updated anytime a cluster configuration is made. This can be as simple as a node being removed from the voting members or as complex as changing the AG listener to have multiple IP addresses. I refer to paxos tag, although the cluster log uses the word epoch. Take a look at this entry in the cluster log.
00000bdc.00001b24::2023/01/14-23:26:37.289 ERR Quorum witness has better epoch than local node, this node must have been on the losing side of arbitration!
Note that the paxos tag should be the same across all nodes in the cluster in order to maintain consistency. In this case, it isn't (FSW has a more updated paxos tag than Node 2). I could be wrong but my guess is that the secondary node (Node 1) was able to update the paxos tag in the FSW but not the primary node (Node 2). So, when the primary node lost connectivity with the secondary node, it checked the FSW and saw that the paxos tag was inconsistent. It tried to update it but failed.
00000bdc.00001b24::2023/01/14-23:26:37.289 ERR [GUM] Node 2: failed to update epoch after one of the nodes went down. error 995
00000bdc.00001b24::2023/01/14-23:26:37.289 ERR Changing Epoch failed (status = 1359)
Then, it sopped the cluster service
00000bdc.00001b24::2023/01/14-23:26:37.289 INFO [NETFT] Cluster Service preterminate succeeded.
I call this the cluster's self-preservation mechanism in action: "I can't make a decision!!! I'll just shutdown so I don't have to make a decision!!!"
Here's a reference documentation if you want to understand the internals of the paxos tag. Ignore the version of Windows Server. It's the same algorithm for newer versions.
>Today I've manually shut down the secondary server and left it off for 10 minutes. The cluster behaved as expected, the primary continued with no interruptions.
Remember the false sense of security of having 2 out of 3 members, thinking you have quorum? This has the same risk the moment you lose the FSW.
Action Item: IGNORE
You don't have high availability anyway. Your AG is in async so there's no way for automatic failover to ever happen. Improve patching process and set this as expected behavior. Just make sure you are still meeting your recovery objectives (RPO/RTO).
You might be tempted to dig deeper as there could be something going on with your NTP, network, VMWare hosts, VM guests, AD, etc. But since high availability isn't a priority based on your setup, it's not worth the time investment to investigate further.
Hi Edwin. Thanks for your comprehensive answer, it's very helpful. I think I understand the process a bit better now, including what's gone wrong. Unfortunately, I'm a DBA with decent Windows knowledge, but I think it's unlikely that I'll get to a level where I fully undertsand this and we don't have anyone else in the organisation who will. As we need SQL Availability Groups for DR, I'm a bit stuck with clusters.
When you say to improve the patching process, could you explain a bit more, please? When manually rebooting a secondary server, I'll always pause it. As patching windows are 3 hours, I haven't done that as it'll prevent databases from synchronising during that time. I could do that if it's advisable, I'll just need to increase space for database logs that will build up.
The most recent issue, however, wasn't during patching but it was an unexpected, out-of-hours reboot of the secondary server, making it impossible to plan for. Other than tracking down why the primary doesn't have the latest paxos tag, is there anything that I can do to make sure the loss of the secondary doesn't affect the primary as long as the FSW is still up?
In an ideal world, I'd have something like the old database mirroring where the primary stays up even if it finds itself completely isolated. There would never be a split brain scenario as the secondary could only take over if done manually.
I no longer recommend a 2-replica Availability Group for DR. In fact, I no longer recommend stretched clusters with AGs for DR. That's because of the dependency on the Windows Server Failover Cluster. If you don't need HA, drop the AG. The complexity and the additional overhead for managing it really isn't worth it.
is there anything that I can do to make sure the loss of the secondary doesn't affect the primary as long as the FSW is still up?
Tweaking the CrossSubnetDelay and CrossSubnetThreshold parameter values on the failover cluster to increase the heartbeat threshold can provide a "workaround". But this is another example of why I no longer recommend AGs if it's simply for DR. You need to make modifications outside of SQL Server just to make it work. Database mirroring is still supported.
In an ideal world, I'd have something like the old database mirroring where the primary stays up even if it finds itself completely isolated. There would never be a split brain scenario as the secondary could only take over if done manually.
The reason why most common database mirroring deployments do not experience split brain is because they are configured for high performance (asynchronous replication without a witness) in a DR scenario. This is similar to your 2-replica AG in asynchronous mode but without a witness. I'll say it again, database mirroring is still supported. Keep the solution simple, especially since you don't have a team to manage AGs properly. And you don't want to be the only person taking care of this.
As for improving the patching process, simply communicate that this is expected behavior. No need to pause data synchronization.
Thanks Edwin, that's really helpful. I'd only moved away from mirroring due to the threat of its imminent demise and AGs felt overly complex for what we often need to do. I'll look at moving back to mirroring for any of our asynchronous/DR setups.
AGs felt overly complex for what we often need to do
It's the common feeling with most of my customers. That's why I always go back to the main goal: What's the RPO/RTO/SLA for HA and for DR? The response to this should drive the solution.