Unable to join a new node to Windows Failover Cluster 2012 R2

JonVeev 46 Reputation points
2023-05-01T20:46:37.01+00:00

Hi,

I am in a weird situation with our Windows 2012 R2 three-node failover cluster. We have a current setup of a three-node failover cluster with file witness share. Two physical nodes died, and I had to evict both nodes from the cluster, but now when I am trying to add a new node back with Windows 2016 in the cluster, I am getting the below error.

Cluster failed to start. The latest copy of cluster configuration data was not available within the set of nodes attempting to start the cluster. Changes to the cluster occurred while the set of nodes were not in membership and as a result were not able to receive configuration data updates. .

Votes required to start cluster: 2

Votes available: 1

Nodes with votes: <new server> <old server>

Guidance:

Attempt to start the cluster service on all nodes in the cluster so that nodes with the latest copy of the cluster configuration data can first form the cluster. The cluster will be able to start and the nodes will automatically obtain the updated cluster configuration data. If there are no nodes available with the latest copy of the cluster configuration data, run the 'Start-ClusterNode -FQ' Windows PowerShell cmdlet. Using the ForceQuorum (FQ) parameter will start the cluster service and mark this node's copy of the cluster configuration data to be authoritative. Forcing quorum on a node with an outdated copy of the cluster database may result in cluster configuration changes that occurred while the node was not participating in the cluster to be lost.

I am not sure what to do here since the cluster is up and running with one remaining node and I do not want to risk it going down as it is running critical applications. Should I disable quorum witness altogether then join the new node and then enable it again? Will doing this affect the cluster in any way? Any help would be greatly appreciated!

Thanks in advance!

Windows Server 2012
Windows Server 2012
A Microsoft server operating system that supports enterprise-level management, data storage, applications, and communications.
1,590 questions
Windows Server
Windows Server
A family of Microsoft server operating systems that support enterprise-level management, data storage, applications, and communications.
13,089 questions
Windows Server Clustering
Windows Server Clustering
Windows Server: A family of Microsoft server operating systems that support enterprise-level management, data storage, applications, and communications.Clustering: The grouping of multiple servers in a way that allows them to appear to be a single unit to client computers on a network. Clustering is a means of increasing network capacity, providing live backup in case one of the servers fails, and improving data security.
999 questions
0 comments No comments
{count} votes

2 answers

Sort by: Most helpful
  1. Limitless Technology 44,336 Reputation points
    2023-05-02T14:49:49.0633333+00:00

    Hello

    Thank you for your question and reaching out.

    1. Please run Cluster validation Wizard and check if all Ok.
    2. Please Check the heartbeat/quorum.
    3. Please check Disk Share with a network share on the same subnets.

    Reference:

    https://social.technet.microsoft.com/Forums/windowsserver/en-US/2137f953-8102-462d-b5d9-d75d2735b7de/windows-ms-cluster-2016-windowsfailoverclustering-event-id-1561?forum=ws2016

    --If the reply is helpful, please Upvote and Accept as answer--


  2. JonVeev 46 Reputation points
    2023-05-15T18:05:01.26+00:00

    Suppose anyone is looking for the answer to being unable to add a node to a cluster. Do this.

    Instead of going around and trying to ask for help from everyone and trying different methods. Use the below command to generate the cluster log and check why it is causing the error. In my case, I tried everything and later found out that the cluster was terminating the connection from a disabled cluster network; upon enabling the network for cluster communication, everything worked flawlessly.

    Get-ClusterLog -Destination C:\ –UseLocalTime

    Here are the instructions from the article I found online:

    https://blog.it-kb.ru/2014/10/31/server-could-not-be-added-to-the-cluster-hyper-v-waiting-for-notification-that-node-is-a-fully-functional-member-of-the-cluster-error-code-is-0x5b4-unable-to-successfully-cleanup/

    Error when adding a node to a Hyper-V cluster: Waiting for notification that node is a fully functional member of the cluster - The error code is 0x5b4 - Unable to successfully cleanup

    31.10.2014Author:[Alexey Maksimov](https://blog.it-kb.ru/author/blogroot/"Перейти на сайт Алексей Максимов")12 359 view4 Comments

    A Windows Server 2012 R2-based Hyper-V cluster has been created with a single node and a disk witness. After that, we try to add a new node to the cluster. As a result, an error appears:

    If you look at the View Report, you can see that the node is undergoing initial validation, and then the operation fails with a message about some kind of waiting timeout:

    Adding KOM-AD01-VM07.holding.com to the cluster.
    Validating cluster state on node KOM-AD01-VM07.
    Getting current node membership of cluster KOM-AD01-VMFC01.
    Adding node KOM-AD01-VM07 to Cluster configuration data.
    Validating installation of the Network FT Driver on node KOM-AD01-VM07.
    Validating installation of the Cluster Disk Driver on node KOM-AD01-VM07.
    Configuring Cluster Service on node KOM-AD01-VM07.
    Waiting for notification that Cluster service on node KOM-AD01-VM07.holding.com has started.
    Waiting for notification that node KOM-AD01-VM07 is a fully functional member of the cluster.
    Cluster service on node KOM-AD01-VM07 did not reach the running state. The error code is 0x5b4. For more information check the cluster log and the system event log from node KOM-AD01-VM07. This operation returned because the timeout period expired.
    Unable to successfully cleanup.
    The server 'KOM-AD01-VM07.holding.com' could not be added to the cluster.
    An error occurred while adding node 'KOM-AD01-VM07.holding.com' to cluster 'KOM-AD01-VMFC01'.
    This operation returned because the timeout period expired
    

    To understand the root of the problem, additional debugging information is required, which will help you get the recommendations in the article Failover Clustering and Network Load Balancing Team Blog - How to Troubleshoot Create Cluster failures in Windows Server 2012. In particular, to get the log, the clustering components will be executed on both servers (on the current cluster node and the server being added to the cluster) Powershell cmdlet that uploads this log to a text file:

    Get-ClusterLog
    

    As a result, a log file with the name of the form KOM-AD01-VM04_cluster.log will appear along the specified path

    In our particular example, the examination of the log on the server being added to the cluster did not bring clarity, however, in the log of the server owning the cluster, multiple warnings of the form were noticed:

    00000ad8.00001dd4::2014/10/29-20:14:55.154 WARN  mscs::ListenerWorker::operator (): (5060)' because of '[FTI][Initiator] Discarding connection from 10.160.35.58:~3343~ to 10.160.35.57:~3343~ (node KOM-AD01-VM07) because it uses a disabled network.'
    

    As a result, it turned out that the subnet related to Live Migration that was turned off in the cluster was the cause of the problem.

    After enabling the appropriate cluster network, the new node was successfully added to the cluster.

    Error when adding a node to a Hyper-V cluster : Waiting for notification that node is a fully functional member of the cluster - The error code is 0x5b4 - Unable to successfully cleanup

    31.10.2014Author:[Alexey Maksimov](https://blog.it-kb.ru/author/blogroot/"Перейти на сайт Алексей Максимов")12 359 view4 Comments

    Windows Server 2012 R2-based Hyper-V cluster has been created with a single node and a disk witness. After that, we try to add a new node to the cluster. As a result, an error appears:

    If you look at the View Report, you can see that the node is undergoing initial validation, and then the operation fails with a message about some kind of waiting timeout:

    Adding KOM-AD01-VM07.holding.com to the cluster.
    Validating cluster state on node KOM-AD01-VM07.
    Getting current node membership of cluster KOM-AD01-VMFC01.
    Adding node KOM-AD01-VM07 to Cluster configuration data.
    Validating installation of the Network FT Driver on node KOM-AD01-VM07.
    Validating installation of the Cluster Disk Driver on node KOM-AD01-VM07.
    Configuring Cluster Service on node KOM-AD01-VM07.
    Waiting for notification that Cluster service on node KOM-AD01-VM07.holding.com has started.
    Waiting for notification that node KOM-AD01-VM07 is a fully functional member of the cluster.
    Cluster service on node KOM-AD01-VM07 did not reach the running state. The error code is 0x5b4. For more information check the cluster log and the system event log from node KOM-AD01-VM07. This operation returned because the timeout period expired.
    Unable to successfully cleanup.
    The server 'KOM-AD01-VM07.holding.com' could not be added to the cluster.
    An error occurred while adding node 'KOM-AD01-VM07.holding.com' to cluster 'KOM-AD01-VMFC01'.
    This operation returned because the timeout period expired
    

    To understand the root of the problem, additional debugging information is required, which will help you get the recommendations in the article Failover Clustering and Network Load Balancing Team Blog - How to Troubleshoot Create Cluster failures in Windows Server 2012. In particular, to get the log, the clustering components will be executed on both servers (on the current cluster node and the server being added to the cluster) Powershell cmdlet that uploads this log to a text file:

    Get-ClusterLog
    

    As a result, a log file with the name of the form KOM-AD01-VM04_cluster.log will appear along the specified path

    In our particular example, the examination of the log on the server being added to the cluster did not bring clarity, however, in the log of the server owning the cluster, multiple warnings of the form were noticed:

    00000ad8.00001dd4::2014/10/29-20:14:55.154 WARN  mscs::ListenerWorker::operator (): (5060)' because of '[FTI][Initiator] Discarding connection from 10.160.35.58:~3343~ to 10.160.35.57:~3343~ (node KOM-AD01-VM07) because it uses a disabled network.'
    

    As a result, it turned out that the subnet related to Live Migration that was turned off in the cluster was the cause of the problem.

    After enabling the appropriate cluster network, the new node was successfully added to the cluster.

    0 comments No comments

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.