Exchange 2016 DAG node chashes

Hi!
Need some help with my Exchange 2016 DAG.
At last few moths we encountered problem with our DAG nodes.
We have one DAG (ExDAG) with two nodes: Ex01 and Ex02.

Situation. By some unknown reason Ex01 periodically crashes. Event log can't give us any explanation about reasons. But that's not the worst thing, after all we have DAG. The worst is that some times after Ex01 crashes DAG node (Ex01) not up to cluster.
Get-clusternode returns: Ex01 - state - down
Get-clusterNetwork returns: Ex01 - state - down
Get-clusterNetworkInterface: Ex01- state - Unavailable
At the same time Ex01 networking working without any problems: ping, telnet and other works fine and show no problems.
We tried many of solutuon. Nothings helps. Except one: changing EX01 IP address to any other cause this node to up in cluster. After that all works perfectly.
Who can help me understand what happens? Why only IP-address change help us? I try to do
netsh int ip reset
netsh winsock reset
but no luck.
Any help will be appreciated!

1 answer

Jake Zhang-MSFT 7,235 Reputation points Microsoft Vendor

2024-11-13T06:08:54.8+00:00
Hi @Евгений Котляревский,

Welcome to the Microsoft Q&A platform!

Based on your description, you are dealing with a problem in a Database Availability Group (DAG). Here are some potential causes and solutions for Exchange 2016 DAG node problems:

Even if your network tools do not show any problems, there may be an IP address conflict or ARP cache problem. Changing the IP address may temporarily resolve this conflict. You may want to check any devices on the network that may be causing this conflict.

Use the "Get-ClusterNetwork" and "Get-ClusterNetworkInterface" commands to check the cluster network configuration and its status. And make sure that the Cluster Network Name is up and running. You can try disabling and enabling the affected network interface in the Failover Cluster Manager.

Sometimes, a network interface card (NIC) can cause problems even if it appears to be functioning properly. Updating the NIC driver or replacing the NIC may help resolve the problem.

There may be a problem with the cluster service on Ex01. Restart the Cluster service on Ex01 and see if it fixes the problem:

Stop-Service ClusSvc Start-Service ClusSvc

Event Log and Cluster Log: While you mentioned that the event log did not show any issues, it may be helpful to enable more detailed logging for the Cluster service. This can sometimes reveal hidden issues. You can use the Get-ClusterLog cmdlet to generate a detailed cluster log for further analysis.

Ensure that the network used for the DAG is isolated from other traffic as much as possible. This helps prevent any interference from other network activity.

Please feel free to contact me for any updates. And if this helps, don't forget to mark it as an answer.

Best,

Jake Zhang
Please sign in to rate this answer.
Евгений Котляревский 21 Reputation points

2024-11-13T09:25:27.6566667+00:00

Hi,@Jake Zhang-MSFT
1. There are definitely no any IP-address conflicts, We checked it many times.
About ARP cache - there is no problem either.
2. "Get-ClusterNetwork" shows that network for ex01 is "Down"
"Get-ClusterNetworkInterface" shows that network interface for ex01 is "Unavailable".
And I can't "And make sure that the Cluster Network Name is up and running. You can try disabling and enabling the affected network interface in the Failover Cluster Manager.". It's IP-less DAG, there is no administrative point for this cluster
3. As I wrote in my start topic - it is virtual machine, and we have replaced virtual NICs many time. Its help for some time, but after VM crash problem returns
4. Restarting the service does not give any results.
5. in ClusterLog there are next messages:
"INFO Cluster node 'ex01' lost communication with cluster node 'EX02'. Network communication was reestablished. This could be due to communication temporarily being blocked by a firewall or connection security policy update. If the problem persists and network communication are not reestablished, the cluster service on one or more nodes will stop. If that happens, run the Validate a Configuration wizard to check your network configuration. Additionally, check for hardware or software errors related to the network adapters on this node, and check for failures in any other network components to which the node is connected such as hubs, switches, or bridges."
"

DBG Cluster node 'EX02' was removed from the active failover cluster membership. The Cluster service on this node may have stopped. This could also be due to the node having lost communication with other active nodes in the failover cluster. Run the Validate a Configuration wizard to check your network configuration. If the condition persists, check for hardware or software errors related to the network adapters on this node. Also check for failures in any other network components to which the node is connected such as hubs, switches, or bridges.

"
"

DBG The Cluster service was halted to prevent an inconsistency within the failover cluster. The error code was '1359'.

"
"

DBG The cluster Resource Hosting Subsystem (RHS) process was terminated and will be restarted. This is typically associated with cluster health detection and recovery of a resource. Refer to the System event log to determine which resource and resource DLL is causing the issue.

"
"

DBG Cluster service failed to start because this node detected that it does not have the latest copy of cluster configuration data. Changes to the cluster occurred while this node was not in membership and as a result was not able to receive configuration data updates.

"
"
ERR Node 'ex01' failed to form a cluster. This was because the witness was not accessible. Please ensure that the witness resource is online and available.

" - I've checked witness share - it's accessible from both nodes, i don't see any problems with it
"

ERR Cluster node 'ex01' failed to join the cluster because it could not communicate over the network with any other node in the cluster. Verify network connectivity and configuration of any network firewalls.

" - both nodes can communicate with each other, no problems was deteceted
"

ERR Cluster failed to start. The latest copy of cluster configuration data was not available within the set of nodes attempting to start the cluster. Changes to the cluster occurred while the set of nodes were not in membership and as a result were not able to receive configuration data updates. .

"
All this messages says to me that there are problems with nodes communications, but no problems ware detected. All test completed successfully (ping,telnet etc).
6. NICs located at different VLANs, and no other NICs in it. So i don't think there is any influence from other network loads.

Any ideas?

Евгений Котляревский 21 Reputation points

2024-11-13T20:44:06.9633333+00:00

@Jake Zhang-MSFT , thank for your assistance!
Let me answer point by point.

There is definitely no any IP-duplication. This was checked many times

Get-ClusterNetwork shows Ex01 status "Down"
Get-ClusterNetworkInterface shows Ex01 Cluster Network status "Unavailable".
And I can't use Failover Cluster Manager, this is IP-less DAG, there is no administrative point

Both DAG nodes are VM (Esxi), and we replaced NIC many times already. This the only way (together with the change of address) to bring failed node UP

Stop/start cluster service has no any results.

You quite right, CLusterLog contains some message, like this:

[System] 000016b4.0000867c::2024/11/13-08:35:25.388 ERR Node 'EX02-SPLX' failed to form a cluster. This was because the witness was not accessible. Please ensure that the witness resource is online and available.

[System] 000016b4.0000867c::2024/11/13-08:35:25.388 ERR Cluster node 'EX02-SPLX' failed to join the cluster because it could not communicate over the network with any other node in the cluster. Verify network connectivity and configuration of any network firewalls.

[System] 000016b4.0000867c::2024/11/13-08:35:25.389 ERR Cluster failed to start. The latest copy of cluster configuration data was not available within the set of nodes attempting to start the cluster. Changes to the cluster occurred while the set of nodes were not in membership and as a result were not able to receive configuration data updates. .
and other like that.
These messages clearly tells me we have network problems. But all our tests show no network problems. Including witness share, it's accessible from both nodes. And both nodes can access to each other. But cluster says "lost communication".

So we have weird situation: The network seems to be there, but the cluster says "Hey! My network not working!"

P.S. i've tried "netsh winsock reset" and "netsh int ip reset" - no luck
We're out of ideas. What can we do else?

Amit Singh 4,986 Reputation points

2024-11-18T06:00:13.3766667+00:00

Based on the detailed logs and your description, it seems like there may be a complex interaction between the network configuration, cluster communication, and possibly even virtual switch or NIC teaming issues on your ESXi host. Running the Test-Cluster cmdlet, checking for any hidden network issues or latency, ensuring consistent time synchronization, and reviewing your quorum configuration should help narrow down the root cause.

Amit Singh 4,986 Reputation points

2024-11-18T06:01:20.6966667+00:00

Let us know if it works.

Jake Zhang-MSFT 7,235 Reputation points Microsoft Vendor

2024-11-19T02:55:14.2833333+00:00

Hi @Евгений Котляревский,

Thanks for the detailed information. It sounds like you've done a lot of troubleshooting. Here are a few steps and considerations that may help resolve the issue:

The error message indicates that the witness is not accessible. Double-check the witness configuration and make sure the witness server is online and accessible from both DAG nodes. Verify the permissions and network settings of the witness share.

Since this is a no-IP DAG, make sure the cluster network settings are configured correctly. Sometimes, the cluster network may need to be reset or reconfigured. You can use the Get-ClusterNetwork and Get-ClusterNetworkInterface cmdlets to check the current status and settings.

Verify that there are no firewall rules or security settings blocking cluster communication. Make sure the ports required for cluster communication are open and there are no security policies that interfere with network traffic.

Make sure the cluster service account has the required permissions to access the network and witness share. Sometimes, permission issues can prevent the cluster from communicating properly.

Since both nodes are virtual machines on ESXi, make sure the virtual network settings are stable and configured correctly. Check for network latency or packet loss issues that may affect cluster communication.

As a last resort, consider removing the problematic node from the DAG and then re-adding it. This can sometimes resolve underlying configuration issues. Use the Remove-DatabaseAvailabilityGroupServer and Add-DatabaseAvailabilityGroupServer cmdlets to do this.

Please feel free to contact me for any updates. And if this helps, don't forget to mark it as an answer.

Best,

Jake Zhang

Jake Zhang-MSFT 7,235 Reputation points Microsoft Vendor

2024-11-25T09:35:03.3766667+00:00

Hi @Евгений Котляревский,

Please kindly consider this comment as a warm follow up, I want to know if your problem has been solved. If you still have any other questions, please feel free to comment here and I'm glad to provide further support.If the above info is helpful to this question, you can click "Accept Answer". Have a nice day!

Best,

Jake Zhang

Jake Zhang-MSFT 7,235 Reputation points Microsoft Vendor

2024-11-26T07:22:25.0633333+00:00

Hi @Евгений Котляревский,

Is there any update on this thread? If the issue has been resolved, please mark the helpful replies as answers, this will make answer searching in the forum easier and be beneficial to other community members as well. Thanks for your understanding.

Best,

Jake Zhang

Евгений Котляревский 21 Reputation points

2024-11-26T08:49:12.3466667+00:00

Hi, @Jake Zhang-MSFT
Since my last post there are no significant updates. The only one we had catch: last cluster crash was preceded by network link problem. This is link between two of our sites where exch servers is placed. Our network team switched to backup channel, but cluster hasn't fixed after that. One node staing in "Down" state, cluster network were in "unreachable" state. Meanwhile witness share was fully accessible from both nodes, no any network problem was detected.
The only thing helped: change IP-address on failed node. After IP-address was changed, node immediately became "up", and whole cluster became health.
So, I can say intersite link fail cause claster fail, but fixing this link do not cause cluster fix.
By the way, in last crash witnes share was in the same site with failed node.
Now I've made some changes: i change witness server to other server in case the problem was in it.
So, I waiting the next crash.

Jake Zhang-MSFT 7,235 Reputation points Microsoft Vendor

2024-11-28T09:45:24.0733333+00:00

Hi @Евгений Котляревский,

Thanks for your response. Here are some additional thoughts and suggestions based on your latest update:

Since there were network link issues before the cluster crashed, it is critical to ensure that the inter-site links are stable and reliable. Even a short outage can cause cluster nodes to lose communication. Monitoring the links for any fluctuations or packet loss may help identify potential issues.

The cluster does not automatically recover after the link is restored, which indicates that there may be a configuration or timeout problem. You can check the cluster network settings and adjust the cluster heartbeat and timeout settings to see if this helps with automatic recovery.

Moving the witness server to a different site is a good step. Make sure the new witness server is consistently accessible from both nodes. This helps maintain quorum and prevent cluster failures due to the witness being unreachable.

The need to change the IP address to recover the node indicates that there may be an issue with the ARP cache or network routing. You may want to investigate the ARP tables and routing configuration on your network devices to see if there are any anomalies.

Continue to monitor the cluster logs for any additional clues. The logs can provide valuable insights into what happened during the failure and recovery process. Look for any patterns or recurring errors that may point to the root cause.

It sounds like the changes you made are correct. Hopefully, the new witness server will help stabilize the cluster.

Please feel free to contact me for any updates. And if this helps, don't forget to mark it as an answer.

Best,

Jake Zhang
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Exchange 2016 DAG node chashes

1 answer

Your answer